kermitt2
commented
3 years ago Hello @m485liuw !
Thanks a lot for the issue and the detailed analysis of errors.
I think it's a bit difficult to conclude anything with 30 examples added to the existing 592, but it's possible that with the existing set of features (which is limited due to the small size of training data) and for a given relatively homogeneous collection like arXiv we don't improve beyond 300 examples. When evaluating against PubMed Central, scores were still improving as I was reaching the 600 examples, so I was not facing this issue yet.
Your approach is the one I was following: adding examples if they were wrong with the current model and this is how I got the best learning curve so far.
One important aspect I think is that the current training data is "small size" but "high quality. Every labels were checked several times and we pay a lot of attention to have very consistent labelling (in term of what exactly is labeled, which words are excluded from the labeled chunks, etc.). Introducing examples with slightly different/inconsistent/incomplete labeling approach can impact a lot the learning (make it less efficient) and introduces errors in prediction as compared to older version. Working with such small training sets makes every small labeling errors extremely impactful. On the other hand, this approach permits to get very good accuracy with limited training data and to add new examples with some effects.
If you'd like to contribute to GROBID with training data and if you're not under time pressure, I would very happy to review the 30 examples and check the overall consistency.
There would be different ways to improve the accuracy:
-
try a different algorithm than CRF, in particular the deep learning ones with layout features are pretty good to take advantage of more training data. SciBERT is actually very good to pick correctly some fields (like title, which reach 98.58 f1-score on PMC sample while we are at 97.0 with CRF, Levenshtein Matching), while being worst at others (keyword/abstract, possibly due to the sequence length input limit and inability to exploit layout features).
-
use an ensemble approach if you're not constrained by runtime... run several ML header models and use a voting to pick the best fields (I am not working on that in Grobid because I think improving base models and extending training data are more a priority, but this would be a normal optimization to maximize the accuracy )
-
in the old CRF approach, review the features and experiment with new ones intuitively relevant for arXiv papers. More training data makes possible to add more features with positive results. But it's more complicated as it requires to dive into the java.
In general, if the error is due to a segmentation error from the segmentation model, indeed it has to be fixed first by retraining the segmentation model - this would be the way to fix "1. foot/header appears in abstract". Title not identified might come from reading order issue, which is something I am still working on in pdfalto. As it is still not stable, I have excluded examples with strong reading order issues from the training data. "3. key words/ccs in abstract" is a relatively common error I also observed with the current mode, and I was thinking so far that more training data would help this.
m485liuw
Hi, we just finished a small scale of data annotation for header model with ArXiv papers (30 pdfs which was annotated wrong by header model) and retrained the header model with the original data plus newly annotated ones. Unfortunately, we found the accuracy stays the same for our generated test set (1000 ArXiv papers). In fact, we found the accuracy barely increases (~88% for abstract, 95% for header) after 300 training data points. We are wondering is it even possible to increase the accuracy by training with more data? If so, any suggestions on designing the dataset (i.e. any rule of selecting?)
Also, we found most of the errors of header model came from: 1. foot/header appears in abstract 2. title not identified 3. key words/ccs in abstract 4. upper model(segmentation model) wrong. After our annotation on papers with the first 3 issues and retraining, some of the papers with these issues got corrected whereas some new pdfs got annotated wrong (originally correct) with these issues. Do you know how to explain this scenario? Shouldn't these issues be supposed to be fixed after retraining (at least, no new errors of this type come out)?
Plus, as mentioned, some errors come from segmentation model. Do you think it is helpful to retrain the segmentation model?
Looking forward to your reply and any plan on improving the header model.