Open de-code opened 4 years ago
de-code
commented
4 years ago BTW the way I found those is by looking at regressions of using my models with the new version of GROBID vs previous version. This affected the title extraction using my models. It could be the models are no longer tolerant to line numbers. I will re-generate the training data etc. and that problem might go away.
kermitt2
commented
3 years ago Hi Daniel,
It's actually working in these examples relatively to these starting numbers. The remaining number 1 is a page number. As the block order follows the PDF stream order by default, the page number appears at the beginning of the page in the ALTO output, although visually it is located at the end of the page.
For the first pdf for instance, we have for the text content stream:
1 Combinatorial effect of promoter activity, mRNA degradation and site-specific
transcriptional pausing in modulating protein expression noise
Sangjin Kim 1,2,3 , Christine Jacobs-Wagner 1,2,3,4*
- second page: 2
ABSTRACT
Genetically identical cells exhibit diverse phenotypes, even when experiencing the same
environment. This phenomenon, in part, originates from cell-to-cell variability (noise) in protein
and so on where this first token is the page number.
This is the same case for the two other examples - page number at the beginning of the page token stream.
BUT
in the second PDF however, there are a few problems with the second and fifth pages for instance with line numbers from `34` to `44` and `66` to `69` still appearing in the ALTO output. In these cases there's a slight change of width and alignment on both sides from `34` (not easy to see) and my clustering method absolutely wants an exact alignment on at least left or right... For covering that, I relaxed slightly the alignment within a 1.0 unit margin.
This case is working too now, following aaac4cd9379395f7e108b20a7a4c13e384545475.
Hi @kermitt2
I have now merged with upstream master and during evaluation I found some error cases where the line numbers are not filtered out.
I can confirm that the line numbers are removed for the example that @lfoppiano was using: https://doi.org/10.1101/2020.04.21.054221 (i.e. it looks like I am doing at least something right).
Here are some examples where it doesn't seem to work. It appears that the first line number (
1is not removed), but subsequent line numbers appear to be removed (I currently don't have a way to visualise the lxml for confirm that more easily). Thus the title is usually affected more.Example 1
https://www.biorxiv.org/content/10.1101/210401v1?versioned=true
Example 2
https://doi.org/10.1101/440115
Example 3
https://doi.org/10.1101/434563
(see also https://github.com/kermitt2/grobid/issues/638#issuecomment-690310642)