Language detection

[AdrienGuille]

• Infer the language from the text using the 'langdetect' library (detects language using naive Bayesian filter, trained on Wikipedia articles)
• Get the actual language by spotting the 'abstract' or 'résumé' keyword on the first page of the pdf file for each article.

[psorianom]

Statistics

Language

I will put here some statistics. Please share your thoughts. The language is defined by, and only by, the title:

• 1041 EGC files, english and french. This should be the same for all. I get the same value with Ciprian's csv.

This may be different according to the language detection approach:

• 935 in French
• 106 in English

PDFs

1page pdf

• All of the 947 files have a valid 1page-pdf URL.
• 3 of them (of the 947) are not convertible into text. They need OCR:
  • One is a very bad scan, impossible to fix. ID=646. Converted to text. Not very good text, still amazingly good, given the input.
  • The two left (IDS=1701,1730) are scans, converted to text. Good results.

Full pdf

For full pdfs things get more complex.

• 736 have a valid fullpdf URL. The 211 left do not or cannot be downloaded for some reason. From those that have a valid URL, the full pdf can be not very good. There are three common problems:
• The pdf is an image, it could be a good image (proper scan) or a bad scan.
• Even if it says full pdf, we download again a 1 page pdf... maybe the pdf do only have 1 page..
• The pdf is all crazy. The quality seems good but somehow the text extracted is garbage. You cannot properly select text in these pdfs. Like ID=1695