Language cleaning

[ybracke]

It might be helpful to exclude non-German text from the input.

Available options

• See: https://github.com/zentrum-lexikographie/eval-de-lang/issues
• MacBERTh (Manjavacas) used Google's Compact Language Identifier (pycld3) and FastText
• TuringBERT (Hosseini) used spaCy
• Luise says langdetect is a standard solution
• adbar on mattermost: py3langid+langid; fasttext-Wrapper like whatthelang
• Check ZDL lit collection's part about language detection: https://www.zotero.org/groups/4391310/zdl-nlp/collections/9TSK5DKW

[ybracke]

Siehe language-clean