gordicaleksa / Open-NLLB

Effort to open-source NLLB checkpoints.

MIT License

419 stars 37 forks source link

Choosing the LID model #8

Open vienneraphael opened 1 year ago

vienneraphael commented 1 year ago

Available models

avoid lid.218 (original nllb LID) -> it's CC-BY-NC
https://github.com/laurieburchell/open-lid-dataset -> GPL License, 201 lang
https://fasttext.cc/docs/en/language-identification.html -> lid.176, CC-BY-SA license, 176 lang

Additional informations

https://huggingface.co/hac541309/fasttext_langID_models -> models and hashes