Open tomersagi opened 1 year ago
Hi, Great job with this tokenizer. Can we have a list of lang codes? It is not clear which ISO standard you are using for language codes, specifically for languages where there is a modern and ancient version. Thanks
The cli interface instructions include:
--lc LANGUAGE-CODE ISO 639-3
Hi, Great job with this tokenizer. Can we have a list of lang codes? It is not clear which ISO standard you are using for language codes, specifically for languages where there is a modern and ancient version. Thanks