Lack of documentation to use LID in https://github.com/facebookresearch/fairseq/tree/nllb

[aloka-fernando]

📚 Documentation

In the NLLB-200 documentation, https://github.com/facebookresearch/fairseq/tree/nllb there is no details on how to use the LID model (ie nllblid218e) for language ID prediction. Could this be shared please?

Thanks!

[ChocoL0rd]

Let me know when you get the answer.

[aloka-fernando]

@ChocoL0rd You can refer the below code.

#LID model
model = fasttext.load_model("/path/to/model/nllblid218e")

#predict LID
def get_lid(text):

    try:
        predictions = model.predict(text, k=1)
        lang_code = predictions[0][0].strip().split('__')[-1]
        prob = predictions[1][0]

        #return lang_code, prob
        return lang_code
    except:
        return "UNK"