Closed xavier-taylor closed 10 months ago
Hi @xavier-taylor , Thanks for letting us know. Our lemmatizer is a neural-based model trained on text corpus so the lemmatization results might vary depending on context. This is quite unavoidable unless we have another text corpus with higher quality of annotations for training.
Input:
` p = Pipeline('indonesian', embedding='xlm-roberta-large')
print(p('Ia menjadi Gubernur Bali menggantikan Anak Agung Bagus Sutedja.')) `
Output:
Expected output:
Words like 'mengantikan' to be reduced to their lemma.
For example, here is the lemmatization from aksara: 1 Ia ia PRON Number=Sing|Person=3|PronType=Prs 2 menjadi jadi VERB Voice=Act 3 Gubernur Gubernur PROPN 4 Bali Bali PROPN 5 menggantikan ganti VERB Voice=Act 6 Anak Anak PROPN 7 Agung Agung PROPN 8 Bagus Bagus PROPN 9 Sutedja Sutedja PROPN SpaceAfter=No 10 . . PUNCT
Note how menjadi becomes jadi and menggantikan becomes ganti.
I think this issue is related to the UD GSD data.
I have seen a similar issue in stanza: https://github.com/stanfordnlp/stanza/issues/1003
Thanks for releasing this great tool.
(Ubuntu, python3.9, trankit 1.1.0)