search

CAMeL-Lab / camel_tools

A suite of Arabic natural language processing tools developed by the CAMeL Lab at New York University Abu Dhabi.
MIT License
409 stars 72 forks source link

[QUESTION] How to change the default Character map? #143

Open israaexol opened 6 months ago

israaexol commented 6 months ago

Hello, I'm facing an issue with the CAMEL tools analyzer, specifically with the default character map. For example, when analyzing the world "والتين" (and the fig), the disambiguator/analyzer reads the word as وآلتين (and two instruments), which is incorrect in my case.

Is there a different Character map that I can provide as a value for the norm_map argument of the analyzer? if CAMEL tools does not provide a different CharMapper, how can I define one so that the letters are processed exactly the way they are without any normalization or conversions?

I really hope you could help me solve this issue which I've been facing for a while now.

Thank you so much in advance!

owo commented 2 months ago

You can change the default CharMapper with the norm_map argument but that will not fix this particular issue. Furthermore, norm_map is used to specify the normalization expected by the morphological database so for all the databases we provide this shouldn't be changed.

This is most likely a limitation of the disambiguation model (ie. the model has seen very few instances of the word in that particular context if at all). Can you tell us which disambiguation model you are using (MLE/BERT) and can you give us the example sentence this appears in?

We are working on implementing a new option to take into account the spelling of a word in the input (particularly input diacritics) and should help with such cases.

We don't have an exact timeline for this but we'll notify you here when it's done.