lemmatization of Greek loan words

[ctschroeder]

I'm noticing some inconsistencies, such as here with the verb metanoei. What is the best way to edit the lemmatizer?

[ctschroeder]

(adding an issue, because we may want to say something in the Lemmatization guidelines about i/ei in addition to fixing specific words)

[ctschroeder]

See also lemma=/ⲟⲩⲟⲓ/|/ⲟⲩⲟⲉⲓ/

[amir-zeldes]

Thanks for catching these! This is actually not a lemmatization issue, but rather a normalization issue. Even if we know that the standard lemma is metanoei, we'd also like the norm to read the same. The lemmatizer will generally guess for an unknown form that its lemma is the same as its norm, so it will continue to postulate a lemma metanoi next to metanoei. What we'd really like is for the normalizer to catch this.

I've updated these two items in the normalizer here: CopticScriptorium/normalizer@40eab033a3625ea8bf8a8ac40549cde8de2e1953

My sense is that the desirable forms are:

• ouoi (it's that way in the lemma list, for example); generally I take 'standard' Sahidic Coptic to spell diphthongs in -i with a plain iota (like 'tai')
• This might lead us to choose 'metanoi' as the form, but since it's a Greek word I'm inclined to take its standard Greek form, which is 'metanoei' (after contraction). The form 'metanoi' doesn't match any analysis in the Perseus morphology tool, for example, and linkability with Perseus is also a factor when making decisions about normalization IMO.

[amir-zeldes]

Oh, and the place to update is therefore the normalizer's norm_table.tab