lemmatization

[ctschroeder]

While editing the corpus I See Your Eagerness, I noticed the following:

• ⲉⲧⲃⲏⲧ doesn't (but should) lemmatize to ⲉⲧⲃⲉ
• ⲉⲙ doesn't (but should) lemmatize to ⲛ
• ⲑⲩⲥⲥⲁⲥⲧⲏⲣⲓⲟⲛ doesn't (but should) lemmatize to ⲑⲩⲥⲓⲁⲥⲧⲏⲣⲓⲟⲛ (maybe normalize too?)
• ⲛⲏⲥⲧⲓⲁ should be ⲛⲏⲥⲧⲉⲓⲁ (otherwise we have two lemmas for the same word)
• lemma for ⲛⲛ ANEGOPT should probably be ⲛⲛⲉ (the more frequent version)
• We currently have ⲛⲛⲁϩⲣⲛ & ⲛⲁϩⲣⲛ as lemmas in existing corpora https://corpling.uis.georgetown.edu/annis/?id=cd66dbf1-8a27-4d88-a393-caba7536efb7 We should decide on one. Crum lists ⲛⲁϩⲣⲛ (and tells people to see ϩⲟ); no entry for ⲛⲛⲁϩⲣⲛ. So I gave everything in this corpus ⲛⲁϩⲣⲛ
• ⲙⲙⲛ should be lemmatized ⲙⲛ not ⲟⲩⲛ
• ⲣⲙ needs to be lemmatized to ⲣⲱⲙⲉ
• ⲥⲟⲧϥ should POS as VSTAT and lemma as ⲥⲱⲧϥ
• ⲧⲁⲙⲉⲓⲟⲛ should lemmatize as ⲧⲁⲙⲓⲟⲛ or vice versa
• ⲧⲣ ACAUS should lemmatize to ⲧⲣⲉ
• ϩⲓⲉⲓⲃ and its spelling variants need to be added to lemmatizer
• ϫⲉⲣⲉ should lemmatize as ϫⲉⲣⲟ
• ϭⲟⲟⲙⲉ should POS as VSTAT and lemmatize as ϭⲱⲱⲙⲉ
• ϭⲟⲟⲩϭ should POS as VSTAT and lemmatize as ϭⲟⲱⲩϭ
• ⲁⲣⲁ should lang as Greek
• do we have an "unknown" for lemma? We currently have some strange or nonexistent lemmas when the POS is "unknown"

[amir-zeldes]

A lot of these are actually normalization issues - I've updated the normalizer and these should hopefully be working now. The ones that are lemma problems (missing VSTATs for example) are now in the tagger lexicon but won't update until the next tagger release, which is planned.

Other misc. answers:

• ⲣⲙ is a morph and doesn't get lemmatized (lemmas are at norm level)
• I think there is a difference between ⲛⲁϩⲣⲛ and ⲛⲛⲁϩⲣⲛ. I would make the latter be two norms (ⲛ + ⲛⲁϩⲣⲛ), like a double prepositions (as in English "X out of Y"). You can see it's originally a noun, but maybe synchronically two PREPs make more sense here. Open to discussion though.
• can't reproduce: (maybe there was manual intervention here?)
  • ⲙⲛ lemmatizing to ⲟⲩⲛ
  • ⲁⲣⲁ not getting tagged as Greek

[ctschroeder]

Thank you so much!

The ones you can't reproduce may be due to an earlier version of the tagger and lemmatizer. I think some of these pre-date the DDGLC list and other updates.

I'm flexible on ⲛⲁϩⲣⲛ and ⲛⲛⲁϩⲣⲛ. I also wondered if ⲛⲛⲁϩⲣⲛ should be two units but saw both as their own lemmas in our existing corpora and became confused. If we go to ⲛⲁϩⲣⲛ only as both lemma and normalized unit (and as preposition), that is fine with me; we just need to be sure to update tokenizer, pos tagger, and lemmatizer right? Should I make a separate issue?

[ctschroeder]

@amir-zeldes what did we decide on ⲛⲁϩⲣⲛ and ⲛⲛⲁϩⲣⲛ?

[amir-zeldes]

Sorry for dropping the ball on this - I tried and saw that two PREPs really stands out in the tagger preps so maybe it's better to avoid it. On the other hand, I still feel strongly that nahrn is not the same as nnahrn. So maybe let's make them two distinct norms and lemmas, but in both cases just one norm. What do you think, should we morph it as two units n+nahrn?

[ctschroeder]

I am not sure this is really a morph, though, which is why I'm hesitant. Crum lists ⲛⲛⲁϩⲣⲛ as ⲛⲁϩⲣⲛ in older manuscripts. But if you feel strongly that they are different, then we should have some way to distinguish. I'm not sure the morph layer is right. Can we check with Eitan? or maybe Eitan and Sebastian both? It looks like only ⲛⲁϩⲣⲛ appears in the online lexicon. I think my preference is either treat nn... as two prepositions OR as a regular spelling variant (=> nn... not normalized to one n but both forms lemmatized to the same lemma). But if you feel strongly that it's not a variant, and the linguistic community thinks the extra n is not a preposition, then we should probably do as you suggest and make it a morph. Let's consult though?

[amir-zeldes]

Sorry for the slow reply - yes, let's ask Eitan. I'll ping him.

[ctschroeder]

Did we ever get an assessment of ⲛⲛⲁϩⲣⲛ from Eitan?

[amir-zeldes]

Not yet, I can check back