mortii
commented
3 months ago @RyanMcEntire thanks for the detailed feedback!
First off, I just want to say that Korean is super weird, so it's tricky to make it work within the same general framework as other languages.
I expect the morphemizer to distill a word to something like a lemma (spacy isn't capable of doing this properly with its korean models, but that may or may not be a separate issue).
This already hits the nail on the head--if spaCy is used as the morphemizer then the lemmas aren't going to be "correct". I basically had to hack the lemmatizer to make it produce something something that conforms to other languages: https://github.com/mortii/anki-morphs/blob/2d47bac7628d913afb335db01d19493f70d03355/ankimorphs/spacy_wrapper.py#L74-L84
To get proper lemmas we would have to use a different morphemizer entirely, which may or may not exist, I don't actually know.
Id expect ankimorph to show me new words and bury variations of the same word. Just as if I were learning english, i don't need a card for walk, walking, walked, will walk, might walk, want to walk, and such for every single word.
That's understandable, however, this is a feature, not a bug. If you want to ignore all inflections after learning a lemma then you might get very confused when encountering "flew" and "flown", after learning "fly". That being said, a lot of people might prefer that approach, which is why we are adding a "treat all lemmas as known" to the new algorithm. I don't know if we should add an equivalent "skip" option before that's implemented, since it would only bury those cards for one day. We should definitely add that option at some point though.
RyanMcEntire
Describe the bug
Marking a word as known does not prevent a new inflection from appearing.
For each inflection of a root word, ankimorph treats it as a new word, and sets it as the new priority. Other inflections of the word are not suspended or marked as known.
Recalcing and changing the settings don't improve this behavior.
Steps to reproduce the behavior
Expected behavior
I expect the morphemizer to distill a word to something like a lemma (spacy isn't capable of doing this properly with its korean models, but that may or may not be a separate issue). Id expect ankimorph to show me new words and bury variations of the same word. Just as if I were learning english, i don't need a card for walk, walking, walked, will walk, might walk, want to walk, and such for every single word.
Currently, it treats each inflection as a new word, so it behaves no different than if it were being separated by spaces.
My setup
Additional context
Spacy has 3 korean models, ko_core_news_sm, ko_core_news_md, and ko_core_news_lg. They all functionally work the same way
The website states that it lemmatizes korean, and this isn't technically true. The Lemma_ value returned by spacy looks like this, with the raw word on the left and "lemma" value on the right:
The lemma isn't a lemma at all, but rather a break down of each word part, and the left-most part is only the "stem", which isn't the dictionary form of the word at all. the verb for "to eat" is 먹다, not 먹. 먹 is a rare noun for an ink stick used for making writing ink.
A proper lemma value for these would look like this, placing them in their dictionary form:
to explain with a single word, this is what spacy produces:
lemmatized properly it would look like this, where these would all be inflections of the same word:
As you can see from the frequency list generated by ankimorph, a word like 괜찮다 takes up 1034 slots on the frequency list.
The value of using a morphemizer other than spaces is basically entirely lost.