ojwb
commented
3 years ago Note the stemming is algorithmic, not dictionary based.
If I follow you, whether this matters is dependent on how the text is word split, which is something external to the Snowball algorithms. Typically though words are split by finding spans of "word characters", which are typically letters or letters and numbers. I'd expect * and : would be treated as non word characters so would be a word break; _ is sometimes included as a word character and sometimes not depending on what's being searched. If the punctuation before innen is treated as a word break then the stemming algorithm would actually get called separately for arbeiter then innen which would produce stems arbeit (as you want) then inn.
For example, your first case is handled by the javascript demo as two words with a between so stems to arbeit and inn (see https://snowballstem.org/demo.html?text=Arbeiterinnen#German) The other two cases are intended to be handled similarly by the demo, but the regexp used to word split the text seems to not handle this as I'd expect for some reason (which is a bug in the demo I wasn't previously aware of).
Maybe it's useful to add rules to remove such suffixes for the _ case. If you think this is worth pursuing in light of the above, please can you propose a patch?
It appears that this dictionary for stemming doesn't deal properly with gender neutral word forms. In German often Texts use for example "Arbeiter*innen", "Arbeiter:innen" or "Arbeiter_innen" (aka gender gap) in order to include persons of all genders while most conservative authors just use "Arbeiter" (aka generic masculine). In my understanding this word forms should all be reduced to the same stem.