AngledLuffa
commented
1 month ago Can confirm this is a problem. Thank you for reporting.
The core of this problem is the expectation that a document ends in a sentence final punctuation, and it's going to make a sentence final punctuation even if that makes no sense.
I tried to add a mechanism where the tokenizer would sometimes drop the final punctuation at the end of a document. It seemed to help with certain clitic pronouns, but clearly there are cases for which this does not fix the problem.
One thing we can do to ameliorate this is to add the words in question as fake "sentences" specifically to the tokenizer training data. Can you confirm that the words which have endings that resemble a clitic - me, lo, la, le - are not verbs? anseriforme for example seems to be a duck, carambolo is a starfruit, but I'm far from a Spanish expert
adno
Describe the bug Single words tend to be over-segmented in Spanish, e.g. "abundoso" is being split as "abundos" (noun) + "o" (conjunction), if it's the only input for the Spanish
tokenize,mwtpipeline. What's particualrly problematic, is that the first token is typically a non-existing word.To Reproduce Run the following code:
Expected behavior All are single words and should therefore be segmented in one token. As a result we should get this output:
Counter({1: 394}).Environment (please complete the following information): OS: Linux and MacOS (reproducible on both) Python version: python 3.11.9 (hb806964_0_cpython conda-forge) Stanza version: current (dev) torch: 2.3.1
Additional context I'm doing an NLP experiment where I need to tokenize/lemmatize words without context. The data are from a psycholinguistic task, where no context was provided. I've found that adding a period to the words would work as a workaround for most of them, but I believe the tokenizer should work reasonably words for single words as well. (Exceptions from the words listed, where adding a period doesn't help, are 'estruendoso', 'fachendoso', 'hacendoso'.)
This issue affects other single words too, but with adjectives ending in "-oso", it seems very prominent and consistent. Other susceptible words often end with -lo (címbalo, crocodilo), -eo (machaqueo, maniqueo), -la (garla, hortícola), -le (diástole), -me (cuneiforme, adarme), -sa (mayonesa, galactosa). Again the first resulting token is typically a non-existing word (though there are some exceptions, e.g. "machaque" + "o"). Here is a longer list of examples, where the resulting tokens seem to contain a non-word (or at least a very rare word), sorted by the last two characters: