Open adno opened 1 month ago
Something very strange about this is that it wants to split s
into its own sentence at the very end of the second sentence you're giving it:
Se cree entonces que desde ese momento optaron por tener un nivel de violencia altísimo para mantener lejos a cualquier otra persona que intente volver a invadirlos
->
Se cree entonces que desde ese momento optaron por tener un nivel de violencia altísimo para mantener lejos a cualquier otra persona que intente volver a invadirlo
s
but that shouldn't be a crashing problem anyway, since it should be happy to tag a single word. So this is strange. I'll dig into it
Okay, two problems here, both of which can & should be fixed.
One is that the tokenizer doesn't know about the word invadir
, so it's trying to tokenize invadirlos
and comes up with invadirlo
and s
The second is that the MWT expander is then processing the second piece, s
, to a blank string as a hallucination in the seq2seq model.
Both are easy to fix, actually. If you come across other words like invadirlos
which cause similar problems, please post them and we can improve the tokenizer going forward.
Well, it's not crashing now in the dev
branch, as I fixed that error. Overall the fix isn't satisfactory yet, though. The tokenizer is splitting los
into lo
and s
, and it's marking s
by itself as an MWT (of only one subword).
As you find more examples of bad tokenization, please let us know, and hopefully over time we can make some progress cleaning those up.
... point being that now it seems to think s
is the sentence final punctuation here, simply because there's no .
at the very end of the document. I'll have to figure out some way to upgrade the training to make the current tokenizer work a bit better, probably with data augmentation, although long term an upgraded sentence splitter would help
Thanks you for the quick fix!
What I found out in the meantime, is that when I split the string s
from my original issue at '\n\n', the problem disappears (I get final two tokens 'invadir" and 'los'). This surprised me, as I had supposed, that Stanza splits strings at '\n\n' into separate sequences to be processed by the models, and so I thought it shouldn't have any effect. But it clearly has. I applied the splitting as a recovery from RuntimeErrors during POS tagging and I avoided crashes in the whole corpus this way.
(To be honest, I have a very dim idea of the inner workings of Stanza.)
... point being that now it seems to think s is the sentence final punctuation here, simply because there's no . at the very end of the document.
Since my corpus consists of subtitles, sentences often miss final punctuation:-(
As you find more examples of bad tokenization, please let us know, and hopefully over time we can make some progress cleaning those up.
I did a quick regex search in frequency list from my corpus (after tokenization) and found that the following strings are split in the same way ('XXXlo' + 's') if I substitute 'invadirlos' from in s
with them:
verlos
hacerlos
haberlos
atarlos
compartirlos
saberlos
decirlos
besarlos
llamarlos
tenerlos
usarlos
There's more potential culprits I have, but I haven't checked if they are results of bad tokenization of a verb + 'los'. Here's the whole list:
carlo
irlo
charlo
merlo
pirlo
verlo
mirlo
hacerlo
arlo
burlo
montecarlo
giancarlo
marlo
haberlo
karlo
atarlo
compartirlo
saberlo
serlo
amarlo
porlo
decirlo
chorlo
besarlo
chirlo
murlo
llamarlo
parlo
estarlo
oírlo
tenerlo
usarlo
lograrlo
harlo
romperlo
matarlo
conocerlo
vivirlo
waterlo
sacarlo
scantamburlo
consumirlo
sentirlo
aceptarlo
creerlo
protegerlo
amaticourlo
urlo
capturarlo
ordenarlo
conquistarlo
ayudarlo
simularlo
cobrarlo
stalkearlo
salvarlo
hacecerlo
buscarlo
distraerlo
obligarlo
sublimarlo
pagarlo
jarlo
conservarlo
subirlo
bajarlo
unificarlo
amoblarlo
overlo
poenrlo
comprarlo
ganarlo
controlarlo
corlo
oberlo
orlo
recogerlo
asaltarlo
explicarlo
escogerlo
construirlo
disfrutarlo
meterlo
barlo
cerlo
aplicarlo
trascenderlo
parecerlo
venderlo
atraparlo
devolverlo
perlo
estudiarlo
cambiarlo
heirlo
detenerlo
convencerlo
aprobarlo
acusarlo
evitarlo
firmarlo
impulsarlo
acordarlo
darlo
emocionarlo
incluirlo
...verlo
moverlo
mirarlo
captarlo
traerlo
perderlo
lmperlo
costearlo
prenderlo
transplantarlo
escucharlo
comprobarlo
pelarlo
predicarlo
arreglarlo
esquivarlo
nulificarlo
I added the 11 you put at the top. I know some of the others you listed in the second section are also verbs, but between the additional tokenization hints and a method for teaching the tokenizer that the last character isn't necessarily a punctuation, I hope it already covers the problem cases. If you find more, please don't hesitate to tell us
I went ahead and added another 20 or so infinitives, as you can see in the linked git checkin. Between that and the training update, that hopefully addresses this issue, but as I mentioned, feel free to tell us when you come across more tokenization errors for Spanish.
Describe the bug When POS tagging a specific string in Spanish a RuntimeError is reproducibly thrown without any apparent reason.
To Reproduce Steps to reproduce the behavior:
Run the following Python script (minimal example):
A RuntimeError is thrown. Complete error output:
Expected behavior No error thrown.
Environment (please complete the following information):
The error happens in pytorch preprocessing (i.e. regardless whether inference happens on GPU/CPU).
Additional context I encountered the problem only in Spanish. I was parsing a longer text, but narrowed it to this minimal example.