Closed flammie closed 6 years ago
This is maybe not directly related to your question, but is it a typo that "ylös" and "tuomiokirkon" are run together without space? Otherwise, it seems that "ylöstuomiokirkon" should have been represented as a multiword token. That is:
12-13 ylöstuomiokirkon - 12 ylös ylös ADV Adv 14 advmod SpaceAfter=No 13 tuomiokirkon tuomio#kirkko NOUN N Case=Gen|Number=Sing 14 nmod:poss _
ylöstuomiokirkon
is a pure typo, so I guess the way this is encoded in conllu is correct.
Yep. That's what I suspected.
@flammie - one can try tokenizing harder, but a splitting heuristic will not be totally trivial to develop without having it erroneously split plenty of valid compounds. The tokenization in the data is manually checked, so if you want to avoid these kinds of problems, use the one pre-tokenized version. Of course in real life, you will then hit these typos, etc. :)
I was just testing some udpipe-loaded omorfi with some brittle scripts and ran sanity checks on the trains but the since I used texts from the comments I ran into tokenisation alignment problems, e.g. in the beginning of the fi-ud-train.conllu I get
so everything gets out of whack. I guess my question is if it's plausible to have tool to try tokenising harder or if I should just start with the pre-existing tokenisation? I guess split heuristics would not be too hard to implement at least.