search

LanguageMachines / ucto

Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --
https://languagemachines.github.io/ucto
GNU General Public License v3.0
65 stars 13 forks source link

ucto breaks on empty FoLiA comment #71

Closed proycon closed 5 years ago

proycon commented 5 years ago

This is just a very minor issue and no priority, just noting it here for the record. Ucto breaks on an empty FoLiA comment: <comment></comment> with:

ucto: value attribute is required for comment

However, both validators do not stumble over this. It's of course arguable whether an empty comment is a valid comment, but let's not be too picky about it.

kosloot commented 5 years ago

libfolia was more picky on emtpy <comment> and <desc> nodes then foliapy. I relaxed this a bit, which should solve this.