value of Textcontent dissappears (empty string) upon add?

[proycon]

Something goes wrong when I add TextContent with value eologico*phijsico*metaphijsicum, libfolia adds an empty text content element instead! I've no idea what triggers this (special meaning for the asterisk perhaps??), other words process fine.

I add TextContent as follows: https://github.com/LanguageMachines/foliautils/blob/wordtranslate/src/FoLiA-wordtranslate.cxx#L134

Debug output, I explicitly check if I'm not passing an empty string (after trimming even):

$ FoLiA-wordtranslate --outputclass contemporary -d lexicon.1637-2010.250.lexserv.vandale.tsv -p preservation2010.txt -r rules.machine aa__001biog01_01.tok.folia.xml

Loading dictionary...                                                                                                                                                                                                                                       
Loading preserve lexicon...                                                                                                                                                                                                                                 
Loading rules...                                                                                                                                                                                                                                            
DEBUG: target before sanity check 'eologico*phijsico*metaphijsicum'                                                                                                                                                                                         
DEBUG: target after sanity check 'eologico*phijsico*metaphijsicum'
DEBUG: text after adding textcontent ''
finished aa__001biog01_01.tok.folia.xml 

[proycon]

Now I can't reproduce the above debug anymore (text content shows fine), but the serialisation to xml still has an empty text..

DEBUG: -- BEFORE APPEND --                                                                                                                                                                                                                                  
DEBUG: target: 'eologico*phijsico*metaphijsicum'
DEBUG: after unicode encoding and decoding 'eologico*phijsico*metaphijsicum'
DEBUG: from textcontent 'eologico*phijsico*metaphijsicum'
DEBUG: length from textcontent  33
DEBUG: -- AFTER APPEND -- 
DEBUG: from textcontent 'eologico*phijsico*metaphijsicum'
DEBUG: length from textcontent  33
DEBUG: from word 'eologico*phijsico*metaphijsicum'

          <w xml:id="aa__001biog01_01.TEI.2.text.body.div.p.10802.s.1.w.2" class="WORD-COMPOUND">
            <t>theologico-physico-metaphysicum</t>
            <t class="contemporary"></t>
            <metric class="modernisationsource" value="rules"/>

[proycon]

Ok, the following debug shows the problem, still no idea why though:

DEBUG: target: 'eologico*phijsico*metaphijsicum'                                                                                                                                                                                                            
DEBUG: from textcontent 'eologico*phijsico*metaphijsicum'
DEBUG: length from textcontent  33
DEBUG: from word 'eologico*phijsico*metaphijsicum'
DEBUG: XML serialisation '<w xmlns="http://ilk.uvt.nl/folia" xml:id="aa__001biog01_01.TEI.2.text.body.div.p.10802.s.1.w.2" class="WORD-COMPOUND"><t>theologico-physico-metaphysicum</t><t class="contemporary"></t></w>'  

Debug code is committed: https://github.com/LanguageMachines/foliautils/blob/wordtranslate/src/FoLiA-wordtranslate.cxx#L143

[proycon]

It also fails on the following other input words, which probably get mangled to asteriskses too by my tool (incorrectly but that's not the issue here):

• Onderwierum-en-Westerdijkshorn
• Hollandsch-Hoogduitsch-Israelitische
• Arrondissements-kiescollegie
• Kollumerland-en-Nieuw-Kruisland

[proycon]

Hah, there seem to be two 0x00 bytes in front of the string! That would explain things. I should have counted the characters better :)

[proycon]

Conclusion: So this seems to happen if there are invalid characters in the string, I think it would be helpful if this could be caught and a warning outputted when appending text, provided it's not too expensive.

[kosloot]

checking for an string to be valid UTF8 is quite expensive. A 0 is even not 'that invalid'. It is a C string terminator, yielding 'empty' strings when in front. I didn't find an easy way to check validity.

[kosloot]

Ok. the problem occurred in a program that incorrectly used the libicu API, yielding iinvalid Unicode strings. That is a quality of implementation problem in libicu. Not in libfolia.