Implement better on-the-fly validation for Text

[kosloot]

See https://github.com/proycon/folia/issues/24 libfolia should be more picky on invalid constructions.

[kosloot]

basis checking is implemented: (optional)

• on parsing
• on creation of a TextContent with settext()

using low-level routines it is still possible to construct invalid FoLiA. needs some work still. Maybe check on write too?

[kosloot]

To summarize: checking text has 2 important directions:

• 1. bottom up: given low lever text (words,sentences,) create higher level text (sentences, paragraphs,..) This is rather easy, and is taken care off in the library when using settext()
• 2. Top down: this is for instance the case in a tokenizer. Given a high level text (paragraph,sentence) create lower level text (paragraph, words) that when combined yields the higher level again.

So the first problem is solved. The second is very hard though: You could argue that this a quality problem of the tokenizer. But creating invalid FoLiA and not detecting that (directly) is undesirable. The library should not allow this. And to pinpoint the problem, it should detect that as soon as possible. But that is impossible to achieve: when adding words recursively to an already existing sentence, the 'correct' reading is reached only after the last word is added. What to do then?

• add a 'checktext()' function and hope that the tokenizer uses that after the last addition?
• add an variant of settext() that assures that the added subtext is a part of the higher text? (this could automagicly add offset then!) But this doesn't enforce complete correctness.
• check on writing of the tree. Also costly, and you get not directly 'punished' .
• what to do then?

[kosloot]

Ok, for a start I implemented the scenario that checks if an added text is a substring of the parent text. For Words is must be an exact substring. For other elements this is relaxed a bit by first removing whitespace totally. This needs to be done because text can have embedded newlines, and for instance the ucto tokenizer is unaware of these.

[kosloot]

OK, that was a bad Bad BAD idea ALL substrings must be (part-of) the parent strings. If the parent has embedded newlines, than that MUST be reflected in
nodes. That isn't easy fixed in libfolia, but can be done in the programs build upon libfolia. Like ucto does now after latest patch. The relaxed check is removed again

[kosloot]

At the end we stick to the 'normalized' string checking, as @proycon suggested in https://github.com/proycon/folia/issues/24 This is implemented now for folia documents with version 1.5 or higher. (which don't exists in the wild yet)

[kosloot]

closing. as this is implemented now.