Open kosloot opened 3 years ago
I can reproduce with a much smaller file too: buggy2.txt
I'm not sure whether the example file is a real-world example or just an example to illustrate this issue, but I see two problems:
I'm still a bit puzzled what caused the actual randomness though..
Actually, I think I the randomness may have been caused by the unordered map from which text classes were read, that order was not guaranteed and behaviour differed based on which was checked first..
Ok, buggy2.txt was just a made-up excerpt. And the use of class-names is questionable, but still: I think it is valid FoLiA, and foliavalidator should be deterministic
Yes, the non-deterministic quirk is definitely not good. I did commit a fix for that.
But the fix isn't released, so I wonder if that helps @martinreynaert using LaMachine?
released now
I am quite sure this is NOT solved completely yet. Given this file: incorrect.txt I really think foliavalidator is wrong here:
foliavalidator incorrect.txt VALIDATION ERROR on full parse by library (stage 2/3), in incorrect.txt ParseError: FoLiA exception in handling of
@ line 35 (in parent
@ parent line 34) : [InconsistentText] Text for <Sentence at 139728228001272 id=KB.DDD17C.NicolineCrowd.73272.p.1.s.3 set=None class=None>, is inconsistent: EXPECTED (deep text after normalization) *> Malvasie , > BUT FOUND (strict text after normalization) > Malvasie komende, *** DEVIATION POINT: Malvasie <HERE>komende, (also checked against older rules prior to FoLiA v2.4.1)
Addition:
I tried to rename the <new>
node to <current>
. Which I thought might solve the problem, but then:
> foliavalidator notsilly.xml
VALIDATION ERROR on full parse by library (stage 2/3), in notsilly.xml
ParseError: FoLiA exception in handling of <correction> @ line 43 (in parent <s> @ parent line 36) : [Exception] Can't add
Original item to Correction if there is a Current item
Which puzzles me too.
I really think foliavalidator is wrong here
Agreed, the document looks okay, the validator seems to miss a word entirely when getting the deep text..
I tried to rename the
node to . Which I thought might solve the problem, but then: Original item to Correction if there is a Current item
That's a legit error, Original can only come with New, not with Current (Current (and New) can come with Suggestion).
That's a legit error, Original can only come with New, not with Current (Current (and New) can come with Suggestion).
Ok, I will add this limitation to libfolia too.
There was already a failing test case I missed that covers this issue actually, the release was a bit too premature...
Ok, I will add this limitation to libfolia too.
Done. Needed quite a bit of rework, unfortunately
This should now be solved, the tests are all green again and the example you gave also validates. Has been released as foliapy v2.5.3.
Still, I assume something is wrong. see: special.txt
Here the (unlogical) correction from class="current" to class="Ticcl" is made. I still assume the FoLiA is correct, but foliavalidator rejects it. folialint accepts this construction.
yes, you're right, there's still some assumption that the "current" class is always the latest/most-current class. I'll see if we can get rid of that without breaking anything else.
I'm not sure about the Python implementation, but libfolia checks the text for every textclass in the document. So in this case both the implicit "current" and "Ticcl".
yeah, foliapy does the same
given this file: buggy.txt
foliavalidator and folia2text give random outcome: