Closed proycon closed 6 years ago
(somewhat related to LanguageMachines/ucto#24 , as ucto can't deal with this either yet)
At the moment this is not possible in Frog. None of the modules. ucto, mbma, mblem etc. are capable of handling splits. Nor do we have any plans to accomplish this. The fastest (but non-trivial) way to handle splits is to post process the Frog output with PiCCL/TiCCL and then rerun Frog on the hopefully resolved tokens.
Another fix would be to have a wrapper that takes any
An even rougher fix would be to delete the spaces before giving them as input to MBLEM or MBMA ("vol daen" -> "voldaen"). This solution is brute, but makes sense, because the presence of a space within a
The rough fix will lead to a lot of problems, i guess: How do you determine which spaces are splits an which are none? You would need a lexicon then, or?
You would just treat "voldaen" as any word that you may or may not have seen before. You delete any whitespace.
The fact that there are one or more spaces within a
this assumes that there are already 'tags' or tokens detected. But what to do in running text? 'ik geloof niet dat dit werkt' 'ikgeloofnietdatditwerkt'
I'm not in favour of the brute solution of deleting spaces and sidetracking the problem that way (= information loss). I think ideally Frog should be able to handle spaces in tokens just as any other character, i.e. Frog should be completely agnostic about it and just accept whatever the tokenizer delivers (it would still be one <w>
after all). Am I right in thinking the source of this issue is that space is used as a delimiter in the underlying timbl modules, rather than tab?
I'm not sure if that is the issue - the underlying delimiter may well be the comma, and the modules may work with spaces just the same. Alternatively the spaces could be written to another character ("_") and the whole process may just work fine. Perhaps first perform a test?
I just ran into Frog indeed stumbling over spaces in tokens as foreseen :) First I thought it was another issue but it proves this issue, so I'll add this here:
frog-pos-tagger-:mismatch between number of <w> tags and the tagger result.
frog-pos-tagger-:words according to <w> tags:
frog-pos-tagger-:w[0]= ‘
frog-pos-tagger-:w[1]= AA
frog-pos-tagger-:w[2]= (
frog-pos-tagger-:w[3]= Floris
frog-pos-tagger-:w[4]= van
frog-pos-tagger-:w[5]= der
frog-pos-tagger-:w[6]= )
frog-pos-tagger-:w[7]= een
frog-pos-tagger-:w[8]= der
frog-pos-tagger-:w[9]= edelen
frog-pos-tagger-:w[10]= die
frog-pos-tagger-:w[11]= in
frog-pos-tagger-:w[12]= 1415
frog-pos-tagger-:w[13]= Jan
frog-pos-tagger-:w[14]= van
frog-pos-tagger-:w[15]= Arkel
frog-pos-tagger-:w[16]= gevankelijk
frog-pos-tagger-:w[17]= naar
frog-pos-tagger-:w[18]= 's
frog-pos-tagger-:w[19]= Hage
frog-pos-tagger-:w[20]= voerden
frog-pos-tagger-:w[21]= ,
frog-pos-tagger-:w[22]= waarvoor
frog-pos-tagger-:w[23]= zij
frog-pos-tagger-:w[24]= eene
frog-pos-tagger-:w[25]= goede
frog-pos-tagger-:w[26]= som
frog-pos-tagger-:w[27]= gelds
frog-pos-tagger-:w[28]= trokken
frog-pos-tagger-:w[29]= .
frog-pos-tagger-:w[30]= ’
frog-pos-tagger-:words according to POS tagger:
frog-pos-tagger-:word[0]='
frog-pos-tagger-:word[1]=AA
frog-pos-tagger-:word[2]=(
frog-pos-tagger-:word[3]=Floris
frog-pos-tagger-:word[4]=van
frog-pos-tagger-:word[5]=der
frog-pos-tagger-:word[6]=)
frog-pos-tagger-:word[7]=een
frog-pos-tagger-:word[8]=der
frog-pos-tagger-:word[9]=edelen
frog-pos-tagger-:word[10]=die
frog-pos-tagger-:word[11]=in
frog-pos-tagger-:word[12]=1415
frog-pos-tagger-:word[13]=Jan
frog-pos-tagger-:word[14]=steen <== THIS ONE MISSES IN THE OTHER
frog-pos-tagger-:word[15]=van
frog-pos-tagger-:word[16]=Arkel
frog-pos-tagger-:word[17]=gevankelijk
frog-pos-tagger-:word[18]=naar
frog-pos-tagger-:word[19]=de
frog-pos-tagger-:word[20]=Haag
frog-pos-tagger-:word[21]=voerden
frog-pos-tagger-:word[22]=,
frog-pos-tagger-:word[23]=waarvoor
frog-pos-tagger-:word[24]=zij
frog-pos-tagger-:word[25]=een
frog-pos-tagger-:word[26]=goede
frog-pos-tagger-:word[27]=som
frog-pos-tagger-:word[28]=geld
frog-pos-tagger-:word[29]=trokken
frog-pos-tagger-:word[30]=.
frog-pos-tagger-:word[31]='
frog-:problem frogging: aa__001biog01_01.tok.translated.folia.xml
frog-:POS tagger is confused IOB tagger is confused NER failed: '' AA ( Floris van der ) een der edelen die in 1415 Jan steen van Arkel gevankelijk naar de Haag voerden , waarvoor zij een goede som geld trokken . '' ==> ''//O AA//B-org (//O Floris//B-per van//I-per der//I-per )//I-per een//I-per der//I-per edelen//O die//O in//O 1415//O Jan//B-per steen//O van//O Arkel//B-loc gevankelijk//O naar//O de//O Haag//B-loc voerden//O ,//O waarvoor//O zij//O een//O goede//O som//O geld//O trokken//O .//O '//O '
The problem here is this word in the input document:
<w xml:id="aa__001biog01_01.TEI.2.text.body.div.p.152.s.1.w.14" class="WORD">
<t>Jan</t>
<t class="contemporary">Jan steen</t>
<lemma class="WNT:M028633.ADD.948" set="https://raw.githubusercontent.com/proycon/folia/master/setdefinitions/int_lemmaid_withcompounds.foliaset.ttl"/>
<lemma class="jan steen" set="https://raw.githubusercontent.com/proycon/folia/master/setdefinitions/int_lemmatext_withcompounds.foliaset.ttl"/>
<metric class="modernisationsource" value="inthistlexicon"/>
</w>
So the word "Jan" modernises to "Jan steen" (which is obviously odd and wrong, but not the actual issue here). Frog runs on the contemporary layer and breaks over the space (as we expected). I'll just have to disallow multiword tokens in the moderniser for now (or do an ugly patch with another delimiter like underscore), but this will at some point come back to haunt us if we build a specialised tagger/lemmatiser for Nederlab with proper multiword support and want to run Frog on its' output.
There are a lot of issues at hand here. First: In this case the 'sanity' check in Frog isn't aware of 'multiword' words. I assume this can be fixed rather easy. (the error message is also confusing, because it used the wrong textclass) But that is just a small part of the multitude of problems at hand. Reverting to the original question:
Then there is also the reverse problem of run-ons where 'voldaen' is to be split in 2 words. The tagger has no clue, the lemmatizer will have no problems, after the modifications. But how and when do we merge this knowledge?
QUICK HACK Proposal for multiple words in 1 \<w> : For tagging, we could remove the spaces, to assure that 1 FoLiA word, leads to 1 Tag.
Example:
<s id="s.1">
<w id="w.1">
<t>Een</t>
</w>
<w id="w.2">
<t>multi word</t>
</w>
<w id="w.3">
<t>test</t>
</w>
</s>
We tag this as if the second word was:
<w id="w.2">
<t>multiword</t>
</w>
The adapted MBMA and MBLEM, we can provide the text include the space.
Result:
<s xml:id="s.1">
<w xml:id="w.1">
<t>Een</t>
<pos class="LID(onbep,stan,agr)" confidence="0.981771" head="LID">
<feat class="onbep" subset="lwtype"/>
<feat class="stan" subset="naamval"/>
<feat class="agr" subset="npagr"/>
</pos>
<morphology>
<morpheme>
<t>een</t>
</morpheme>
</morphology>
<lemma class="een"/>
</w>
<w xml:id="w.2">
<t>multi word</t>
<pos class="N(soort,ev,basis,onz,stan)" confidence="0.733484" head="N">
<feat class="soort" subset="ntype"/>
<feat class="ev" subset="getal"/>
<feat class="basis" subset="graad"/>
<feat class="onz" subset="genus"/>
<feat class="stan" subset="naamval"/>
</pos>
<lemma class="multi word"/>
<morphology>
<morpheme>
<t>multi word</t>
</morpheme>
</morphology>
</w>
<w xml:id="w.3">
<t>test</t>
<pos class="N(soort,ev,basis,zijd,stan)" confidence="0.789112" head="N">
<feat class="soort" subset="ntype"/>
<feat class="ev" subset="getal"/>
<feat class="basis" subset="graad"/>
<feat class="zijd" subset="genus"/>
<feat class="stan" subset="naamval"/>
</pos>
<lemma class="test"/>
<morphology>
<morpheme>
<t>test</t>
</morpheme>
</morphology>
</w>
</s>
So for now, this simple solution is implemented: Frog accepts FoLiA with embedded spaces now. All spaces are removed for all taggers AND the parser, converting multi words into singe words.
In historical dutch, certain words may be written apart although they can be considered one token: "vol daen" (voldaan) and represented as a single
<w>
in FoLiA. Would the various Frog modules (mblem, mbpos etc) be able to deal with spaces in tokens?