ES-CT: linguistics annotations

matyaskopp commented 1 year ago

@rjzevallos, I compared TEI and TEI.ana versions of one file. I don't know how complicated it is to fix these issues - I will be delighted if they are fixed or at least documented because these bugs can decrease corpus usability.

Syntactic words and join collision

[ ] syntactic words + join right

TEI version:

<seg xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8" xml:lang="ca"><!-- 
... --> Hi havia, com vostè ha recordat, el segon tripartit. <!-- ... --></seg>

TEI.ana version:

<seg xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8" xml:lang="ca">
<!-- ... -->
    <s xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3" xml:lang="ca">
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.1" msd="UPosTag=PRON|PronType=Prs|Person=3" lemma="hi">Hi</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.2" msd="UPosTag=VERB|Mood=Ind|Tense=Imp|Person=3|Number=Sing" lemma="heure" join="right">havia</w>
        <pc xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.3" msd="UPosTag=PUNCT|PunctType=Comm">,</pc>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.4" msd="UPosTag=SCONJ" lemma="com">com</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.5" msd="UPosTag=PRON|PronType=Prs|Person=2|Number=Sing|Polite=Form" lemma="vostè">vostè</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.6" msd="UPosTag=AUX|Mood=Ind|Tense=Pres|Person=3|Number=Sing" lemma="haver">ha</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.7" msd="UPosTag=VERB|Mood=Par|Number=Sing|Gender=Masc" lemma="recordar" join="right">recordat</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.10">
           recordatel
           <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.8" msd="UPosTag=PUNCT|PunctType=Comm" norm="," lemma=","/>
           <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.9" msd="UPosTag=DET|PronType=Art|Gender=Masc|Number=Sing" norm="el" lemma="el"/>
        </w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.11" msd="UPosTag=NOUN|Gender=Masc|Number=Sing" lemma="segon">segon</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.12" msd="UPosTag=VERB|Mood=Par|Number=Sing|Gender=Masc" lemma="tripartir" join="right">tripartit</w>
        <pc xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.8.3.13" msd="UPosTag=PUNCT|PunctType=Peri">.</pc>
        <linkGrp type="UD-SYN" targFunc="head argument">
          <!-- ... -->
        </linkGrp>
    </s>
<!-- ... -->
</seg>

TEI version contains sentence:

Hi havia, com vostè ha recordat, el segon tripartit.

but TEI.ana version contains different a sentence:

Hi havia, com vostè ha recordatrecordatel segon tripartit.

Missing UD features in named entities, wrong UPosTag

[ ] UD features in NE
[ ] UPosTag in NE

<name type="MISC">
   <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.28" msd="UPosTag=PROPN" lemma="reglament">Reglament</w>
   <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.29" msd="UPosTag=PROPN" lemma="de">de</w>
   <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.30" msd="UPosTag=PROPN" lemma="el">el</w>
   <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.31" msd="UPosTag=PROPN" lemma="parlament">Parlament</w>
   <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.32" msd="UPosTag=PROPN" lemma="de">de</w>
   <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.33" msd="UPosTag=PROPN" lemma="catalunya">Catalunya</w>
</name>

No syntactic words in named entities + missing join right

[ ] syntactic words and NEs
[ ] join right in NEs

TEI version:

               <seg xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3" xml:lang="ca"><!-- ... --> Reglament del Parlament de Catalunya.</seg>

TEI.ana version:

<seg xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3" xml:lang="ca">
    <s xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1" xml:lang="ca">
        <name type="MISC">
           <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.28" msd="UPosTag=PROPN" lemma="reglament">Reglament</w>
           <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.29" msd="UPosTag=PROPN" lemma="de">de</w>
           <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.30" msd="UPosTag=PROPN" lemma="el">el</w>
           <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.31" msd="UPosTag=PROPN" lemma="parlament">Parlament</w>
           <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.32" msd="UPosTag=PROPN" lemma="de">de</w>
           <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.33" msd="UPosTag=PROPN" lemma="catalunya">Catalunya</w>
        </name>
        <pc xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.3.1.34" msd="UPosTag=PUNCT|PunctType=Peri">.</pc>
    </s>
</seg>

TEI version contains:

Reglament del Parlament de Catalunya.

but TEI.ana version contains:

Reglament de el Parlament de Catalunya .

Missing join right in articles

[ ] articles + join

TEI:

<seg xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.1" xml:lang="ca">D’acord amb l’article 146

TEI.ana

<seg xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.1" xml:lang="ca">
    <s xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.1.1" xml:lang="ca">
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.1.1.1" msd="UPosTag=ADP|AdpType=Prep" lemma="de">D'</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.1.1.2" msd="UPosTag=NOUN|Gender=Masc|Number=Sing" lemma="acord">acord</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.1.1.3" msd="UPosTag=ADP|AdpType=Prep" lemma="amb">amb</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.1.1.4" msd="UPosTag=DET|PronType=Art|Number=Sing" lemma="el">l'</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.1.1.5" msd="UPosTag=NOUN|Gender=Masc|Number=Sing" lemma="article">article</w>
        <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.1.0.1.1.6" msd="UPosTag=NUM" lemma="146">146</w>

TEI version contains:

D’acord amb l’article 146

but TEI.ana version contains:

D’ acord amb l’ article 146

Syntactic words at the beginning of sentence ???

[ ] Syntactic words at the beginning of sentence

TEI:

<seg xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25" xml:lang="ca"><!-- 
... --> Dels tribunals ordinaris de la justícia catalana. <!-- ... --></seg>

TEI.ana:

<seg xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25" xml:lang="ca">
<!-- ... --> 
    <s xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5" xml:lang="ca">
       <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5.1" msd="UPosTag=ADP|AdpType=Prep" lemma="de">De</w>
       <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5.2" msd="UPosTag=DET|PronType=Art|Gender=Masc|Number=Plur" lemma="el">els</w>
       <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5.3" msd="UPosTag=NOUN|Gender=Masc|Number=Plur" lemma="tribunal">tribunals</w>
       <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5.4" msd="UPosTag=ADJ|Gender=Masc|Number=Plur" lemma="ordinari">ordinaris</w>
       <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5.5" msd="UPosTag=ADP|AdpType=Prep" lemma="de">de</w>
       <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5.6" msd="UPosTag=DET|PronType=Art|Gender=Fem|Number=Sing" lemma="el">la</w>
       <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5.7" msd="UPosTag=NOUN|Number=Sing" lemma="justícia">justícia</w>
       <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5.8" msd="UPosTag=ADJ|Gender=Fem|Number=Sing" lemma="català" join="right">catalana</w>
       <pc xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.25.5.9" msd="UPosTag=PUNCT|PunctType=Peri">.</pc>
<!-- ... -->
    </s>
<!-- ... -->
</seg>

TEI version contains:

Dels tribunals ordinaris de la justícia catalana.

but TEI.ana version contains:

De els tribunals ordinaris de la justícia catalana.

misplaced join right

[ ] misplaced join right

TEI:

... fixin-s’hi en ...

TEI.ana

    <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.37.2.67">
    fixin-s
    <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.37.2.65" msd="UPosTag=VERB|Mood=Sub|Tense=Pres|Person=3|Number=Plur" norm="fixin" lemma="fixar"/>
    <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.37.2.66" msd="UPosTag=PRON" norm="-s" lemma="es"/>
    </w>
    <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.37.2.68" msd="UPosTag=PRON|PronType=Prs|Person=3" lemma="hi" join="right">'hi</w>
    <w xml:id="ParlaMint-ES-CT_2015-11-10-0302.4.0.37.2.69" msd="UPosTag=ADP|AdpType=Prep" lemma="en">en</w>

TEI: fixin-s’hi en vs TEI.ana: fixin-s ’hien

nuriabel commented 1 year ago

Dear Matyás, Thanks a lot for your careful analysis. Yes, we are going to correct most of the errors you found, but for the Named Entities analysis. We will keep you updated. Best regards N.

El mié, 26 abr 2023 a las 10:20, Matyáš Kopp @.***>) escribió:

Assigned #639 https://github.com/clarin-eric/ParlaMint/issues/639 to @nuriabel https://github.com/nuriabel.

— Reply to this email directly, view it on GitHub https://github.com/clarin-eric/ParlaMint/issues/639#event-9100435882, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGFJPVYMD53FHSTV55CJFQLXDDLGHANCNFSM6AAAAAAXMCOQFM . You are receiving this because you were assigned.Message ID: @.***>

maartenpt commented 9 months ago

contraction seem not to have been dealt with properly in general, not only at the beginning of a word - take the word "del", which is never split into de+el, and has a wide range of upos tags:

PARLAMINT-31-PARLAMINT-ES-CT> Matches = [form="del"];
PARLAMINT-31-PARLAMINT-ES-CT> group Matches match upos;
#---------------------------------------------------------------------
(all)                         NOUN                               95942
                              ADJ                                12833
                              PUNCT                              10153
                              ADP                                 9990
                              VERB                                8936
                              ADV                                 8071
                              PROPN                               8013
                              NUM                                 4084
                              CCONJ                               3954
                              DET                                 2099
                              AUX                                 1382
                              PRON                                 915
                              SCONJ                                308
                              INTJ                                   1

matyaskopp commented 9 months ago

[ ] 100_per_cent in annotated data

<seg xml:id="ParlaMint-ES-CT_2020-10-22-6402.4.0.2" xml:lang="ca">Les mesures efectives són: política de garantia de rendes, amb vies d'incidència directa en els infants; més pressupost –o, més aviat, més inversió– en polítiques d'infància, i indicadors i avaluació per garantir que les mesures i els recursos que s'hi destinin siguin efectius. Perquè també ja fa dos anys els deia, amb dades de l'Idescat, que les polítiques catalanes en aquest tema són molt ineficients. Les transferències socials només estaven aconseguint reduir un 15 per cent el risc de pobresa en la població menor de divuit anys a Catalunya. Amb dades del 2018. Una xifra que és ridícula, si la comparem amb què passa amb altres sectors de la població. Entre les persones de més de seixanta-cinc anys les transferències socials fan reduir un 82 per cent el risc de pobresa. Només un 15 per cent, en el cas dels infants; un 82 per cent en el cas de les persones grans.</seg>

annotated version:

<seg xml:id="ParlaMint-ES-CT_2020-10-22-6402.4.0.2" xml:lang="ca">
<!-- ... -->
                  <s xml:id="ParlaMint-ES-CT_2020-10-22-6402.4.0.2.7" xml:lang="ca">
<!-- ... -->
<w xml:id="ParlaMint-ES-CT_2020-10-22-6402.4.0.2.7.3" msd="UPosTag=NUM" lemma="15/100" join="right">15_per_cent</w>
<!-- ... -->
<w xml:id="ParlaMint-ES-CT_2020-10-22-6402.4.0.2.7.14" msd="UPosTag=NUM" lemma="82/100">82_per_cent</w>
</seg>

I understand that you have probably use it to fix wrong tokenization, but you forget to remove underscores _

same issue different unit:

clarin-eric / ParlaMint