Open lukavdplas opened 1 year ago
Some remarks:
We all know Frog isn't perfect, but it already knows about 880 different heden to heid lemma's, which isn't that bad.
But the Frog lemmatizer does miss some (not all) versions outside that 880
It isn't that hard to train extra lemma's into frogs datafiles. All you need is a list of
word <tab> lemma <tab> POStag
cases.
In that way you could improve rather quickly imho,
Background: The lemmatizer is trained on sentences from CGN with additions from an extra list of know lemma's
which can easily be expanded. With other missing lemmas too.
Training can be done using froggen
. Not a very difficult task, once used to it :P
If you have a list in the right format, I am willing to do this task, IFF we may use this data to add to the Frog project as a whole.
(see also the froggen manual )
Thanks! I'm not very familliar with frog myself, so I did not know that it could be retrained. This may be the best way forward for us, what do you think, @oktaal ?
Definitely. Thank you! We are going to collect a new list for Frog. You will see it some time in the future @kosloot
looking forward.In the meantime I improved on froggen a bit, to make everybody's life mor comfortable.
NB: the POS tags must be from the CGN set.
@kosloot hi! It's been a bit over a year, but here we finally have a list: https://github.com/CentreForDigitalHumanities/dutch-plurals/blob/main/output.tsv (I also have some time to look into it now again, it's been sitting there for a while). I'm not quite sure if I also need to add rows just containing singulars, so e.g. it only has rows like this:
EK's EK N(eigen,meervoud,basis)
I could easily add rows such as:
EK EK N(eigen,enkelvoud,basis)
if needed.
Words which cannot be pluralized such as "adrenaline", "marmer", "toedoen", etc only have a row such as:
adrenaline adrenaline N(basis,enkelvoud,basis)
I'm not sure if these need to be marked in some other way.
Thanks. I hope to be able to look into this "rsn". Some remarks after skimming the data:
werkenelektro-encefalograaf N(basis,meervoud,basis)
Frog is already trained with:
elektro-encefalografen elektro-encefalograaf N(soort,mv,basis)
so your entry
So maybe it is wise to reconsider this list a bit
Additional remark:; One of the lines reads:
Puerto Rico Puerto Rico N(eigen,enkelvoud,basis)
But multiword entries are NOT supported, so this entry will be skipped
It's been awhile again, but I've updated the list to no longer have multiword entries (there were just two) and modified the tags to be compliant with VanEynde's format. I wonder if I also need to add word gender?
Ok, we are getting close. But there are still a few problems
a lot of words are tagged as: N(soort,ev,basis), but vanEijnde uses a more fine-grained:
a lot of proper names are tagged as: N(eigen,ev,basis) here too, vanEijnde is more specific
BUT, the Frog tagger is trained on data where all proper names are tagged as SPEC(deeleigen)
so this is a nice shortcut I would advice.
To be clear: Whenever the Tagger tags a word as SPEC(deeleigen)
the lemmatizer will take a shortcut,
and will use the word as the lemma. Effectively ignoring the lemma assigned in de lemmata data.
one entry is ietsje ietsje N(soort,ev,dim)'
this tag is also not known, but probably N(soort,ev,dim,onz,stan) will do?
That's all folks
Ok, it is even more complex then I thought. I took a better look at the second case of N(eigen,mv,basis)
words
And not ALL of them should be (exclusively) tagged a SPEC(deeleigen)
.
There is also a range of N(soort,mv,basis)
tags among them. Words like Zuid-Molukkers
and Zwitsers
And others are to seen as an ADJ
. not as an N
.
This are cases like
Zuid-Nederlandse Zuid-Nederlands ADJ(prenom,basis,met-e,stan)
,
Zuid-Amerikaans Zuid-Amerikaaans ADJ(prenom,basis,zonder)
Zuid-Amerikaanse Zuid-Amerikaans ADJ(prenom,basis,met-e,stan)
To cite vanEynde:
Nominaal (of zelfstandig) gebruikte adjectieven worden niet als substantie- ven behandeld, maar als adjectieven.
(page 20 of my copy)
BUT: some of these cases are ambiguous, we will also need:
Zuid-Amerikaanse Zuid-Amerikaanse SPEC(deeleigen)
or
Zuid-Chinese Zuid-Chinese SPEC(deeleigen)
This is quite clumsy, but that's the historic way
There are some noticable inaccuracies in the output from the frog lemmatiser (such as
*heden
not being lemmatised to*heid
), perhaps we can improve the lemmatisation.One option is to add a different lemmatisation service that can be used instead of frog. We should investigate if there is a lemmatiser for Dutch with significantly better results.
Another option is to use some combination of the frog and alpino output for the final lemmatisation. Suggestion from @oktaal