Evaluate dicationary creation on LRL - Githubissues

janetzki / GUIDE

Create semantic domain dictionaries for low-resource languages

MIT License

4 stars 0 forks source link

Evaluate dicationary creation on LRL #10

Closed janetzki closed 1 year ago

janetzki commented 1 year ago

Goal

We can evaluate the dictionary creation on at least one LRL (and not only on HRL/MRL). Motivation: evaluate DC -> create 1 LRL dict with F1 > 30%

Tasks

[x] get LRL dictionaries (e.g., #8)
[x] compute MRR for tpi
[x] compute MRR for meu
[x] compute MRR for Daui
[ ] Add a test
[x] Check that test coverage is >= 98%
[x] Look at todos in comments
[ ] Review all changes and merge

Results

metric	value	note
MRR for tpi	0.281	642 / 5059 tpi questions selected
MRR for meu	0.230	842 / 5052 meu questions selected
MRR for swp (Daui)	0.171	1272 / 4518 swp questions selected

Observation

The MMR lowers with an increasing number of selected questions. Apparently, getting more questions right is "more difficult".