[Task Submission] Divergent DepRel Distributions (`dbca_deprel`)

[anmoisio]

Divergent DepRel Distributions

To assess NMT models' capacity to translate novel syntactical structures, we split the Europarl parallel corpus into training and testing sets with divergent distributions of the syntactical structures. We derive the data splitting method from the distribution-based compositionality assessment (DBCA) method introduced by Keysers et al. (2020). We define the atoms as the lemmas and dependency relations, and the compounds as the three-element tuples of two lemmas (the head and the dependant), and their relation, for instance (appreciate, dobj, vigilance).

Authors

• Anssi Moisio anssi.moisio@aalto.fi
• Mathias Creutz mathias.creutz@helsinki.fi
• Mikko Kurimo mikko.kurimo@aalto.fi

Implementation

This submission modifies the task.py module: evaluate_predictions() is modified to evaluate translations, and auxiliary methods are added to calculate divergences between train-test compound and atom distributions.

Usage

To evaluate generalisation, both low- and high-compound-divergence data splits should be evaluated. More data splits will be added to the final submission.

Checklist:

• [x] I and my co-authors agree that, if this PR is merged, the code will be available under the same license as the genbench_cbt repository.
• [x] Prior to submitting, I have ran the GenBench CBT test suite using the genbench-cli test-task tool.
• [x] I have read the description of what should be in the doc.md of my task, and have added the required arguments.
• [x] I have submitted or will submit an accompanying paper to the GenBench workshop.

[kazemnejad]

Thanks for submitting a task to GenBench. It seems that you're submitting the data files of the task in your PR. For the final submission, you will need to host the dataset files somewhere else (preferably as a HuggingFace dataset).

[vernadankers]

Hello!

We are getting quite close to the deadline (September 1, 11:59PM anywhere on earth), which is why I wanted to remind you of the fact that your PR still needs some attention: see Amir's message above.

Please don't forget to submit your accompanying paper to Openreview via https://openreview.net/group?id=GenBench.org/2023/Workshop by September 1.

Good luck finalising your PR and paper, feel free to tag us if you have questions. Cheers, Verna On behalf of the GenBench team

[anmoisio]

Hi GenBench team! To accommodate multiple datasets, I created a new Task Submission with subtasks (https://github.com/GenBench/genbench_cbt/pull/33), which replaces this old submission. I hope this is ok!

[vernadankers]

Yes, that's fine! Could you remove the PR that is no longer in use, please? Thanks!

[anmoisio]

Closing this old PR.