search

lena-voita / good-translation-wrong-in-context

This is a repository with the data and code for the ACL 2019 paper "When a Good Translation is Wrong in Context: ..." and the EMNLP 2019 paper "Context-Aware Monolingual Repair for Neural Machine Translation"
97 stars 18 forks source link

Consistency testset for Docrepair model? #5

Closed Archonsh closed 4 years ago

Archonsh commented 4 years ago

Dear Authors,

I am trying to evaluate a Docrepair model trained with your emnlp2019 data. However, I didn't find the consistency testset for the Docrepair model. The /consistency_testsets/scoring_data is from English to Russian.

Could you please hint on how to get the consistency score for the Docrepair model?

lena-voita commented 4 years ago

Dear Archonsh,

That is right - the test set is from English to Russian: since the final task is translation, test set is also for the task (not for the model).

When using DocRepair, your pipeline is the following: 1) base (context-agnostic) model generates translations of sentences, 2) these translations are fed to DocRepair, 3) the resulting DocRepair translations (or scores) are evaluated.

For this, you have to use English-Russian test sets provided in our repo.