Evaluation script for task 1 of WMT2016 QE shared task.

[kepler]

• Includes a much more efficient DeltaAvg method (the Perl version takes 16s, while this Python version takes less than 1s).
• Accepts files in either the tab-separated submission format or just with HTER scores.
• Various predicted/submitted files can be passed at once, and the script will print ordered results for both scoring and ranking metrics.

[varvara-l]

Thanks for the contribution! I'm afraid though that this script is a bit irrelevant for this particular project -- Marmot is targeted at subsentence-level QE (task 2 of WMT16), not sentence-level QE. I think this script should be more relevant for QuEst++: https://github.com/ghpaetzold/questplusplus

[kepler]

No problem. I had sent the script as a gist to Lucia and Chris just for you guys to think about using it in next year's shared task. Chris then suggested a pull request to Marmot.

Either way, I'll leave the script here: https://gist.github.com/kepler/6043a41ed8f3ed0be1e68c5942b99734

A qua, 8/06/2016, 09:52, varvara-l notifications@github.com escreveu:

Thanks for the contribution! I'm afraid though that this script is a bit irrelevant for this particular project -- Marmot is targeted at subsentence-level QE (task 2 of WMT16), not sentence-level QE. I think this script should be more relevant for QuEst++: https://github.com/ghpaetzold/questplusplus

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qe-team/marmot/pull/34#issuecomment-224528581, or mute the thread https://github.com/notifications/unsubscribe/AAEgHXDg7FR59RhPRYOQv07jdClpzWUOks5qJoKwgaJpZM4IwZS1 .

[chrishokamp]

I'm merging because our original ideas was to support all levels of qe, and the evaluation script itself doesn't break any other code. On Jun 8, 2016 2:04 AM, "Fabio Natanael Kepler" notifications@github.com wrote:

No problem. I had sent the script as a gist to Lucia and Chris just for you guys to think about using it in next year's shared task. Chris then suggested a pull request to Marmot.

Either way, I'll leave the script here: https://gist.github.com/kepler/6043a41ed8f3ed0be1e68c5942b99734

A qua, 8/06/2016, 09:52, varvara-l notifications@github.com escreveu:

Thanks for the contribution! I'm afraid though that this script is a bit irrelevant for this particular project -- Marmot is targeted at subsentence-level QE (task 2 of WMT16), not sentence-level QE. I think this script should be more relevant for QuEst++: https://github.com/ghpaetzold/questplusplus

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qe-team/marmot/pull/34#issuecomment-224528581, or mute the thread < https://github.com/notifications/unsubscribe/AAEgHXDg7FR59RhPRYOQv07jdClpzWUOks5qJoKwgaJpZM4IwZS1

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/qe-team/marmot/pull/34#issuecomment-224531457, or mute the thread https://github.com/notifications/unsubscribe/ABicP8LBwijwbaRhDdP9yas729Lp_Vk4ks5qJoWIgaJpZM4IwZS1 .