How to evaluate the wmt17 English-Chinese model?

[chynphh]

Hi! Thanks for your nice code! When I evaluated the wmt17 English-Chinese, I only get 19.14(DisCo + Mask-Predict, step4)

My script: CUDA_VISIBLE_DEVICES=0 python generate_disco.py ${data_path} --path ${model_dir}/checkpoint_top5_average.pt \ --task translation_self --max-sentences 10 --remove-bpe --decoding-iterations 4 \ --decoding-strategy mask_predict --length-beam 5 --sacrebleu

[jungokasai]

I haven't used the --sacrebleu in the fairseq codebase. Could you try running the sacrebleu library directly like this?

pip install sacrebleu
cat <PAHT_TO_OUTPUT> | grep -P ‘^H’ | cut -c3- | sort -n -k 1 |uniq | cut -f 2 > out
# extract H hypotheses to out
cat out | sacrebleu -l en-zh  -t wmt17

[chynphh]

I haven't used the --sacrebleu in the fairseq codebase. Could you try running the sacrebleu library directly like this?

pip install sacrebleu
cat <PAHT_TO_OUTPUT> | grep -P ‘^H’ | cut -c3- | sort -n -k 1 |uniq | cut -f 2 > out
# extract H hypotheses to out
cat out | sacrebleu -l en-zh  -t wmt17

It works! Thanks！