search

neulab / awesome-align

A neural word aligner based on multilingual BERT
https://arxiv.org/abs/2101.08231
BSD 3-Clause "New" or "Revised" License
325 stars 47 forks source link

Not able to reproduce the AERs in the table #20

Closed jinyiyang-jhu closed 3 years ago

jinyiyang-jhu commented 3 years ago

Hi, I'm new to this toolkit and trying to run a simple test with your pretrained model in the table (the last row: "Ours (multilingually fine-tuned w/ --train_co, softmax)"). I use this model as following: python tools/aer.py examples/roen.gold examples/roen.awesome-align.out the results I got is: examples/roen.awesome-align.out: 59.5% (45.3%/36.6%/5014), F-Measure: 0.405. For the other language pairs, I got "En-Fr" -> 42.9%, "Ja-En"->81.3%, "Zh-En"-> 69.0%, which are all much worse comparing to the numbers reported in your table. I tried to run the command awesome-align with *.src-tgt data under "examples/" and the same model, as guided in the README page, the results are similar (or slightly worse than) the ones shown above. Could you let me know what the issue might be?

zdou0830 commented 3 years ago

Hi, the references are one-indexed. You can try the command:

aer.py ${reference_path} ${file_path} --oneRef --fAlpha 0.5

jinyiyang-jhu commented 3 years ago

Hi, the references are one-indexed. You can try the command:

aer.py ${reference_path} ${file_path} --oneRef --fAlpha 0.5

Thanks!