Adds the implementation of Neural vocabulary selection to Sockeye as presented in our NAACL 2022 paper "The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation" (Tobias Domhan, Eva Hasler, Ke Tran, Sony Trenous, Bill Byrne and Felix Hieber).
Pull Request Checklist
[x] Changes are complete (if posting work-in-progress code, prefix your pull request title with '[WIP]'
until you can check this box.
[x] Unit tests pass (pytest)
[x] Were system tests modified? If so did you run these at least 5 times to account for the variation across runs?
[x] System tests pass (pytest test/system)
[x] You have considered writing a test
[x] Passed code style checking (./style-check.sh)
[x] Updated major/minor version in sockeye/__init__.py. Major version bump if this is a backwards incompatible change.
[x] Updated CHANGELOG.md
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Adds the implementation of Neural vocabulary selection to Sockeye as presented in our NAACL 2022 paper "The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation" (Tobias Domhan, Eva Hasler, Ke Tran, Sony Trenous, Bill Byrne and Felix Hieber).
Pull Request Checklist
pytest)pytest test/system)./style-check.sh)sockeye/__init__.py. Major version bump if this is a backwards incompatible change.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.