[Enhancement] Add sequence-level distillation to NMT training

dmlc / gluon-nlp

NLP made easy

https://nlp.gluon.ai/

Apache License 2.0

2.56k stars 538 forks source link

[Enhancement] Add sequence-level distillation to NMT training #1463

Open sxjscience opened 3 years ago

sxjscience commented 3 years ago

Description

Add the sequence-level distillation to NMT training. This means, we draw samples from the teacher model with beam-search and train the student model with the generated samples.

References

https://arxiv.org/pdf/2006.10369.pdf
https://arxiv.org/abs/1606.07947