[WIP][AMP] AMP for machine translation

[sxjscience]

Description

• [ ] Add amp for machine translation
• [ ] Fix horovod training code.

Checklist

Essentials

• [ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• [ ] Changes are complete (i.e. I finished coding on this PR)
• [ ] All changes have test coverage
• [ ] Code is well-documented

Changes

• [ ] Feature1, tests, (and when applicable, API doc)
• [ ] Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

cc @dmlc/gluon-nlp-team

[github-actions[bot]]

The documentation website for preview: http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR1443/nmt_amp/index.html

[codecov[bot]]

Codecov Report

Merging #1443 (5b282f6) into master (f5c5b73) will decrease coverage by 4.08%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1443      +/-   ##
==========================================
- Coverage   85.37%   81.29%   -4.09%     
==========================================
  Files          53       53              
  Lines        6987     6986       -1     
==========================================
- Hits         5965     5679     -286     
- Misses       1022     1307     +285     

Impacted Files	Coverage Δ
src/gluonnlp/layers.py	87.11% <ø> (-0.04%)	:arrow_down:
src/gluonnlp/utils/testing.py	58.33% <0.00%> (-35.84%)	:arrow_down:
src/gluonnlp/models/xlmr.py	58.82% <0.00%> (-29.42%)	:arrow_down:
src/gluonnlp/models/__init__.py	68.75% <0.00%> (-28.13%)	:arrow_down:
src/gluonnlp/models/electra.py	58.47% <0.00%> (-18.37%)	:arrow_down:
src/gluonnlp/utils/config.py	61.53% <0.00%> (-15.39%)	:arrow_down:
src/gluonnlp/models/bart.py	78.75% <0.00%> (-15.00%)	:arrow_down:
src/gluonnlp/models/roberta.py	84.25% <0.00%> (-9.37%)	:arrow_down:
src/gluonnlp/models/albert.py	87.71% <0.00%> (-7.72%)	:arrow_down:
src/gluonnlp/utils/parameter.py	86.53% <0.00%> (-7.70%)	:arrow_down:
... and 11 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f5c5b73...5b282f6. Read the comment docs.