sxjscience commented 3 years ago

Description

Add AMP support + update benchmarking script.

[x] Testing utility of fp16 backbone
[x] Update tests and benchmarking script
[x] Update SQuAD finetuning to use amp
[x] Use boolean masking

Some issues:

The test for GPT-2 is turned off due to https://github.com/apache/incubator-mxnet/issues/19463
The test of MobileBERT is turned off since it has numerical isssues in float16.
TVM test has been disabled because it has some problems when we turned on boolean masking: https://github.com/dmlc/gluon-nlp/issues/1425 . We need to fix it on the TVM side.

Checklist

Essentials

[x] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage
[x] Code is well-documented

cc @dmlc/gluon-nlp-team

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR1405/amp/index.html

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR1405/amp/index.html

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR1405/amp/index.html

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR1405/amp/index.html

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR1405/amp/index.html

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR1405/amp/index.html

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR1405/amp/index.html

codecov[bot] commented 3 years ago

Codecov Report

Merging #1405 into master will increase coverage by 0.42%. The diff coverage is 87.83%.

@@            Coverage Diff             @@
##           master    #1405      +/-   ##
==========================================
+ Coverage   85.13%   85.55%   +0.42%     
==========================================
  Files          53       53              
  Lines        6928     6987      +59     
==========================================
+ Hits         5898     5978      +80     
+ Misses       1030     1009      -21

Impacted Files	Coverage Δ
src/gluonnlp/models/transformer.py	`98.94% <ø> (ø)`
src/gluonnlp/optimizer.py	`82.22% <ø> (-0.20%)`	:arrow_down:
src/gluonnlp/models/transformer_xl.py	`80.80% <14.28%> (-1.65%)`	:arrow_down:
src/gluonnlp/utils/testing.py	`94.16% <94.11%> (-0.04%)`	:arrow_down:
src/gluonnlp/attention_cell.py	`80.39% <100.00%> (ø)`
src/gluonnlp/data/sampler.py	`96.57% <100.00%> (+0.02%)`	:arrow_up:
src/gluonnlp/models/bart.py	`93.75% <100.00%> (+7.50%)`	:arrow_up:
src/gluonnlp/models/gpt2.py	`98.26% <100.00%> (+0.01%)`	:arrow_up:
src/gluonnlp/utils/misc.py	`58.58% <0.00%> (-0.62%)`	:arrow_down:
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1726dd2...6b2e1ea. Read the comment docs.