dmlc / gluon-nlp

NLP made easy

https://nlp.gluon.ai/

Apache License 2.0

2.56k stars 538 forks source link

Add fp16 support for Bert QA inference #1264

Closed MoisesHer closed 4 years ago

MoisesHer commented 4 years ago

Description

Adding FP16 in QA-BERT inference

Checklist

Essentials

[x] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[x] Code is well-documented

Changes

[x] Added option to use fp16 data type in run_squad script for evaluation (--eval_dtype). Will cast the network.
[x] Propagated the datatype to masked_softmax, where was using fp32 always and causing nans if fp16 was selected

Comments

cc @dmlc/gluon-nlp-team, @sxjscience

codecov[bot] commented 4 years ago

Codecov Report

Merging #1264 into numpy will not change coverage. The diff coverage is 100.00%.

@@           Coverage Diff           @@
##            numpy    #1264   +/-   ##
=======================================
  Coverage   82.53%   82.53%           
=======================================
  Files          38       38           
  Lines        5446     5446           
=======================================
  Hits         4495     4495           
  Misses        951      951

Impacted Files	Coverage Δ
src/gluonnlp/attention_cell.py	`79.91% <100.00%> (ø)`
src/gluonnlp/models/bert.py	`84.42% <100.00%> (ø)`

sxjscience commented 4 years ago

Will need to add test later.