dmlc / gluon-nlp

NLP made easy

https://nlp.gluon.ai/

Apache License 2.0

2.56k stars 538 forks source link

BERT pretraining #1376

Closed ZiyueHuang closed 3 years ago

ZiyueHuang commented 3 years ago

Description

@sxjscience

Checklist

Essentials

[ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[ ] Code is well-documented

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

cc @dmlc/gluon-nlp-team

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1376/bert/index.html

codecov[bot] commented 3 years ago

Codecov Report

Merging #1376 into master will decrease coverage by 0.04%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1376      +/-   ##
==========================================
- Coverage   71.09%   71.04%   -0.05%     
==========================================
  Files         107      107              
  Lines       12607    12607              
==========================================
- Hits         8963     8957       -6     
- Misses       3644     3650       +6

Impacted Files	Coverage Δ
src/gluonnlp/data/filtering.py	`78.26% <0.00%> (-4.35%)`	:arrow_down:
src/gluonnlp/data/tokenizers/yttm.py	`81.89% <0.00%> (-0.87%)`	:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 4fb41d7...9f83ab7. Read the comment docs.

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1376/bert/index.html

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1376/bert/index.html

sxjscience commented 3 years ago

LGTM overall. But we may later need to consider to merge the Electra + Bert (nsp + mlm) + Albert (sop + mlm) implementations.

sxjscience commented 3 years ago

Would you try to merge the upstream/master? It seems that there's something wrong in the GPU test.

ZiyueHuang commented 3 years ago

Would you try to merge the upstream/master? It seems that there's something wrong in the GPU test.

done

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1376/bert/index.html