sxjscience commented 4 years ago

Description

Since the horovod support has been fixed, improve our docker image. Now, the CI docker will depend on the base docker image, which supports:

horovod training
TVM

Checklist

Essentials

[x] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[x] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[ ] Code is well-documented

cc @dmlc/gluon-nlp-team

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

codecov[bot] commented 4 years ago

Codecov Report

Merging #1378 into master will decrease coverage by 0.09%. The diff coverage is 19.04%.

@@            Coverage Diff             @@
##           master    #1378      +/-   ##
==========================================
- Coverage   71.09%   71.00%   -0.10%     
==========================================
  Files         107      107              
  Lines       12607    12619      +12     
==========================================
- Hits         8963     8960       -3     
- Misses       3644     3659      +15

Impacted Files	Coverage Δ
...ipts/datasets/pretrain_corpus/prepare_gutenberg.py	`0.00% <0.00%> (ø)`
...ts/datasets/question_answering/prepare_searchqa.py	`0.00% <0.00%> (ø)`
...ripts/datasets/question_answering/prepare_squad.py	`0.00% <0.00%> (ø)`
src/gluonnlp/models/__init__.py	`96.87% <ø> (ø)`
src/gluonnlp/utils/lazy_imports.py	`54.32% <20.00%> (-2.26%)`	:arrow_down:
tests/test_models.py	`100.00% <100.00%> (ø)`
src/gluonnlp/data/filtering.py	`78.26% <0.00%> (-4.35%)`	:arrow_down:
src/gluonnlp/data/tokenizers/subword_nmt.py	`79.43% <0.00%> (+0.93%)`	:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 4fb41d7...3e1a326. Read the comment docs.

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

szha commented 4 years ago

@sxjscience ready to merge?

sxjscience commented 4 years ago

No, there are some problems of the GPU docker due to the fact that " libcuda is required just to import mxnet" https://github.com/apache/incubator-mxnet/issues/19139#issuecomment-692395341

sxjscience commented 4 years ago

Basically, horovod relies on runtime check of MXNet to fill in the cmake flags. But the import mxnet will fail if you call it inside docker build. I'm figuring out the solution to that.

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html

github-actions[bot] commented 4 years ago

The documentation website for preview: http://gluon-nlp-dev.s3-accelerate.amazonaws.com/PR1378/fix_docker/index.html