search

dmlc / gluon-nlp

NLP made easy
https://nlp.gluon.ai/
Apache License 2.0
2.55k stars 538 forks source link

[SCRIPT] use cpu count as default parallelism #1489

Closed szha closed 3 years ago

szha commented 3 years ago

Description

use cpu count as default parallelism in data processing

Checklist

Essentials

Changes

cc @dmlc/gluon-nlp-team

codecov[bot] commented 3 years ago

Codecov Report

Merging #1489 (eb6f039) into master (c582b64) will increase coverage by 3.89%. The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1489      +/-   ##
==========================================
+ Coverage   81.98%   85.87%   +3.89%     
==========================================
  Files          52       52              
  Lines        6909     6909              
==========================================
+ Hits         5664     5933     +269     
+ Misses       1245      976     -269
Impacted Files Coverage Δ
src/gluonnlp/models/mobilebert.py 87.82% <0.00%> (+0.25%) :arrow_up:
src/gluonnlp/utils/misc.py 53.92% <0.00%> (+0.35%) :arrow_up:
conftest.py 86.25% <0.00%> (+1.25%) :arrow_up:
src/gluonnlp/data/tokenizers/sentencepiece.py 78.44% <0.00%> (+2.39%) :arrow_up:
src/gluonnlp/utils/parameter.py 92.80% <0.00%> (+3.20%) :arrow_up:
src/gluonnlp/embedding/embed_loader.py 81.52% <0.00%> (+5.73%) :arrow_up:
src/gluonnlp/models/bert.py 94.80% <0.00%> (+7.26%) :arrow_up:
src/gluonnlp/models/gpt2.py 98.26% <0.00%> (+7.26%) :arrow_up:
src/gluonnlp/models/albert.py 95.43% <0.00%> (+7.71%) :arrow_up:
src/gluonnlp/models/roberta.py 93.61% <0.00%> (+9.36%) :arrow_up:
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c582b64...eb6f039. Read the comment docs.

github-actions[bot] commented 3 years ago

The documentation website for preview: http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR1489/cpu_count/index.html