Lightning-Universe / lightning-transformers

Flexible components pairing 🤗 Transformers with :zap: Pytorch Lightning

https://lightning-transformers.readthedocs.io

Apache License 2.0

607 stars 77 forks source link

Allow streaming datasets for the language modeling task #256

Closed SeanNaren closed 2 years ago

SeanNaren commented 2 years ago

Allows us to use streaming datasets. Only tested for the language modeling task.

codecov[bot] commented 2 years ago

Codecov Report

Merging #256 (7a0e982) into master (c141628) will increase coverage by 0%. The diff coverage is 93%.

@@          Coverage Diff          @@
##           master   #256   +/-   ##
=====================================
  Coverage      75%    75%           
=====================================
  Files          73     74    +1     
  Lines        1622   1641   +19     
=====================================
+ Hits         1210   1228   +18     
- Misses        412    413    +1