search

Lightning-AI / litdata

Transform datasets at scale. Optimize datasets for fast AI model training.
Apache License 2.0
334 stars 39 forks source link

Flexibility to set device number < total device count #155

Closed yhl48 closed 3 months ago

yhl48 commented 3 months ago
Before submitting - [X] Was this discussed/agreed via a Github issue? (no need for typos and docs improvements) - [ ] Did you read the [contributor guideline](https://github.com/Lightning-AI/lit-data/blob/main/.github/CONTRIBUTING.md), Pull Request section? - [ ] Did you make sure to update the docs? - [ ] Did you write any new necessary tests?

What does this PR do?

Fixes #152. However, this implementation is not fully correct, it will break if there is more than one node.

yhl48 commented 3 months ago

@tchaton is there a way to get the devices variable?

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@935eef5). Learn more about missing BASE report.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #155 +/- ## ===================================== Coverage ? 77% ===================================== Files ? 30 Lines ? 4126 Branches ? 0 ===================================== Hits ? 3186 Misses ? 940 Partials ? 0 ```
yhl48 commented 3 months ago

the problem might be more complicated than I thought, usually devices is set in the lightning Trainer class, but _DistributedEnv is called before that