🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support
Added a new field use_stateful_dataloader in DataLoaderConfiguration. Passing this into the config makes it so that all DataLoaders prepared and returned by the Accelerator are StatefulDataLoader objects from the torchdata library
Create a class DataLoaderAdapter which can wrap around and act as either PyTorch's DataLoader, or other variants of it such as StatefulDataLoader
Refactor DataLoaderShard, DataLoaderDispatcher, and SkipDataLoader to inherit from DataLoaderAdapter instead of DataLoader
Testing
Added new unit tests to test that StatefulDataLoader can be dropped in and loaded and saved from.
Running python -m pytest -sv ./tests with nightly torchdata installed does not pass all tests, output is actually 1 failed, 245 passed, 109 skipped, 27 warnings
However, if torchdata is not installed, all existing tests pass, suggesting this is not regressive.
torchdata.stateful_dataloader.StatefulDataLoader is only available in the beta build of torchdata, this is not a stable feature.
Adding another dependency (on a nightly package) means that almost none of the tests added in this PR is done underneath the existing images or imports.
This has only been tested on my local workstation using a single GPU.
The implementation of DataLoaderAdapter is somewhat invasive and uses some questionable reflection
Before submitting
[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
What does this PR do?
Fixes https://github.com/huggingface/accelerate/issues/2859
This PR does the following:
use_stateful_dataloader
inDataLoaderConfiguration
. Passing this into the config makes it so that allDataLoader
s prepared and returned by the Accelerator areStatefulDataLoader
objects from the torchdata libraryDataLoaderAdapter
which can wrap around and act as either PyTorch'sDataLoader
, or other variants of it such asStatefulDataLoader
DataLoaderShard
,DataLoaderDispatcher
, andSkipDataLoader
to inherit fromDataLoaderAdapter
instead ofDataLoader
Testing
Added new unit tests to test that StatefulDataLoader can be dropped in and loaded and saved from.
Caveats
torchdata
package is installed. The package itself may have conflicts withaccelerate
, see https://github.com/huggingface/accelerate/issues/2894python -m pytest -sv ./tests
with nightlytorchdata
installed does not pass all tests, output is actually1 failed, 245 passed, 109 skipped, 27 warnings
torchdata
is not installed, all existing tests pass, suggesting this is not regressive.torchdata.stateful_dataloader.StatefulDataLoader
is only available in the beta build oftorchdata
, this is not a stable feature.DataLoaderAdapter
is somewhat invasive and uses some questionable reflectionBefore submitting
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.
@muellerzr