Stateful dataloader - Githubissues

pytorch / data

A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.

BSD 3-Clause "New" or "Revised" License

1.08k stars 142 forks source link

Stateful dataloader #1241

Closed andrewkho closed 2 months ago

andrewkho commented 3 months ago

Adds a StatefulDataLoader - drop in replacement for torch.utils.data.DataLoader - that offers state_dict/load_state_dict for mid-epoch checkpointing.

please see [RFC] Stateful DataLoader (TODO: link)

Please read through our contribution guide prior to creating your pull request.

Note that there is a section on requirements related to adding a new DataPipe.

Fixes #{issue number}

Changes

Add torchdata.stateful_dataloader.StatefulDataLoader class
copy test_dataloader unit-tests from pytorch/torch/utils/data for regressions
adds test_state_dict unit-tests testing stateful functionality.

facebook-github-bot commented 3 months ago

@andrewkho has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 3 months ago

@andrewkho has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 months ago

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 months ago

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 months ago

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 months ago

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 months ago

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 months ago

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 months ago

@gokulavasan merged this pull request in pytorch/data@0c98f9ceff7af5d3cc4d9f9ef5a23ee5561c6063.