Save iterator state once instead of storing it again in dataset state

pytorch / data

A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.

BSD 3-Clause "New" or "Revised" License

1.08k stars 142 forks source link

Save iterator state once instead of storing it again in dataset state #1258

Closed gokulavasan closed 1 month ago

gokulavasan commented 1 month ago

Summary: Fixes https://github.com/pytorch/data/issues/1259

Test Plan: Added unit tests

Reviewers: @andrewkho

Subscribers:

Tasks: https://github.com/pytorch/data/issues/1259

Tags:

Please read through our contribution guide prior to creating your pull request.

Note that there is a section on requirements related to adding a new DataPipe.

Fixes #{1259}

Changes

Store state only once in fetcher_state if the dataset is same as the iterator object

andrewkho commented 1 month ago

@pytorchbot merge

andrewkho commented 1 month ago

@pytorchbot drci

pytorch-bot[bot] commented 1 month ago

:link: Helpful Links

:test_tube: See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/data/1258

:page_facing_up: Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

:white_check_mark: No Failures

As of commit 50821259411452078268b50297d095c6b5464fbc with merge base a0412de86211f30d9c79acbb5dae73fce23e5739 (): :green_heart: Looks good so far! There are no failures yet. :green_heart:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot commented 1 month ago

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.