search

eto-ai / rikai

Parquet-based ML data format optimized for working with unstructured data
https://rikai.readthedocs.io/en/latest/
Apache License 2.0
136 stars 19 forks source link

torch.utils.data.BufferedShuffleDataset was removed #599

Open changhiskhan opened 2 years ago

changhiskhan commented 2 years ago

not documented but the class no longer exists as of torch 1.9

eddyxu commented 2 years ago

Hmm, so it means we lose pytorch random shuffling again? What should be the action items here?

eddyxu commented 2 years ago

TODO: Verify that rikai.torch.Dataset can work with new PyTorch's shuffling capability (or add back shuffling ourselves?)

eddyxu commented 2 years ago

https://pytorch.org/blog/pytorch-1.11-released/ Probably need to implement this new API