Torchdata Future is up in the air

openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference

MIT License

13 stars 11 forks source link

Torchdata Future is up in the air #227

Closed jacobbieker closed 9 months ago

jacobbieker commented 10 months ago

https://github.com/pytorch/data/issues/1196

Future of datapipes is being debated now, not sure when it will be announced what is going to happen with it or how it might get refactored/changed, but this will have a very large effect on this repo. Nothing should change soon, and these datapipes should keep working with future versions of PyTorch, but seemingly might not have new features or bug fixes (like the odd issue with Dataloader2 not working as well as Dataloader) happen to it.

jacobbieker commented 9 months ago

Apparently, DataPipes have been merged into PyTorch core: https://github.com/pytorch/pytorch/tree/main/torch/utils/data/datapipes so we could switch to that implementation instead, to keep up with PyTorch. We lose DataLoader2 and the like, but seems relatively straightforward, as we've mostly implemented our own datapipes.

peterdudfield commented 9 months ago

ready to close?