search

openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference
MIT License
13 stars 11 forks source link

Project future with regars to torchdata #307

Open sehoffmann opened 2 months ago

sehoffmann commented 2 months ago

Dear Developers,

I recently stumbled about this project and I am very delighted to see such a project! In the past, I took it on my own to build a similar library (https://github.com/sehoffmann/atmodata) which also built on torchdata. Given that the future of torchdata is very uncertain at the moment, I discontinued my efforts.

What is your stance on torchdata? Are you planning to migrate to a custom solution in the future? Will you keep using it for the time being? Are you planning to fork it?

I am considering migrating my dataloading to this library and potential also adapt it to my needs (and hence also contribute). But, I don't want to sink time into a project again with no clear long term support.

jacobbieker commented 2 months ago

Hi,

We plan on continuing with this repo, at least at this time. With regards to the lack of support in torchdata, we did #241, #248 and #227 to move from the torchdata datapipes to the internal PyTorch datapipes. They don't have as much features, but the core implementation is still there, and now those specific datapipes should stay supported as they are in PyTorch core. So we are not dependant on torchdata anymore for this repo.

We would love your contributions!