Should we remove `datapipes`? Yes

peterdudfield commented 2 months ago

Detailed Description

The idea is to remove torch datapipes from this repo. We would essentially replace this with normal python functions instead. For our ML models, we can then wrap these in torch datasets afterwards

pros and cons

pros	cons
Less work not to do it	Not sure what the benefits are for the extra code
Torch data is good for steaming data	We use xarray, which is good for streaming large data
	Its complex and hard to make changes
	datapipes not well support by the community
	forking is annoying
	we have used some infinite loops are bad
	debugging and logging is hard
	we can use torch dataset which is widely used

Possible Implementation

Start a fresh repo and copy over the functions we need
Refactor this repo, -- pull out functions from all datapipes -- rebuild dataflow using new function. rebuilding these files

1. pros	2. pros
Nice to start with a fresh repo	No code duplication
Easier to refactor, don need to worry about breaking tests	Can continue developing
Could get one entire pipeline working first .e.g PVnet	Dont need to setup CI
	Might be able to itterate and solve each function

What I would like to keep

Configuration
batch strucutre
making building blocks, so it doesnt matter what order we do things. passing around mainly xarray objects seemed to work
There's some good readme that help, and here
folder structure i think is nice

Other things to do

We should remove any pipelines that are not being used like metnet, and perceiver
Asset memory usage, make sure the pipelines are effecient
refactor nwp duplicate code
348

peterdudfield commented 1 month ago

Here's a list of all the current datapipes we have

peterdudfield commented 1 month ago

More detailed analysis here

openclimatefix / ocf_datapipes

Should we remove `datapipes`? Yes #342

Detailed Description

pros and cons

Possible Implementation

Other things to do

348