openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference
MIT License
13 stars 11 forks source link

Buffer fix #317

Closed dfulu closed 1 month ago

dfulu commented 1 month ago

Fix assumption causing buffer overflow. Previously we always assumed we would slice in time before slicing in space. When diffing the accumulated variables that order is less efficient than slicing in space and then slicing in time (where we need to load data).

This pull request means we can do slicing in either order.

Also slight tidy up to address warning about upcoming version of pandas