add nan checks - Githubissues

openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference

MIT License

13 stars 11 forks source link

add nan checks #235

Closed peterdudfield closed 10 months ago

peterdudfield commented 10 months ago

Detailed Description

Add checks on nwp and satellite if there are any nans

Context

good to check data is validate
should we log a warning, or raise an assertion?

Possible Implementation

could check when we load the data, and when we have made the batches
we should check for, nans, if all the values are exactly the same

peterdudfield commented 10 months ago

@jacobbieker would you be able to take this?

jacobbieker commented 10 months ago

Yep, I'll start working on it today

peterdudfield commented 10 months ago

Thanks @jacobbieker

peterdudfield commented 10 months ago

What do you think about raising or logging?

jacobbieker commented 10 months ago

I would go with raising an error. The only thing we should check is if any of the data sources use NaNs as fill values, in which case for those, I guess that would be a warning instead

jacobbieker commented 10 months ago

Finished with #236