openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference
MIT License
13 stars 11 forks source link

chore: Add test for checking physical limits and zeroes in NWP data #… #340

Open glitch401 opened 4 days ago

glitch401 commented 4 days ago

Pull Request

Description

This pull request addresses issues identified in #335 and #337 by implementing checks for zeros and physical limits in NWP data processing. The changes ensure that the OpenNWP class correctly raises a ValueError when encountering NWP data arrays containing zeros (addressing #335) and when NWP data values are outside specified physical limits (addressing #337). These enhancements are crucial for maintaining data integrity and reliability in our processing pipeline.

Fixes #337 , #335

How Has This Been Tested?

The modifications have been validated through comprehensive unit tests. Specifically, tests were added to verify that a ValueError is raised both when zeros are present in the data array and when data values fall outside of physical limits. These tests were conducted using sample Zarr datasets designed to mimic real-world scenarios where such issues might arise.

A sanity check was performed by visually inspecting the processed data to ensure that the new checks effectively identify and handle data with zeros and data outside physical limits.

Checklist:

glitch401 commented 3 days ago

@peterdudfield are there any other suggestion for this PR?

peterdudfield commented 3 days ago

@peterdudfield are there any other suggestion for this PR?

Thanks so much, ive put a few comments, but then i think it should be ready