NVIDIA / earth2mip

Earth-2 Model Intercomparison Project (MIP) is a python framework that enables climate researchers and scientists to inter-compare AI models for weather and climate.
https://nvidia.github.io/earth2mip/
Apache License 2.0
196 stars 46 forks source link

move normalize and denormalize into perturb #60

Closed yairchn closed 1 year ago

yairchn commented 1 year ago

Earth-2 MIP Pull Request

Description

This PR move the normalize and denormalize into perturb as a first step to decouple the normalization (which is model dependent) from the time loop. To do this requires providing perturb with the means and stds of the channels for all models. To that I added two numpy files to channel_means.py and channel_stds.py which are dicts that contain the largest dataset we currently have (235 channels).

Checklist

Dependencies

yairchn commented 1 year ago

I have tested a small ensemble inference in the notebook and the results below look as expected

Screenshot 2023-10-06 at 9 04 51 PM
nbren12 commented 1 year ago

/blossom-ci

yairchn commented 1 year ago

/blossom-ci

nbren12 commented 1 year ago

/blossom-ci

nbren12 commented 1 year ago

/blossom-ci

NickGeneva commented 1 year ago

/blossom-ci

nbren12 commented 1 year ago

There is a test failure looking up the missing "a" data: https://github.com/NVIDIA/earth2mip/actions/runs/6475309470/job/17582074381#step:2:923

Suggest manually adding it to the means/stds dicts in the test function. Or you could change the default behavior of normalization to e..g. not perturb variables with no included mean/var.

NickGeneva commented 1 year ago

/blossom-ci