search

NVIDIA / earth2mip

Earth-2 Model Intercomparison Project (MIP) is a python framework that enables climate researchers and scientists to inter-compare AI models for weather and climate.
https://nvidia.github.io/earth2mip/
Apache License 2.0
183 stars 40 forks source link

🐛[BUG]: Running inference_ensemble with multiple ensemble members per rank causes perturbations to be applied multiple times #164

Closed ankurmahesh closed 7 months ago

ankurmahesh commented 7 months ago

Version

source - main @ git commit: ebd86637ec853c11e9ff5e717b053351c740e526

On which installation method(s) does this occur?

Source

Describe the issue

When running 32 ensemble members with 1 rank, the later ensemble members have the perturbation applied multiple times. So the second ensemble member on each rank has a perturbation applied twice. The third ensemble member has a perturbation applied three times. Etc. Is this behavior expected?

This could make the magnitude of the perturbations very large, unless there is some perturbation magnitude normalization applied later.

Using the correlated IC perturbation method with noise amplitude 0.05, the first ensemble member has this magnitude of perturbation: image

The 15th ensemble member has this perturbation: image

And the 31st ensemble member has this perturbation: image

I think there could be two causes:

  1. x is being overwritten by the perturb method here, so as the loop moves forward, the perturbed x gets perturbed again, instead of the original x getting perturbed.
  2. x is mutable, and x is being modified here.

Environment details

No response

nbren12 commented 7 months ago

How strange. Thanks for reporting this issue @ankurmahesh.