search

ecmwf-lab / ai-models

Apache License 2.0
399 stars 68 forks source link

NaNs in Fourcastnet #46

Open Isaaciwd opened 5 months ago

Isaaciwd commented 5 months ago

There are NaNs present in both the input and output of Fourcastnet. For the single-layer variables (tcwv, u10, v10, t2m, u100, v100, msl, tp, sp), there are no NaNs at the 0th timestep, which represents the initial conditions. However, at latitude = -90, every value is NaN for all longitudes at all subsequent timesteps.

For variables with pressure as a coordinate (z, t, u, v, r), the pattern is less consistent. These variables contain NaNs at the 0th timestep, typically at the highest level (50 hPa). Additionally, for all these variables except z, all values at 50 hPa are NaNs at every timestep. There are other instances of NaNs, but I have not yet identified a clear pattern for them.

I am using the latest versions of ai-models (0.6.1) and ai-models-fourcastnet (0.0.7). I also saw similar behavior with ai-models 0.4.0 and ai-models-fourcastnet 0.0.2. I tested this on two machines: one running openSUSE Leap 15.4 and the other on Red Hat 8.8, and I ran on both CPU and GPU.

1301970063 commented 3 months ago

The reason is that you didn't provide the correct initial conditions.

Isaaciwd commented 2 months ago

Yes, that could be it. But the ai-models package downloads, processes and provides the initial conditions so this is still an issue with the package. I should have added that I'm running using CDS input, I'd be interested to know if this problem still arises with MARS input.

1301970063 commented 2 months ago

Mars requires an authorized account to access, and I haven't used it. I also tried downloading the ERA5 data myself and found that it can run successfully.

Isaaciwd commented 2 months ago

Okay, so just to confirm, using ERA5 data you downloaded yourself you were able to get fourcastnet to run without producing NaNs?