Floating-point exception encountered with LSG and daily cycle activated

jodemaey commented 3 years ago

Hi there,

I would like to report an issue with PLASIM+LSG and the daily cycle.

I am running ensemble runs of PLASIM with the LSG. In the experiments I make, when I start with the daily cycle on, some members will eventually crash with floating-point exceptions. The design of the experiments is such that I have to stop all the members once a member crash occured, so I don't know if they will eventually all fail. The crashes seem to happen quite randomly. I monitor the runs output after each pass (20 years by 20 years) and nothing seems wrong in the data before a crash.

Please find attached the error output and the diag file of one crashing member, as well as the namelist files used to configure PLASIM: standard_error.zip standard_namelist.zip

Please also note that PLASIM was compiled using most.x with the following options:

Model Planet Simulator + Earth
Modules ML Ocean
               LSG Ocean
               Sea Ice
Parallelism 1 Cores
Resolution T42
Options Write output
Earth CO2 420.
MPSTEP 0

In addition, please note that:

I tried with both double and single precision, and the problem happens with both.
The runs with the daily cycle being deactivated do not crash.
This problem was encountered on a non-standard Cray architecture. But I do not think that it is the problem, or at least it should not be.
The compiler being used is gfortran 6.3. No particular error or warning were displayed at compilation time.

I can maybe give one way of research about this: Could it possibly be due to the LSG large steps (10 or 20 days?) in comparison with the daily cycle, which depending on how the coupling is made could lead to inconsistency?

I will also try to reproduce this problem on a classic x64 architecture as soon as I have the bandwidth, but I do not expect to encounter a different behavior.

Best Regards,

Jonathan Demaeyer.

jhardenberg commented 3 years ago

Hi, we also experienced similar problems at T42 (with and without LSG), particularly at very high CO2 levels (we were actually NOT using the daily cycle). The model would crash after a random number of years. We traced the problem back to CFL violations in zonal wind velocities. Our solution was to use a shorter timestep for the atmosphere in Plasim. Since the default for T42 is 30min, setting MPSTEP=15 solved the problem ... Could you please try and see if this solves your issue?

jodemaey commented 3 years ago

Hi Jost,

Thank you for your answer. I will try again with MPSTEP=15 and let you know the outcome, probably somewhere next week.

Thanks again,

Jonathan.

jodemaey commented 3 years ago

Hi,

Decreasing MPSTEP to 15 indeed solved the problem. No more member crash. I'm closing this issue. Thank you for your help,

Jonathan.

jhardenberg / PLASIM

Floating-point exception encountered with LSG and daily cycle activated #7