search

E3SM-Project / E3SM

Energy Exascale Earth System Model source code. NOTE: use "maint" branches for your work. Head of master is not validated.
https://docs.e3sm.org/E3SM
Other
339 stars 343 forks source link

Probably NaN in `conv_water.F90` with `ERS_D_Ld3.ne30pg2_r05_IcoswISC30E3r5.WCYCL1850.pm-cpu_gnu.allactive-nlmaps` #6179

Open ndkeen opened 5 months ago

ndkeen commented 5 months ago

Using Jan23 master and ERS_D_Ld3.ne30pg2_r05_IcoswISC30E3r5.WCYCL1850.pm-cpu_gnu.allactive-nlmaps or the older: ERS_D_Ld3.ne30pg2_r05_EC30to60E2r2.WCYCL1850.pm-cpu_gnu.allactive-nlmaps

256: Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.
256:
256: Backtrace for this error:
256: #0  0x148089453dbf in ???
256: #1  0x6ac0b09 in __shr_infnan_mod_MOD_shr_infnan_isnan_double
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/share/util/shr_infnan_mod.F90.in:235
256: #2  0x1fc2bdf in __conv_water_MOD_conv_water_4rad
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/components/eam/src/physics/cam/conv_water.F90:370
256: #3  0x1f7e3db in __cloud_diagnostics_MOD_cloud_diagnostics_calc
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/components/eam/src/physics/cam/cloud_diagnostics.F90:371
256: #4  0x149e051 in tphysbc
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/components/eam/src/physics/cam/physpkg.F90:3046
256: #5  0x14b76d5 in __physpkg_MOD_phys_run1
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/components/eam/src/physics/cam/physpkg.F90:1175
256: #6  0x653d00 in __cam_comp_MOD_cam_run1
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/components/eam/src/control/cam_comp.F90:268
256: #7  0x643747 in __atm_comp_mct_MOD_atm_init_mct
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/components/eam/src/cpl/atm_comp_mct.F90:523
256: #8  0x48d2a9 in __component_mod_MOD_component_init_cc
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/driver-mct/main/component_mod.F90:248
256: #9  0x47dff3 in __cime_comp_mod_MOD_cime_init
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/driver-mct/main/cime_comp_mod.F90:2331
256: #10  0x485d4c in cime_driver
256:    at /global/cfs/cdirs/e3sm/ndk/repos/me37-jan23/driver-mct/main/cime_driver.F90:122

Also fails with DEBUG and intel

ndkeen commented 4 months ago

Same issue with master of Feb26th -- even using the updated resolution: ERS_D_Ld3.ne30pg2_r05_IcoswISC30E3r5.WCYCL1850.pm-cpu_gnu.allactive-nlmaps

Tested again with master of April 22. Same issue as above.

ndkeen commented 2 months ago

Same issue with master of May 4th