New cam_dev files cause non-simple physics compsets to fail

[gdicker1]

Real-data IC and topo files added in PRs EWOrg/CAM #16 and EWOrg/CAM #20 cause run-time fails of compsets that use "full physics" - especially those with chemistry and radiation interactions.

"Simple-physics" compsets like FADIAB and any MPAS-A standalone runs are successful. (NOTE: some compsets like FHS94 are also unaffected since they use separate "notopo" files.)

Compsets affected: F2000climo (when requesting these files), F2000dev, and CHAOS2000dev.

Example case on Derecho: /glade/derecho/scratch/gdicker/tstgrids_F2000dev_2_mpasa480_20240606-110547/run

• Especially the files associated w/ job number 4744653. The rank104.cesm.log.4744653.txt file maybe helpful

How to reproduce:

1. Clone the most recent release or the development branch of EarthWorks
2. Create a compset that uses full physics and mpas grid (e.g. ./create_newcase --compset F2000dev --res mpasa480_mpasa480 )
3. Submit the case after setting up and building

Symptoms seen:

• In between "Dynamics timestep ..." output there will be lines in the atm.log.* file like "imp_sol: Time step 1.1250000000000E+01 failed to converge @ (lchnk,lev,col,nstep) = 2 32 4 0" (see first the code-block below for a longer example).
  • These convergence failures cause the simulation time to increase (due to repeated iterations) and eventually lead to the simulation crashing.
• There may also be a message from the MPAS-A dycore in the atm.log.* file and in one log.atmosphere.${rank}.err file per rank: "CRITICAL ERROR: NaN detected in 'w' field." This crashes the simulation, but typically only after a few timesteps.
• There may be a segmentation fault that occurs in the rrtmgp radiation which also cause runs to fail (TODO fill-in these details).
• The number of timesteps before failure seems to be resolution dependent. 480km tests fail after a few timesteps while 120km test cases require runs longer than 5 days to fail.

Sample output seen in atm.log.* and cesm.log.*

 imp_sol: Time step   1.1250000000000E+01 failed to converge @ (lchnk,lev,col,nstep) =      2    32     4     0
 imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =      2    32     4     0  1.1250000000000E+01  1.7887500000000E+03
 num_a1    1.000E+00
 so4_a1    1.000E+00
 imp_sol: Time step   1.1250000000000E+01 failed to converge @ (lchnk,lev,col,nstep) =      2    32     4     0
 imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =      2    32     4     0  1.1250000000000E+01  1.8000000000000E+03
 num_a1    1.000E+00
 so4_a1    1.000E+00
 imp_sol : @ (lchnk,lev,col) =            2          32           4  failed          165  times

Workaround

While I continue to investigate, this issue can be mitigated by just not using these files. Supply paths to the relevant files in the ncdata and bnd_topo files in a case's user_nl_cam file. If you wish to use cam_dev physics in a case with the old files, you will need to supply use_gw_front = .false. in the user_nl_cam file.

[adamrher]

This is a shot in the dark, but Francis found a bug cam_dev where the liquid and ice paths being fed into radiation aren't being initialized. Maybe try his changes to physpkg.F90 and cloud_diagnostics.F90 in https://github.com/ESCOMP/CAM/pull/1074 ?

[gdicker1]

The changes in https://github.com/EarthWorksOrg/CAM/pull/23 will address this issue. It is waiting on an upstream version of the PR in ESCOMP/CAM (#1095) before being merged into EarthWorks and the files in https://github.com/EarthWorksOrg/CAM/pull/16 and https://github.com/EarthWorksOrg/CAM/pull/20 will be added again.