Problematic hybrid coefficients in L93

[adamrher]

What happened?

I discovered some ill conditioned aspects of the L93 grid when trying to run ne480pg3, which supports higher mountain tops and smaller surface pressures than coarser grids. @JulioTBacmeister has generated profiles of the level thickness for different values of the surface pressure, and as you can see for a 400 hPa surface pressure, level 62 goes negative:

The problem seems to be some type of kink in the coef's between 300 and 100 hPa (black line):

The red line corrects for this kink, and Julio has performed science validation for this new grid.

The problematic grid is from this ncdata file in namelist_defaults.xml

<ncdata dyn="se" hgrid="ne30np4" npg="3" nlev="93">atm/cam/inic/se/cam7_FMT_ne30pg3_mg17_L93_c221118.nc</ncdata>

HOWEVER, we have not checked the other L93 ncdata files in namelist_defaults.xml

<ncdata dyn="se" hgrid="ne30np4" nlev="93" chem="trop_strat_mam5_vbs">atm/cam/inic/se/f.cam6_3_112.FCMTHIST_v0c.ne30.non-ogw-ubcT-effgw0.7.001.cam.i.1998-01-01-00000_c230810.nc</ncdata>
<ncdata dyn="se" hgrid="ne3np4"   nlev="93"             ic_ymd="101" >atm/cam/inic/se/cam6_QPC6_topo_ne3pg3_mg37_L93_01-01-31_c221214.nc</ncdata>
<ncdata dyn="se" hgrid="ne3np4"   nlev="93"  aquaplanet="1" ic_ymd="101" >atm/cam/inic/se/cam6_QPC6_aqua_ne3pg3_mg37_L93_01-01-31_c221214.nc</ncdata>

I'm going to look into those and post an update to this thread.

What are the steps to reproduce the bug?

Run any L93 configuration.

What CAM tag were you using?

cam6_3_153

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

/glade/derecho/scratch/aherring/cam6_3_153_FMTHIST_ne480pg3_ne480pg3_mt13_28800pes_240430_nogw_test.005

Will you be addressing this bug yourself?

Yes

Extra info

No response

[adamrher]

I have verified that all four L93 ncdata files in namelist_defaults.xml have this problematic issue. They are two ne30 files (FMTHIST and FCMTHIST) and two ne3 files (FMTHIST and QPC6). @JulioTBacmeister has already generated the ne30 FMTHIST file. I can work on the ne3 files.

I'm not exactly sure how to address the FCMTHIST file. @tilmes what requirements are needed for this file f.cam6_3_112.FCMTHIST_v0c.ne30.non-ogw-ubcT-effgw0.7.001.cam.i.1998-01-01-00000_c230810.nc Is it just that it has all the necessary species, or do we need to run a simulation to spin-up all the species?

[tilmes]

@adamrher We need the species from the specific IC file, and yes, this file has been spun up, so the tracers need to have the values in the original IC file, Simone

[adamrher]

How about we (1) swap in the corrected hyam,hybm,hyai,hybi,levs,ilevs into the current FCMTHIST file (2) and run FCMTHIST for a year (assuming the swapped inic are stable) and generate a new cam.i. file at the end of the simulation? That way all the species will have adjusted to the new level thicknesses.

[JulioTBacmeister]

The following file was written after 5 years of an FMTHIST run. Could be used as the IC default for FMTHIST.

/glade/derecho/scratch/juliob/archive/c153_ne30pg3_FMTHIST_x02/rest/1990-01-01-00000/c153_ne30pg3_FMTHIST_x02.cam.i.1990-01-01-00000.nc

[tilmes]

@adamrher I can setup a new FCMTHIST run, and can spinup for 1 year, if I have the hyam,hybm,hyai,hybi,levs,ilevs information (I assume from /glade/derecho/scratch/juliob/archive/c153_ne30pg3_FMTHIST_x02/rest/1990-01-01-00000/c153_ne30pg3_FMTHIST_x02.cam.i.1990-01-01-00000.nc) or if you produce the file. Also, we need to produce a file for this case, since I am testing the simpler climate chemistry case: /glade/derecho/scratch/tilmes/archive/f.cam6_3_153.FCMTnudged_climate_chemistry_ne30.factor_fix/rest/1996-01-01-00000/f.cam6_3_153.FCMTnudged_climate_chemistry_ne30.factor_fix.cam.i.1996-01-01-00000.nc

[adamrher]

That would be great if you could do the two (one simple one "default") 1 year long runs. Modifying the inic is just:

ncks -A -v hyam,hybm,hyai,hybi,lev,ilev in.nc out.nc

where in.nc is just Julio's file as you mentioned. Note that it's "lev" not "levs" as I previously mistook it.

[tilmes]

@adamrher @JulioTBacmeister should I use this code base: cam6_3_160 Also, I want to run a 10 year run, to compare the climate chemistry and the simple-chemistry (FMTHIST). Would this be the right run? f.cam6_3_160.FMTHIST_ne30.001 For future runs, could you add soa_a2 to the fincl 7 list, and for easier processing for the diagnostics tool, could we add the fincl7 back into fincl1?

[adamrher]

Yes, that code base or even older code bases would be fine, we just want the cam.i file. I'm not sure if it defaults to inithist='ENDOFRUN' so I usually just set this explicitly in user_nl_cam.

Yes and probably yes on the fincl issues.

[tilmes]

I want to do a longer 10-year run. should I used the latest sandbox for that?

[adamrher]

You could either run cam6_3_160 so you can compare to the f.cam6_3_160.FMTHIST_ne30.001 baseline, but you would need to make sure to copy all the SourceMods and user_nl_xxx. Or you could start a new clean baseline with the new cesm tag cesm2_3_alpha17f in which case you would just run out of the box as our prior SourceMods are in this latest code base now.

I went ahead and made the modified ncdata file for these runs here (unless you already did this):

/glade/derecho/scratch/aherring/restart/L93v2/f.cam6_3_112.FCMTHIST_v0c.ne30.non-ogw-ubcT-effgw0.7.001.cam.i.1998-01-01-00000_c230810.L93v2.nc

[tilmes]

@JulioTBacmeister can use the new cesm2_3_alpha17f version, and also add the Source mods from this case to have better dynamics, /glade/campaign/cesm/cesmdata/cseg/runs/cesm2_0/f.cam6_3_160.FMTHIST_ne30.moving_mtn.001 Is that correct Julio?

Actually, I think, I just use the cam6_3_160 version with the moving mountain and add chemistry, does that make sense?

[JulioTBacmeister]

@tilmes I think using the cam6_3_160 is good. You can copy the SourceMods in Cecile's run:

/glade/p/cesmdata/cseg/runs/cesm2_0/f.cam6_3_160.FMTHIST_ne30.moving_mtn.001/SourceMods/src.cam/

That will turn on the current version of moving mountains.

[adamrher]

@tilmes let me know when you have a year completed, so I can start the PR. Thanks.

[tilmes]

@adamrher I have a new IC file, but my run seems to not restart anymore, I did a startup with the new IC worked, but not a restart, pleas see: /glade/derecho/scratch/tilmes/f.cam6_3_160.FCMT_ne30.moving_mtn.001/run/cesm.log.4568717.desched1.240523-050740 any idea what is going on?

[adamrher]

It looks like it did restart successfully once - did it run a full year 1995, restart, and run a full year 1996, and then the error occurred at init for the 1997 job? The atm.log is flagging some gw code. I'm not sure what's going on.

Is the new ncdata file for FCMTHIST just in the archive directory for this case f.cam6_3_160.FCMT_ne30.moving_mtn.001?

[tilmes]

I ran 1995 but the model crashed to restart after the first year. Then I used the new ncdata file and started again in 1996, but again, the model would not restart properly in Jan 1997.

[tilmes]

The new ncdata file is already in the namelist.

[adamrher]

The crash is coming from the init phase of gw_drag.F90, which you are SourceMod'ing in. Can you try without the SourceMods and see if it still fails?