billsacks
commented
1 month ago After some consultation with @mvertens and additional testing, we feel that xgrid is working as intended here. The 0 values over sea ice grid cells also appear in runs with aoflux_grid = "xgrid" but where the atmosphere and ocean are running on different grids. CAM crashes in the same way in SMS_D_Ln9.f09_g17.FCnudged_GC.derecho_intel.cam-outfrq9s with aoflux_grid = "ogrid" as I noted above with SMS_D_Ln9.f09_f09_mg17.FCnudged_GC.derecho_intel.cam-outfrq9s with "xgrid". I believe this is a CAM issue, unless CAM wants to push for changes in the long-standing behavior of the mediator in this respect (which could be a reasonable solution). So I have moved this to a CAM issue:
I should say up-front that I'm not sure if this is actually a bug with aoflux_grid=xgrid, but it is a difference between aoflux_grid=xgrid and aoflux_grid=ogrid, and it seems like it might be contributing to a model crash. So I'm opening this to start a discussion on whether this might be a bug.
I'll start with the conclusion before diving into details: It seems like, with aoflux_grid=ogrid, we get 0 values sent to atm from the atm-ocn flux calculations over land points, but NOT over 100% ice points; but with aoflux_grid=xgrid, we get 0 values sent to atm from the atm-ocn flux calculations over any points with (lnd+ice)=1 (i.e., any points with ofrac=0). This MAY be contributing to a crash in some configurations with xgrid, though I'm not sure yet if that's the cause. @jedwards4b @mvertens @uturuncoglu @DeniseWorthen - do any of you have a sense of what's right here?
This started with an investigation of a failure in the test SMS_D_Ln9.f09_f09_mg17.FCnudged_GC.derecho_intel.cam-outfrq9s. I'm running this from cesm3_0_alpha03c but with CMEPS updated to cmeps1.0.18 (which is needed to fix some other issues with the exchange grid). Out-of-the-box, this test passes. However, when adding the following to user_nl_cpl –
aoflux_grid = "xgrid"– this test fails with a divide by zero in drydep_mod:which is here:
As a side-note: This test also fails in non-debug mode, though more cryptically (so I'm not positive what's going on here):
This test passes with
aoflux_grid = "ogrid", and a similar test without chemistry –SMS_D_Ln9.f09_f09_mg17.FHIST.derecho_intel.cam-outfrq9s– passes even with xgrid.I haven't dug deeply into the relevant CAM code, but I decided to look for differences in variables sent to the atmosphere in the test
SMS_D_Ln9.f09_f09_mg17.FHIST.derecho_intel.cam-outfrq9swith xgrid vs. ogrid. I started by looking at ustar, since that's one of the terms in the line with divide-by-zero (though note that I have not confirmed that this is the term causing the problem).Here is ustar sent to atm using xgrid:
And here using ogrid:
Both have 0 values over land, but note that the xgrid run has additional 0 values in the Arctic ocean and near Antarctica. I see these same extra 0 values in Med_aoflux_atm_So_ustar, but not in Med_aoflux_ocn_So_ustar. I spot-checked some other fields from the atm-ocn flux calculation and see the same thing with other fields.
Here is a map showing where ofrac is essentially 0:
And here are points where ifrac is essentially 1:
By eye, these seem to match up very well with the grid cells that have 0 values in the run with aoflux_grid=xgrid. This leads me to the conclusion at the top of the issue.