olyson
commented
5 years ago My branch for this is here:
https://github.com/olyson/ctsm/tree/hrv_xsmrpool_to_atm
I have done very limited testing just to make sure restarts work. The following tests pass (or fail) as expected:
ERP_Ld5.f09_g17.I1850Clm50BgcCropCru.cheyenne_intel.clm-ciso ERS_D_Ld10.f10_f10_musgs.IHistClm50SpG.cheyenne_intel.clm-glcMEC_decrease ERS_D_Ld3.f09_g17_gl4.I1850Clm50BgcCrop.cheyenne_intel.clm-clm50KitchenSink
We'd like to have this activated via a namelist item, so that we can turn it off for runs either than scenario runs. I'm suggesting a software engineer do that if possible. I can either run the full test suite as is right now or perhaps we should wait until the namelist item is added.
klindsay28
ekluzek
wwieder
dlawrenncar
danicalombardozzi
Details of support request
I'm doing a coupled model run using CLM_CO2_TYPE=prognostic with CESM2. The run is aborting a few years in with the error message and stack traceback: 239: ENDRUN: 239: ERROR: 239: lnd_import ERROR: CO2 is outside of an expected range 239:Image PC Routine Line Source
239:cesm.exe 0000000003529B7D Unknown Unknown Unknown 239:cesm.exe 0000000002C9B7D2 shr_abort_modmp 114 shr_abort_mod.F90 239:cesm.exe 00000000019C9C3F abortutils_mp_end 50 abortutils.F90 239:cesm.exe 00000000019C6E46 lnd_import_export 251 lnd_import_export.F90 239:cesm.exe 00000000019BE266 lnd_comp_mct_mp_l 401 lnd_comp_mct.F90 239:cesm.exe 0000000000425BE4 component_modmp 728 component_mod.F90 ...
The abort occurs at model date 0004-09-28.
To diagnose what is going on, I have rerun the month 0004-09 with nstep cpl hist and nstep clm hist on CMIP6 h0, h1, h2 files. This output is in $RUNDIR, mentioned below.
Examining the last few cpl.hi files, it looks like a2x_Sa_coprog at (i,j)=(9,143) (zero based indexing) drops from ~286 ppmv to 4.2 ppmv in a single timestep. This is below the 10 ppm threshold of #427 and abort is called.
It looks like this is being caused by a larger drawdown of CO2. l2x_Fall_fco2_lnd is large and positive in the last cpl hist file.
Looking at clm2.h0 files, NEE at that same point is negative with a large magnitude. There is a block of values around northern Italy with large magnitude negative NEE in the last clm2.h0 file. The structure of the anomalous NEE values does not show up in GPP, but it does show up in NPP. It shows up in AR as a large magnitude negative value, kinda a red flag. Neither GR or MR show the anomalous values, so I'm inferring that the uptake is from one of the C terms thrown into AR to maintain C balance (e.g., XSMR related). I don't understand the nature of these terms.
Any suggestions on how to investigate this further?
The clm2 h1 and h2 files have sub-grid cell info, so hopefully that is helpful for deciphering what is going wrong.
If I look at animations of this nstep clm (or cpl) output, I see sporadic blips of very large CO2 uptake occurring at numerous grid points. I do not see an obvious pattern to the geographic distribution of the blips. They generally seem to appear for a single timestep. The blip at (i,j)=(9,143) happens to be enough of a CO2 drawdown to bring CO2 below the #427 10 ppmv threshold.
Important details of your setup / configuration so we can better assist you
release-clm5.0.14
**Have you made any modifications to code, xml files, etc.? no
compset=1850_CAM60_CLM50%BGC-CROP-CMIP6DECK_CICE%CMIP6_POP2%ECO%ABIO-DIC_MOSART_CISM2%NOEVOLVE_WW3_BGC%BPRP
aside from using BPRP, instead of BDRD, the compset is B1850
(The run was originally a hybrid off of b.e21.B1850.f09_g17.CMIP6-piControl.001, with RUN_REFDATE=0501-01-01, RUN_STARTDATE=0001-01-01, and modified CAM CO2 constituents (to bring their surface means closer to the 1850 values. After encountering the abort, I reran the last month with higher frequency output, having the case branch off of itself at 0004-09-01.)