sporadic large CO2 uptake causing coupled CLM_CO2_TYPE=prognostic run to abort

[klindsay28]

Details of support request

I'm doing a coupled model run using CLM_CO2_TYPE=prognostic with CESM2. The run is aborting a few years in with the error message and stack traceback: 239: ENDRUN: 239: ERROR: 239: lnd_import ERROR: CO2 is outside of an expected range 239:Image PC Routine Line Source
239:cesm.exe 0000000003529B7D Unknown Unknown Unknown 239:cesm.exe 0000000002C9B7D2 shr_abort_modmp 114 shr_abort_mod.F90 239:cesm.exe 00000000019C9C3F abortutils_mp_end 50 abortutils.F90 239:cesm.exe 00000000019C6E46 lnd_import_export 251 lnd_import_export.F90 239:cesm.exe 00000000019BE266 lnd_comp_mct_mp_l 401 lnd_comp_mct.F90 239:cesm.exe 0000000000425BE4 component_modmp 728 component_mod.F90 ...

The abort occurs at model date 0004-09-28.

To diagnose what is going on, I have rerun the month 0004-09 with nstep cpl hist and nstep clm hist on CMIP6 h0, h1, h2 files. This output is in $RUNDIR, mentioned below.

Examining the last few cpl.hi files, it looks like a2x_Sa_coprog at (i,j)=(9,143) (zero based indexing) drops from ~286 ppmv to 4.2 ppmv in a single timestep. This is below the 10 ppm threshold of #427 and abort is called.

It looks like this is being caused by a larger drawdown of CO2. l2x_Fall_fco2_lnd is large and positive in the last cpl hist file.

Looking at clm2.h0 files, NEE at that same point is negative with a large magnitude. There is a block of values around northern Italy with large magnitude negative NEE in the last clm2.h0 file. The structure of the anomalous NEE values does not show up in GPP, but it does show up in NPP. It shows up in AR as a large magnitude negative value, kinda a red flag. Neither GR or MR show the anomalous values, so I'm inferring that the uptake is from one of the C terms thrown into AR to maintain C balance (e.g., XSMR related). I don't understand the nature of these terms.

Any suggestions on how to investigate this further?

The clm2 h1 and h2 files have sub-grid cell info, so hopefully that is helpful for deciphering what is going wrong.

If I look at animations of this nstep clm (or cpl) output, I see sporadic blips of very large CO2 uptake occurring at numerous grid points. I do not see an obvious pattern to the geographic distribution of the blips. They generally seem to appear for a single timestep. The blip at (i,j)=(9,143) happens to be enough of a CO2 drawdown to bring CO2 below the #427 10 ppmv threshold.

Important details of your setup / configuration so we can better assist you

release-clm5.0.14

**Have you made any modifications to code, xml files, etc.? no

compset=1850_CAM60_CLM50%BGC-CROP-CMIP6DECK_CICE%CMIP6_POP2%ECO%ABIO-DIC_MOSART_CISM2%NOEVOLVE_WW3_BGC%BPRP

aside from using BPRP, instead of BDRD, the compset is B1850

• Location of case directory: CASEROOT=/glade/work/klindsay/cesm20_cases/B1850/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.001

(The run was originally a hybrid off of b.e21.B1850.f09_g17.CMIP6-piControl.001, with RUN_REFDATE=0501-01-01, RUN_STARTDATE=0001-01-01, and modified CAM CO2 constituents (to bring their surface means closer to the 1850 values. After encountering the abort, I reran the last month with higher frequency output, having the case branch off of itself at 0004-09-01.)

• Location of run directory: RUNDIR=/glade/scratch/klindsay/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.001/run

[wwieder]

I can look at this when I get in today, @klindsay28. To my knowledge, this is something we haven't seen in offline simulations, but I guess we wouldn't have that abort flag when running with prescribed CO2? @ekluzek what was similar behavior the motivation for #427 in the first place?

[dlawrenncar]

@olyson , @rosiealice Thanks Will, we should investigate as a high priority. Adding in Keith and Rosie to discussion to have a few more in the loop in case not following this issue.

I agree with Will that we are not aware of this issue happening in offline runs, but we may not have caught it if it happens only very rarely ... though I would expect it to show up as a large NEE anomaly even at global or regional / annual level that we would see in standard timeseries in CLM diagnostics plots. I looked at CLM5 runs and didn't see any evidence either globally or regionally in an NEE or AR spike that was outside the norm.

[rosiealice]

I've never seen anything like that either, and second what Dave said, we'd probably expect to see it... Very strange :/

[dlawrenncar]

For what it's worth, I also checked CLM standard diagnostics from a coupled simulation (without prognostic CO2) and I didn't see any evidence of spikes in annual NEE or AR.

http://webext.cgd.ucar.edu/I20TR/clm50_r270_1deg_GSWP3V1_iso_newpopd_hist/lnd/clm50_r270_1deg_GSWP3V1_iso_newpopd_hist.1995_2014-b.e21.BHIST.f09_g17.CMIP6-historical.001.1995_2014/set6/set6.html

On Fri, Dec 7, 2018 at 7:18 AM Rosie Fisher notifications@github.com wrote:

I've never seen anything like that either, and second what Dave said, we'd probably expect to see it... Very strange :/

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-445245887, or mute the thread https://github.com/notifications/unsubscribe-auth/AUAcVDY_hpfY2k4kEd89mOOgcsGspCw2ks5u2nilgaJpZM4ZH0Ek .

[klindsay28]

Attached is a figure of AR integrated over 0E-40E, 35N-55N. AR is sampled every timestep, and plotted for 3 days preceding the abort. There are numerous dips that correspond to negative AR somewhere in this domain. Because of their short duration, I expect them to be evident with longer duration sampling. The dip that causes the abort is not shown, it is in the next time step. The value of AR integrated over this box for that timestep is -934, which is way off the scale of the plot. So the abort is caused by something anomalous even compared to the dips evident in this plot.

(I'll be offline for most of today, I'm about to board a flight to AGU.)

[wwieder]

There are a few points in the code that might generate these negative AR fluxes https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/CNVegCarbonFluxType.F90#L4121 (when the crop model is on) or https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/CNVegCarbonFluxType.F90#L4129 (when fun is active).

I'm not sure what either of these fluxes comes from, but will try cloning @klindsay28 's case and writing these out to history files.

[klindsay28]

If you run into issues with the cloning, I’d be happy to rerun my case with additional output.

On Fri, Dec 7, 2018 at 9:10 AM will wieder notifications@github.com wrote:

There are a few points in the code that might generate these negative AR fluxes

https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/CNVegCarbonFluxType.F90#L4121 (when the crop model is on) or

https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/CNVegCarbonFluxType.F90#L4129 (when fun is active).

I'm not sure what either of these fluxes comes from, but will try cloning @klindsay28 https://github.com/klindsay28 's case and writing these out to history files.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-445280814, or mute the thread https://github.com/notifications/unsubscribe-auth/AO2Xu_qN_0bDYJNMluXzeA3vpujHkGI2ks5u2pMMgaJpZM4ZH0Ek .

[wwieder]

@slevisconsulting and @danicalombardozzi, do you have any old memories of what is being done with xsmr fluxes in the crop model? I'll trace this through the code, but am curious how or why we're handling AR fluxes differently in the crop model?

[danicalombardozzi]

I don't have any old memories of this, but Bin Peng and Maoyi Huong have both encountered problems with very negative NEE during crop harvest. Bin implemented a fix previously that constrained the xsmrpool_to_atm>=0. Maoyi's student tried to implement this fix in CLM5 but got carbon balance errors. I haven't followed up with them (this came to my attention right before travel and then fell off my radar), and I'm not sure whether they have a working solution at this point.

Danica

On Fri, Dec 7, 2018 at 9:19 AM will wieder notifications@github.com wrote:

@slevisconsulting https://github.com/slevisconsulting and @danicalombardozzi https://github.com/danicalombardozzi, do you have any old memories of what is being done with xsmr fluxes in the crop model? I'll trace this through the code, but am curious how or why we're handling AR fluxes differently in the crop model?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-445283632, or mute the thread https://github.com/notifications/unsubscribe-auth/AY9tQZg_bjNQdM7oFBB7ADZLLcRyngr6ks5u2pT2gaJpZM4ZH0Ek .

-- Dr. Danica Lombardozzi she/her/hers Terrestrial Sciences Section Climate and Global Dynamics National Center for Atmospheric Research Boulder, CO 80305

email: dll@ucar.edu office: (303) 497-1777

[wwieder]

@dlawrenncar , @danicalombardozzi mentioned that you'd had a previous discussion with Maoyi and Bin about a similar issue with negative xsmr fluxes associated with harvest. From @klindsay28 runs it looks like the strongly negative fluxes may be associated with harvest in agricultural regions? The attached image shows AR fluxes from Sept 28, just before the coupled model crashes.

[wwieder]

This feature is also evident in the offline CLM5 runs we've done and can likely be seen in the monthly .h0. history files where we have negative AR in some grids for particular months. This is June 1851 from Pete's recent simulation i.e21.IHIST.f09_g17.CMIP6-land-hist.001

[jinyuntang]

I am not sure if this is the same thing I experienced with ELM, but I found the algorithm in carbon allocation could lead to negative litter fluxes in some circumstances. So I guess clm has the same problem given its similarities with ELM. I fixed in my ELM branch with the multiflux limiter I presented tow years ago at the land bgc meeting. Jinyun

Sent from my iPhone

On Dec 7, 2018, at 10:32 AM, will wieder notifications@github.com wrote:

This feature is also evident in the offline CLM5 runs we've done and can likely be seen in the monthly .h0. history files where we have negative AR in some grids for particular months. This is June 1851 from Pete's recent simulation i.e21.IHIST.f09_g17.CMIP6-land-hist.001

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[wwieder]

At harvest, any C in the xsmr pool will go to the atmosphere. https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/dynHarvestMod.F90#L342 Globally, this pool is large, and very negative. @ckoven, I know you've looked at this. Can you shed any light into what's going on here? Why?

[wwieder]

It wouldn't fix the underlying issue, but would a quick fix be to apply something like the hrv_xsmrpool_to_atm_dribbler to smooth out how the atmosphere sees this negative C flux? @billsacks maybe you can commend on what the 'dribbler' is supposed to do? https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/CNVegCarbonFluxType.F90#L745

Alternatively, could we transfer the xsmr to litter pools, instead of the atmosphere?
hrv_deadcrootc_to_litter(p) = deadcrootc(p) m hrv_deadcrootc_to_litter(p) = hrv_deadcrootc_to_litter(p) + xsmrpool(p) m https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/dynHarvestMod.F90#L341

Neither of these are very satisfying fixes, so other ideas would be welcome

[wwieder]

Oner more thought on this, @ckoven maybe you have thoughts here? Would allowing a quicker recovery of the xsmrpool potentially alleviate these negative AR fluxes with harvest? this could be done by changing the daysrecover parameter (currently set to 30 days). The issue here, is we'll also be changing global C fluxes, by modifying the calculation of availc. @dlawrenncar , I'd guess this is a non-starter at this stage of the game? xsmrpool_recover(p) = -xsmrpool(p)/(dayscrecover*secspday) https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/NutrientCompetitionFlexibleCNMod.F90#L1450

[dlawrenncar]

Neither of these solutions is likely to solve the massive flux, right? That is something different, like a divide by a near zero value or something somewhere.

On Dec 7, 2018, at 1:56 PM, will wieder notifications@github.com wrote:

It wouldn't fix the underlying issue, but would a quick fix be to apply something like the hrv_xsmrpool_to_atm_dribbler to smooth out how the atmosphere sees this negative C flux? @billsacks https://github.com/billsacks maybe you can commend on what the 'dribbler' is supposed to do? https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/CNVegCarbonFluxType.F90#L745

Alternatively, could we transfer the xsmr to litter pools, instead of the atmosphere? hrv_deadcrootc_to_litter(p) = deadcrootc(p) m hrv_deadcrootc_to_litter(p) = hrv_deadcrootc_to_litter(p) + xsmrpool(p) m https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/dynHarvestMod.F90#L341

Neither of these are very satisfying fixes, so other ideas would be welcome

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-445330587, or mute the thread https://github.com/notifications/unsubscribe-auth/AUAcVMPaTDvxVfieTiT_VcwZKJW90xfXks5u2rnegaJpZM4ZH0Ek .

[wwieder]

@dlawrenncar & @danicalombardozzi I've gotten lost in the crop phenology code, but don't really understand why these fluxes are zeroed out in CNCStateUpdate1? I'd assume that this is the flux giving us a large negative AR flux?

https://github.com/ESCOMP/ctsm/blob/7755baf3f729ff9adb0e39088c02113bbc9ebb58/src/biogeochem/CNCStateUpdate1Mod.F90#L480

[wwieder]

@klindsay28 can you rerun your high frequency output simulations with this block of code around line 1540 of CNVegCarbonFluxType.F90? I'd like to confirm that crop harvest is causing the negative AR fluxes and big drop in CO2 concentrations.

    this%ar_patch(begp:endp) = spval
    call hist_addfld1d (fname='AR', units='gC/m^2/s', &
         avgflag='A', long_name='autotrophic respiration (MR + GR)', &
         ptr_patch=this%ar_patch)

! -- WW added for #590 this%hrv_xsmrpool_to_atm_patch(begp:endp) = spval call hist_addfld1d (fname='HRV_XSMR_TO_ATM', units='gC/m^2/s', & avgflag='A', long_name='harvest xsmr to atm', & ptr_patch=this%hrv_xsmrpool_to_atm)

    this%xsmrpool_to_atm_patch(begp:endp) = spval
    call hist_addfld1d (fname='XSMR_TO_ATM', units='gC/m^2/s', &
         avgflag='A', long_name='xsmr to atm', &
         ptr_patch=this%xsmrpool_to_atm)

! -- WW added

    this%npp_patch(begp:endp) = spval
    call hist_addfld1d (fname='NPP', units='gC/m^2/s', &
         avgflag='A', long_name='net primary production', &
         ptr_patch=this%npp_patch)

[klindsay28]

Sure thing @wwieder. The run is now in progress. I went ahead and added the new fields to the h1 and h2 tapes too (per pft and landunit respectively), just in case you want to see these terms subgridcell.

[ckoven]

just got off a plane and saw this. nothing obvious comes to mind to me.

[slevis-lmwg]

Sorry for the delayed response, though I'm afraid I don't have suggestions at this time.

[wwieder]

@klindsay28 's runs confirm that the large negative flux is coming from xsmrpool_to_atm_patch, which occurs at harvest in a single time step. You can see a map here /glade/scratch/wwieder/negAR_test.nc

The magnitude of the negative flux is really big, compared to MR and GR fluxes, but could occur with XSMRPOOL ~ -20 gC/m2 (assuming 100% crop coverage over the gridcell).

It seems this is the code responsible for the large instantaneous flux: CNCStateUpdate1Mod.F90:
cf_veg%xsmrpool_to_atm_patch(p) = cf_veg%xsmrpool_to_atm_patch(p) + cs_veg%xsmrpool_patch(p)/dt

@dlawrenncar and @klindsay28 I'm a little unsure of the code modifications that would be preferred here, or the timeframe we have to work with? One patch would be to dribbling this flux to the atmosphere, as with wood harvest. This would likely alleviate the large negative AR fluxes that are causing the prognostic CO2 runs to fail. Alternatively, the harvest xsmr flux could be passed to litter, instead of the atmosphere, but I'm unsure if this could generate negative litter pools in some cases and would certainly change answers. Neither patch addresses the underlying issue with large xsmr pools that are accumulating (esp. in crops).

more of Keith's simulation can be found here if you're interested, which includes pft-level h1 files if we want to dive into the causes. /glade/scratch/klindsay/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.001/run

[dlawrenncar]

@wwieder Recalling @klindsay28's statement at the beginning of thread. The atmospheric CO2 drops down to something like 4ppm. A huge, huge drop, right? Maybe I am not understanding something, but if we dribbled the negative AR flux out over a year, we would still see a massive drop in CO2. My guess is still that what is happening here in the crash is an extreme version with a crazy high negative AR flux that is not really similar to the more standardly poor negative AR fluxes around harvest that are occurring frequently. @wwieder do you agree?

[klindsay28]

@dlawrenncar, I think dribbling has the potential to be effective. While there was an O(280) ppm drop in a2x_Sa_coprog, this was at a single grid cell. The blip that led to the abort occurred at local night, so I suspect that the effect of the CO2 uptake was concentrated into a shallow boundary layer, enhancing the effect's magnitude. If the flux were dribbled out over time, it would give atmospheric transport time to distribute the anomaly, both laterally and vertically. It doesn't take too much transport to reduce the effect's impact by 100x, which would be manageable.

[dlawrenncar]

@klindsay28 Thanks Keith. Yes, after discussing with Will just now, we agree with you that the flux dribbler this has the potential to solve the issue. @olyson Can you attempt to implement the flux dribbler solution this week? Longer term, we need to resolve the accumulation/occurrence of negative XSMRPOOL. Jinyun has thoughts about that so we will follow up with him.

[rosiealice]

That's a good point, about the shallow boundary layer. I was beginning to think everything was perhaps broken/ suffering massive silent XSMR issues...

n.b. that in FATES I resolved this (large carbon deficits killing things too much in marginal/seasonal/inter-annually varying places) by only allowing some (parameterized) fraction of the C demand from tissue turnover to be met, and allowing the remaining tissue to senesce, when NPP was negative. Thus, the live pools decline and so do respiratory costs, so the accumulation of carbon debt goes down. That means you get, say, seasonal oscillations in LAI instead of seasonal oscillations in carbon debt/storage/XSMR.

Obviously the carbon allocation model would need to look different in CLM5 to accomodate that, The only other options I can see are 1) declining maintenance respiration as a function of XSMR (i thought we did that already?) and 2) adding some sort of specifically enhanced tissue turnover to mimic the mortality which would result in real life from running a huge carbon deficit.

Le sam. 8 déc. 2018 à 15:06, David Lawrence notifications@github.com a écrit :

@klindsay28 https://github.com/klindsay28 Thanks Keith. Yes, after discussing with Will just now, we agree with you that the flux dribbler this has the potential to solve the issue. @olyson https://github.com/olyson Can you attempt to implement the flux dribbler solution this week? Longer term, we need to resolve the accumulation/occurrence of negative XSMRPOOL. Jinyun has thoughts about that so we will follow up with him.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-445461786, or mute the thread https://github.com/notifications/unsubscribe-auth/AMWsQzhWg424pGu2zMq8JoFZ4cZ1rz8Nks5u28dOgaJpZM4ZH0Ek .

--

Dr Rosie A. Fisher

Staff Scientist Terrestrial Sciences Section Climate and Global Dynamics National Center for Atmospheric Research 1850 Table Mesa Drive Boulder, Colorado, 80305, USA

and

Visitor @ C.E.R.F.A.C.S Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique 42 Avenue Gaspard Coriolis 31057, Toulouse, France http://www.cgd.ucar.edu/staff/rfisher/

[klindsay28]

The line of code pointed to by @wwieder in CNCStateUpdate1Mod.F90 is preceded with the block of comments below. It's not clear to me if the concerns mentioned in these comments are applicable to xsmrpool. If so, it might be prudent to keep on eye on isotopes when dribbling is enabled. (The coupled runs do have land C isotopes enabled.)

! DML (06-20-2017) While debugging crop isotope code, found that cpool_patch and frootc_patch ! could occasionally be very small but nonzero numbers after crop harvest, which persists ! through to next planting and for reasons that could not 100% ! isolate, caused C12/C13 ratios to occasionally go out of ! bounds. Zeroing out these small pools and putting them into the flux to the ! atmosphere solved many of the crop isotope problems

[wwieder]

one more favor @klindsay28 can you repeat the high run one more time with XSMRPOOL added to the h1 and h2 files? I'd like to see how big some of the crop XSMRPOOLs get?

[klindsay28]

Sure. The rerun with XSMRPOOL in the h1 and h2 files is complete.

[wwieder]

Here's a snapshot of XSMR pools in the time step just before the model crashes. I'm suspecting many of the zero values are regions where harvest has already occurred for the growing season, but locally XSMR pools can be fairly large with particular crop types.
b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.001_CropXSMRPOOL_0004-09-28-00000.pdf

[wwieder]

same as above, but hopefully with display in the issue?

[olyson]

Ok, as soon as I figure out what a flux dribbler is...

[ekluzek]

@olyson @billsacks can give more details on this. But, basically a flux dribbler takes the yearly land-use change flux and "dribbles" it over the year. So instead of doing a once a year big whack it's done more slowly.

[billsacks]

Sorry, I missed the earlier query to me about the flux dribbler. I haven't followed this whole discussion, but from a skim through it, I don't think the dribbler solution is appropriate here: These annual flux dribblers are designed for one particular purpose, which is to smooth fluxes that are generated at the first time step of the year. They are unable to handle fluxes generated at any other point in the year: I couldn't come up with an easy / efficient way to implement something like this this that could accept fluxes at any point in the year.

We do have somewhat similar infrastructure for gradually releasing other fluxes that can be generated at any point in the year. See CNProductsMod: subroutine UpdateProducts. (I forget if we have other, similar examples that might be simpler than that.) The key difference is that the annual flux dribbler guarantees that the entire source flux is released throughout the following year, whereas the product pool fluxes use an exponential decay, so the source flux will never be completely released to the atmosphere.

[wwieder]

Seems like it makes sense to handle this XSMR pool as we are grain harvest in UpdateProducts as suggested by @billsacks. This requires generating a new pool to handle harvested XSMR and since we don't want to generate another large negative C pool over time maybe it should have even a faster decay rate, roughly 0.5 years?

kprod05 = 1.44e-7

Does this plan seem reasonable @dlawrenncar & @olyson?

[dlawrenncar]

That seems like a potentially reasonable idea to add this to the crop 1yr product pool or to setup a new 1yr pool if we want to avoid messing with the crop product pool.

On Dec 10, 2018, at 7:55 AM, Bill Sacks notifications@github.com wrote:

Sorry, I missed the earlier query to me about the flux dribbler. I haven't followed this whole discussion, but from a skim through it, I don't think the dribbler solution is appropriate here: These annual flux dribblers are designed for one particular purpose, which is to smooth fluxes that are generated at the first time step of the year. They are unable to handle fluxes generated at any other point in the year: I couldn't come up with an easy / efficient way to implement something like this this that could accept fluxes at any point in the year.

We do have somewhat similar infrastructure for gradually releasing other fluxes that can be generated at any point in the year. See CNProductsMod: subroutine UpdateProducts. (I forget if we have other, similar examples that might be simpler than that.) The key difference is that the annual flux dribbler guarantees that the entire source flux is released throughout the following year, whereas the product pool fluxes use an exponential decay, so the source flux will never be completely released to the atmosphere.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-445805585, or mute the thread https://github.com/notifications/unsubscribe-auth/AUAcVKYLxvb9Qmo-ybjvp7vDge7AU8B3ks5u3lnMgaJpZM4ZH0Ek .

[olyson]

I'm not sure I understand how to do this, but I thought I'd start by showing some pseudo-code below that people can comment on (this is in CNCStateUpdate1Mod.F90). Although it seems like this would break carbon balance.

     if (ivt(p) >= npcropmin) then ! skip 2 generic crops
        cs_veg%xsmrpool_patch(p) = cs_veg%xsmrpool_patch(p) - cf_veg%livestem_xsmr_patch(p)*dt
        cs_veg%xsmrpool_patch(p) = cs_veg%xsmrpool_patch(p) - cf_veg%grain_xsmr_patch(p)*dt
        if (harvdate(p) < 999) then ! beginning at harvest, send to atm
           ! TODO (mv, 11-02-2014) the following lines are why the cf_veg is
           ! an intent(inout)
           ! fluxes should not be updated in this module - not sure where
           ! this belongs
           ! DML (06-20-2017) While debugging crop isotope code, found that cpool_patch and frootc_patch
           ! could occasionally be very small but nonzero numbers after crop harvest, which persists
           ! through to next planting and for reasons that could not 100%
           ! isolate, caused C12/C13 ratios to occasionally go out of
           ! bounds. Zeroing out these small pools and putting them into the flux to the
           ! atmosphere solved many of the crop isotope problems

! KO

           ! Save xsmrpool to loss for dribbling
           cs_veg%xsmrpool_loss_patch(p) = cs_veg%xsmrpool_patch(p)

! KO

! KO cf_veg%xsmrpool_to_atm_patch(p) = cf_veg%xsmrpool_to_atm_patch(p) + cs_veg%xsmrpool_patch(p)/dt

           cs_veg%xsmrpool_patch(p)        = 0._r8
           cf_veg%xsmrpool_to_atm_patch(p) = cf_veg%xsmrpool_to_atm_patch(p) + cs_veg%cpool_patch(p)/dt
           cs_veg%cpool_patch(p)           = 0._r8
           cf_veg%xsmrpool_to_atm_patch(p) = cf_veg%xsmrpool_to_atm_patch(p) + cs_veg%frootc_patch(p)/dt
           cs_veg%frootc_patch(p)          = 0._r8
        end if

! KO

        ! the following (1/s) rate constant results in ~90% loss of initial state over 0.5 years,
        ! using a discrete-time fractional decay algorithm.
        kprod05 = 1.44e-7
        ! calculate flux out of xsmrpool
        cf_veg%xsmrpool_loss_to_atm_patch(p) = cs_veg%xsmrpool_loss_patch(p) * kprod05
        ! add flux to atmosphere
        cf_veg%xsmrpool_to_atm_patch(p) = cf_veg%xsmrpool_to_atm_patch(p) + cf_veg%xsmrpool_loss_to_atm_patch(p)
        ! update xsmrpool loss state variable
        cs_veg%xsmrpool_loss_patch(p) = cs_veg%xsmrpool_loss_patch(p) - cf_veg%xsmrpool_loss_to_atm_patch(p) * dt

! KO

     end if

[olyson]

I've moved on to incorporating this into the balance check and got a two year land-only run completed. I spot-checked the AR in the monthly average history files, max negative values are much smaller than before, on the order of -e-23. I moved this over to my clone of Keith's case and ran successfully past the error for one month. Again, a spot check of max AR indicates values of -e-23. I probably need to rerun with cpl output to see what a2x_Sa_coprog is doing, my clone didn't have the settings for this output. I haven't dealt with isotopes or restarts. I'm also not positive that carbon is balancing for the right reasons or because I've made it balance trivially. And the code should probably be moved over to CNProducts, it was just easier to develop the code in CNCStateUpdate1.

[olyson]

I"m just going to point you to some code. A comparison of my branch with these changes compared to the latest clm release tag is here:

https://github.com/ESCOMP/ctsm/compare/release-clm5.0...olyson:hrv_xsmrpool_to_atm

If it helps to look at the full modules, the code I used for an offline historical is here:

/gpfs/fs1/work/oleson/release-clm5.0.15_hrv_xsmrpool_to_atm/src

The sourcemods I used for my clone of Keith's case are here:

/gpfs/fs1/work/oleson/cesm2_0_runs/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.001/SourceMods/src.clm

These mods differ from my branch in that I had to comment out the new restart variables since Keith's run was a branch.

I'm going to run a pair of short land-only historical runs to check energy/water/flux changes as suggested by Will.

[olyson]

As discussed I've added a gridcell average xsmrpool_to_atm flux to nbp. Seems to have reduced the differences in NBP by a couple of orders of magnitude. Before:

http://webext.cgd.ucar.edu/I20TR/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.nonbp/lnd/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.1855_1864-clm50_release-clm5.0.15_1deg_GSWP3V1_hist.1855_1864/set2/set2_ANN_NBP.png

After:

http://webext.cgd.ucar.edu/I20TR/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist/lnd/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.1855_1864-clm50_release-clm5.0.15_1deg_GSWP3V1_hist.1855_1864/set2/set2_ANN_NBP.png

And the global average is nearly the same as the control now. Before:

http://webext.cgd.ucar.edu/I20TR/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.nonbp/lnd/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.1855_1864-clm50_release-clm5.0.15_1deg_GSWP3V1_hist.1855_1864/set1/set1_NBP.png

After:

http://webext.cgd.ucar.edu/I20TR/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist/lnd/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.1855_1864-clm50_release-clm5.0.15_1deg_GSWP3V1_hist.1855_1864/set1/set1_NBP.png

[wwieder]

Seems like this is as good as can be expected. I'm assuming Keith lindsay's case now continues without the co2 error?

On Wed, Dec 19, 2018, 7:26 AM Keith Oleson <notifications@github.com wrote:

As discussed I've added a gridcell average xsmrpool_to_atm flux to nbp. Seems to have reduced the differences in NBP by a couple of orders of magnitude. Before:

http://webext.cgd.ucar.edu/I20TR/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.nonbp/lnd/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.1855_1864-clm50_release-clm5.0.15_1deg_GSWP3V1_hist.1855_1864/set2/set2_ANN_NBP.png

After:

http://webext.cgd.ucar.edu/I20TR/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist/lnd/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.1855_1864-clm50_release-clm5.0.15_1deg_GSWP3V1_hist.1855_1864/set2/set2_ANN_NBP.png

And the global average is nearly the same as the control now. Before:

http://webext.cgd.ucar.edu/I20TR/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.nonbp/lnd/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.1855_1864-clm50_release-clm5.0.15_1deg_GSWP3V1_hist.1855_1864/set1/set1_NBP.png

After:

http://webext.cgd.ucar.edu/I20TR/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist/lnd/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.1855_1864-clm50_release-clm5.0.15_1deg_GSWP3V1_hist.1855_1864/set1/set1_NBP.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-448614068, or mute the thread https://github.com/notifications/unsubscribe-auth/AHqLJPCcwP1xGDRaIS2eSme-ZQtSNfGaks5u6kyjgaJpZM4ZH0Ek .

[olyson]

I'll give it a try.

[olyson]

Keith's case ran to completion for 1 month. The lowest a2x_Sa_coprog timestep value I see for the last five days of the month is 261.153 ppm.

[wwieder]

seems like the fix is doing it's job. What do you think, Dave?

On Wed, Dec 19, 2018 at 10:59 AM Keith Oleson notifications@github.com wrote:

Keith's case ran to completion for 1 month. The lowest a2x_Sa_coprog timestep value I see for the last five days of the month is 261.153 ppm.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-448688284, or mute the thread https://github.com/notifications/unsubscribe-auth/AHqLJAxlPw3amZc1IRsWAeZNilw4njUAks5u6n5-gaJpZM4ZH0Ek .

-- Will Wieder Project Scientist CGD, NCAR 303-497-1352

[dlawrenncar]

Yep. I think we should pass this back over to Keith Lindsay so that he can run a longer simulation. Pending the outcome of that simulation, we can then make a decision about how to proceed in the context of CESM2 CMIP6 simulations.

On Wed, Dec 19, 2018 at 11:17 AM will wieder notifications@github.com wrote:

seems like the fix is doing it's job. What do you think, Dave?

On Wed, Dec 19, 2018 at 10:59 AM Keith Oleson notifications@github.com wrote:

Keith's case ran to completion for 1 month. The lowest a2x_Sa_coprog timestep value I see for the last five days of the month is 261.153 ppm.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-448688284, or mute the thread < https://github.com/notifications/unsubscribe-auth/AHqLJAxlPw3amZc1IRsWAeZNilw4njUAks5u6n5-gaJpZM4ZH0Ek

.

-- Will Wieder Project Scientist CGD, NCAR 303-497-1352

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/ctsm/issues/590#issuecomment-448694031, or mute the thread https://github.com/notifications/unsubscribe-auth/AUAcVCCEylNv9BrZOAqcg3DAgYipzR9nks5u6oLOgaJpZM4ZH0Ek .

[olyson]

Keith,

I have source code for your simulation to try here:

/gpfs/fs1/work/oleson/cesm2_0_runs/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.001/SourceMods/src.clm

Note that I added a few restart variables in CNVegCarbonStateType.F90. But I commented those lines out to run my clone of your simulation which was a branch. If you are not running a branch then you would need to uncomment those lines. They are marked/bracketed by "!KO".

Keith

[klindsay28]

Keith, I'm setting up a new case with your mods. CASEROOT=/glade/work/klindsay/cesm20_cases/B1850/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.002 It is a hybrid off of the CMIP6 piControl. It's not clear to me what I should be doing regarding the new restart variables. After running preview_namelist, lnd_in the new case includes the line: finidat = 'b.e21.B1850.f09_g17.CMIP6-piControl.001.clm2.r.0501-01-01-00000.nc' If I include the new restart variables, will the new case be able to start from the above file? If I don't include the new restart variables, will the new case restart exactly? Keith

[olyson]

The new case should be able to start from the above file even if you include the new restart variables. My offline historical case starts from an initial file that doesn't have those restart variables on it. So I would uncomment the lines in CNVegCarbonStateType.F90 and try it.

[olyson]

From Keith Lindsay:

The coupled model run with CLM mods to avoid the previous aborts is nearly out 40 years. Good news: The fix appears to have solved the problem with the aborts.

Attached are some timeseries plots of globally averaged annual mean CO2 surface values and surface fluxes (+ up) (plots not attached here). The lower panel has a 9 year filter.

It looks like the system might be stabilizing atm CO2, but with a steady transfer of C from lnd to ocn, via atm, at a rate of about 0.1 PgC/y.

Could someone please run LMWG diags on this case, so that we can look for anything suspicious?

CASE b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.002 CASEROOT /glade/work/klindsay/cesm20_cases/B1850/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.002 DOUT_S_ROOT /glade/scratch/klindsay/archive/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.002

The run is a hybrid off of b.e21.B1850.f09_g17.CMIP6-piControl.001 at 0501-01-01, so maybe it makes sense to compare years 1-30 of the new run to 501-531 of b.e21.B1850.f09_g17.CMIP6-piControl.001.

[olyson]

The land diagnostics for the coupled runs are here:

http://webext.cgd.ucar.edu/B1850/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.002/lnd/b.e21.B1850_BPRP.f09_g17.CMIP6-piControl.tst.002.1_30-b.e21.B1850.f09_g17.CMIP6-piControl.001.501_530/setsIndex.html

The diagnostics for the 20-year land-only historical run I did are here:

http://webext.cgd.ucar.edu/I20TR/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist/lnd/clm50_release-clm5.0.15_hrv_xsmrpool_to_atm_1deg_GSWP3V1_hist.1855_1864-clm50_release-clm5.0.15_1deg_GSWP3V1_hist.1855_1864/setsIndex.html