Carbon balance error

[jkshuman]

In testing new fire mortality diagnostics, I updated to the current form of master (commit 81ca2d4). I am now getting a carbon balance error in the first year of simulation with and without fire using a 2PFT file with tropical trees and grass across a 1 degree grid in South America. The error occurs for the base version of master, so this is not a result of my new branch with fire mortality diagnostics changes. Same error, date, and location with DEBUG set to true.

cesm.log: carbon balance error detected carbon balance error detected error fraction relative to biomass stock: 4.539406930433628E-005 call index: 1 flux in (npp): 0.745153098471362
flux out (fragmentation/harvest): 3.359360275547218E-002 net: 0.711559495715890
dstock: 0.703577932072193
error=net_flux-dstock: 7.981563643697065E-003 biomass 157.806828360285
litter 16.5292936942136
seeds 1.49221336496183
previous total 175.124757487388
lat lon -30.6282722513089 295.000000000000
aborting on date: 1 4 21 carbon balance error detected error fraction relative to biomass stock: 1.172474974742245E-005 error fraction relative to biomass stock: 6.306847025935279E-005 call index: 1 flux in (npp): 1.02526755514027
flux out (fragmentation/harvest): 3.205436931665112E-002 net: 0.993213185823620
dstock: 0.980130826197353
error=net_flux-dstock: 1.308235962626658E-002 error fraction relative to biomass stock: 3.443964011212682E-005 carbon balance error detected error fraction relative to biomass stock: 4.137384756892985E-004 seeds 4.74976125894274
previous total 213.836875513445
lat lon -36.2827225130890 302.500000000000
aborting on date: 1 4 21 ENDRUN: ERROR in EDMainMod.F90 at line 537

[jkshuman]

while I figure out how to post these things to git... path to parameter file: /glade/work/jkshuman/FATES_data/parameter_files/param_file_2PFT_Obrienh_Saldaa_Saldal_10032018.nc path to script: /glade/work/jkshuman/FATES_data/case_1deg_tropics Run script: (caveat that this is the Cheyenne version, while my runs were on Hobart.)

#!/usr/bin/env bash

SRCDIR=/glade/work/jkshuman/git/ctsm
cd ${SRCDIR}
### CTSM git version 
GITHASH1=`git log -n 1 --format=%h`

cd src/fates
### FATES git version
GITHASH2=`git log -n 1 --format=%h`

### repeat runs with fire
PARAM=TreeGrass
#PARAM=TreeGrass_clm45

SETUP_CASE=${PARAM}_1x1

PARAM_FILE=param_file_2PFT_Obrienh_Saldaa_Saldal_10032018.nc

PARAM_DIR=/glade/work/jkshuman/FATES_data/parameter_files/

CASE_NAME=${SETUP_CASE}_${GITHASH1}_${GITHASH2}

casedir=/glade/work/jkshuman/FATES_cases/Param_test/

basedir=$SRCDIR/cime/scripts/

##LMWG project numbers
#project=93300041
project=93300641

#### load_machine_files
cd $basedir

./create_newcase --case ${casedir}${CASE_NAME} --res f09_g16 --compset 2000_DATM%GSWP3v1_CLM50%FATES_SICE_SOCN_RTM_SGLC_SWAV --run-unsupported
#./create_newcase --case ${casedir}${CASE_NAME} --res f09_g16 --compset 2000_DATM%GSWP3v1_CLM45%FATES_SICE_SOCN_RTM_SGLC_SWAV --run-unsupported

cd ${casedir}${CASE_NAME}

export CLM_DOMAIN_DIR=/glade/work/jkshuman/sfcdata
export CLM_SURFDAT_DIR=/glade/work/jkshuman/sfcdata

##export SITE_NAME=Tropics
##export CLM_USRDAT_DOMAIN=domain.lnd.fv0.9x1.25_gx1v6.Tropics.nc 
##export CLM_USRDAT_SURDAT=surfdata_0.9x1.25_16pfts_Irrig_CMIP6_simyr2000_Tropics.nc

export SITE_NAME=SAmerica
export CLM_USRDAT_DOMAIN=domain.lnd.fv0.9x1.25_gx1v6.SA.nc 
export CLM_USRDAT_SURDAT=surfdata_0.9x1.25_16pfts_Irrig_CMIP6_simyr2000_SA.nc

##modify env_mach_pes file
./xmlchange NTASKS_ATM=-1
./xmlchange NTASKS_CPL=-15
./xmlchange NTASKS_GLC=-15
./xmlchange NTASKS_OCN=-15
./xmlchange NTASKS_WAV=-15
./xmlchange NTASKS_ICE=-15
./xmlchange NTASKS_LND=-15
./xmlchange NTASKS_ROF=-15
./xmlchange NTASKS_ESP=-15

./case.setup

##modify env_run
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=1
./xmlchange REST_OPTION=nyears
./xmlchange RESUBMIT=10

./xmlchange JOB_WALLCLOCK_TIME=1:00
./xmlchange JOB_QUEUE=economy

##./xmlchange CONTINUE_RUN=TRUE
##./xmlchange DEBUG=FALSE

./xmlchange DATM_MODE=CLMGSWP3v1
./xmlchange DATM_CLMNCEP_YR_ALIGN=1985
./xmlchange DATM_CLMNCEP_YR_START=1985
./xmlchange DATM_CLMNCEP_YR_END=2004

./xmlchange RTM_MODE=NULL
./xmlchange ATM_DOMAIN_FILE=${CLM_USRDAT_DOMAIN}
./xmlchange ATM_DOMAIN_PATH=${CLM_DOMAIN_DIR}
./xmlchange LND_DOMAIN_FILE=${CLM_USRDAT_DOMAIN}
./xmlchange LND_DOMAIN_PATH=${CLM_DOMAIN_DIR}
./xmlchange CLM_USRDAT_NAME=${SITE_NAME}

cat > user_nl_clm <<EOF

fsurdat = '${CLM_SURFDAT_DIR}/${CLM_USRDAT_SURDAT}'
fates_paramfile = '${PARAM_DIR}${PARAM_FILE}'
hist_fincl1 =
'M1_SCPF','M2_SCPF','M3_SCPF','M4_SCPF','M5_SCPF','M6_SCPF','MORTALITY_CANOPY_SCPF','MORTALITY_UNDERSTORY_SCPF','Fire_Closs','PATCH_AREA_BY_AGE','CANOPY_AREA_BY_AGE','BIOMASS_BY_AGE',\
'PFTcrownarea','DDBH_CANOPY_SCPF','DDBH_UNDERSTORY_SCPF','NPLANT_CANOPY_SCPF','NPLANT_UNDERSTORY_SCPF','BSTOR_CANOPY_SCPF','BSTOR_UNDERSTORY_SCPF','CWD_AG_CWDSC',\
'NPLANT_SCAG','NPLANT_SCPF','DDBH_SCPF','BA_SCPF','RECRUITMENT'
use_fates=.true.
use_fates_spitfire=.false. 
EOF

qcmd -- ./case.build
qcmd -- ./case.submit

[jkshuman]

Tried this again for FATES master on Hobart with an adjusted error tolerance of 10 -4 rather than 10 -6 in EDMainMod. Same type of failure and date of failure for both a fire and non-fire run.

[rgknox]

@jkshuman and I have had some back-and-forths on this issue. I'm trying to reproduce the error as well. One thing that caught my eye is that the minimum height for the grass recruits is quite small.

fates_recruit_hgt_min = 1.25, 1.25, 0.125 ;

I can't think of specific reason why this would trigger a carbon balance error, but for such a small plant, the target proportions for each organ might come out strange.

[jkshuman]

Thanks, @rgknox I tried this with a larger grass recruit size, and got the same carbon balance error. At the same location from above, the error is slightly bigger though (if I am reading this correctly). I also tested this on Cheyenne and it throws the same failure. error on Hobart for test with grass recruit set to 0.625 (50% of max height):

 carbon balance error detected
 flux in (npp):     1.09877630250704     
 net:   0.683787632195564     
 dstock:   0.452209783818290     
 error=net_flux-dstock:  0.231577848377273     
 biomass   168.408284889331     
 litter   24.7667138127425     
 seeds   1.43993511919627     
 error fraction relative to biomass stock:  5.894515681120725E-004
 error fraction relative to biomass stock:  1.491202571983473E-003
 call index:            1
 flux in (npp):    0.533777277337335     
 flux out (fragmentation/harvest):   5.285672719780393E-002
 net:   0.480920550139531     
 dstock:   0.197051221737865     
 error=net_flux-dstock:  0.283869328401665     
 flux out (fragmentation/harvest):   4.666213578276053E-002
 previous total   194.162724037452     
 lat lon  -30.6282722513089        295.000000000000     
 aborting on date:           1           4          21
 carbon balance error detected
 biomass   164.834552429934     
 litter   24.5179635639208     
 seeds   1.01017096428694     
 previous total   190.165635736404     
 lat lon  -26.8586387434555        296.250000000000     
 aborting on date:           1           4          21
 net:    1.05211416672428     
 ENDRUN:
 ERROR in EDMainMod.F90 at line 537                                             

[rgknox]

@jkshuman I tried your script and generated the same run on Cheyenne, but using the new parteh branch. The parteh changes forced me to do a little cleaning on various things carbon balance. This run is in year 82 right now, maybe might solve your problem?

updated parameter file: /gpfs/u/home/rgknox/ctsm/src/fates/parameter_files/param_file_3pft_ObSalda_parteh_181007.nc

my script: /gpfs/u/home/rgknox/ctsm/cime/scripts/create_jks_3pfttest.sh

*Important note: Need new nl parameter: fates_parteh_mode=1

ctsm branch: https://github.com/rgknox/ctsm/tree/fates-parteh-api fates branch: https://github.com/rgknox/fates/tree/rgknox-parteh

[jkshuman]

Based on @rgknox success using parteh for this error, I created a merge of @rgknox parteh branch with master (master hash commit b2e89f3 with the fire_leaf_flux updates), and it is showing success. Into years 9 and 10 with reasonable accumulation for both fire and non-fire runs. I previously had an error/issue because I was using an old parameter file and parteh requires updates to the parameter file. I am going to run these out to year 100 on Hobart to confirm this parteh-master branch is working, make a few more fixes, and move forward with my runs.

@rgknox, @ckoven @rosiealice we should discuss if we can close this issue based on the use of parteh, or if it is worth keeping open and exploring the carbon balance error on master from my file/case set. This parteh-master merge branch also serves as an endorsement for parteh use, and a discussion point for testing and using parteh more broadly.

[rosiealice]

That's great. Just so I understand, the state of things is that the non-parteh fire mortality update has a c balance error, but the post-parteh version does not. Therefore we suspect that parteh has fixed the problem with it's better consideration of C allocation etc. And so we need to decide whether to fix the bug in the non-parteh version? My feeling is that you should perhaps push on with the fire runs with the working version, but perhaps note here or elsewhere that this issue might exist if one turned parteh off. Do we plan to keep/maintain the pre-parteh version of FATES?

[rgknox]

@rosiealice , we could maintain a pre-parteh branch. But once parteh is pulled into master, I would not see it as an optional thing that could be switched on or off (of course the different hypotheses inside can).

[rosiealice]

I wasn't really advocating that we keep a pre-parteh branch. Particularly if it plausibly fixes pre existing c accounting issues...

On Thu, Oct 11, 2018, 8:43 PM Ryan Knox notifications@github.com wrote:

@rosiealice https://github.com/rosiealice , we could maintain a pre-parteh branch. But once parteh is pulled into master, I would not see it as an optional thing that could be switched on or off (of course the different hypotheses inside can).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NGEET/fates/issues/424#issuecomment-429074253, or mute the thread https://github.com/notifications/unsubscribe-auth/AMWsQ0uY1_n50z8z4LdQwHBVo6p9Pa1Eks5uj5FtgaJpZM4XI2lr .

[jkshuman]

Closing this issue based on the success of the use of the master branch with the parteh branch carbon allocation tracking (described above, and parteh in detail from @rgknox in issue #421 ). Test runs on NCAR Hobart machine are successfully into year 96 for the control without fire, and year 101 with active fire for a 2 PFT (grass, tropical tree) regional 1 degree run across South America with the GSWP3 forcing data and a bare ground start. As @rosiealice mentioned there is still the possibility of an underlying carbon balance issue with the master branch, but I am not going down the rabbit hole of finding that error since parteh "fixes it" for my file set and setup. Parteh is on its way into master, as the parteh branch is being further tested by @rgknox (see issue #421). I plan on moving forward with my science runs with parteh active and a recent master hash (commit b2e89f3 with the fire_leaf_flux updates ).