FATES hydro mass-balance issues

Brief summary of bug

FATES-hydro is a special mode where vegetation can uptake and store water via the transpiration stream. This requires an accounting of how much water is stored in FATES vegetation, which must be passed back to CLM for its water mass balance checking. The water mass balance check (BalanceCheckMod.F90 at line 760) is failing. Details below.

General bug information

CTSM version you are using: [output of git describe]

hash: 3ac4305
tag: ctsm5.1.dev029-3-g3ac43057

Does this bug cause significantly incorrect results in the model's science? [Yes / No]

Not sure, but I don't think so.

Configurations affected: [Fill this in if known.]

These namelist options must be set:

use_fates = .true. use_fates_planthydro = .true.

Details of bug

I generate this error using a specialized site setup for Barro Colorado Island Panama, where I also initialize a forest stand structure from inventory data. The simulation runs 26+ months and then fails with the following error:

 clm: calling FATES model        38449
FATES Dynamics: 1902-03-13
 dbh counter:           13  is woody:  T
 dbh counter:           13  is woody:  T
 dbh counter:           17  is woody:  T
 dbh counter:           17  is woody:  T
 clm: leaving fates model           1           1
 WARNING:  grid cell-level water balance error  nstep=        38449  local indexg=            1  errh2o_grc=   -2.2061354929889532E-003
 CTSM is stopping because errh2o >    1.0000000000000001E-005  mm
 nstep                     =        38449
 errh2o_grc                =   -2.2061354929889532E-003
 forc_rain                 =    0.0000000000000000     
 forc_snow                 =    0.0000000000000000     
 endwb_grc                 =    2989.3396629148806     
 begwb_grc                 =    2989.3595745851790     
 qflx_evap_tot             =    1.9854765757965773E-002
 qflx_sfc_irrig            =    0.0000000000000000     
 qflx_surf                 =    0.0000000000000000     
 qflx_qrgwl                =   -2.1492309525845731E-003
 qflx_drain                =    0.0000000000000000     
 qflx_ice_runoff           =    0.0000000000000000     
 qflx_snwcp_discarded_ice  =    0.0000000000000000     
 qflx_snwcp_discarded_liq  =    0.0000000000000000     
 deltawb                   =   -1.9911670298370154E-002
 deltawb/dtime             =   -1.1062039054650086E-005
 qflx_drain_perched        =    0.0000000000000000     
 forc_flood                =   -0.0000000000000000     
 qflx_glcice_dyn_water_flux =    0.0000000000000000     
 CTSM is stopping
 local  gridcell index =            1
 global gridcell index =            1
 gridcell longitude    =    280.15390000000002     
 gridcell latitude     =    9.1530000000000005     
 ENDRUN:
 ERROR: ERROR in /raid1/rgknox/SyncLRC/ctsm/src/biogeophys/BalanceCheckMod.F90 at line 760

From the log, I can see that the check is failing after a FATES dynamics call. I have to investigate further to see a call to FATES hydraulics follows. If only the dynamics call has transpired (pun intended) prior to the balance check fail, then we should expect that that the total water stored in vegetation should had remained unchanged during the process. During dynamics, we only shuffle water around internally in FATES, but should conserve it. We do have a check in place to verify this (but checks can be wrong too of course).

Key variables that FATES passes to CLM are:

The total water stored in vegetation, an example of the transfer found here: https://github.com/ESCOMP/CTSM/blob/master/src/utils/clmfates_interfaceMod.F90#L1011-L1012

The water uptake by roots: https://github.com/ESCOMP/CTSM/blob/master/src/utils/clmfates_interfaceMod.F90#L2632

Key variable that CLM passes to FATES: Transpiration rate of the patch: https://github.com/ESCOMP/CTSM/blob/master/src/utils/clmfates_interfaceMod.F90#L2752

Important details of your setup / configuration so we can reproduce the bug

I will need to reproduce this error on a supported resolution, this error was generated on a user-defined single cell grid.

Here is the compset and key namelist variables though: 2000_DATM%CRUv7_CLM50%FATES_SICE_SOCN_SROF_SGLC_SWAV use_fates = .true. use_fates_planthydro = .true.

ESCOMP / CTSM