bug: CLM water/snow/radiation/energy balance checks turned off when DA turned on

braczka commented 5 months ago

:bug:

Describe the bug

List the steps someone needs to take to reproduce the bug.
Run CLM-DART with version cesm2.2.0 versions or later. For example, run CLM-DART tutorial. Run CLM-DART with DA enabled (i.e. DATA_ASSIMILATION_LAND=TRUE)
What was the expected outcome?

The balance checks within CLM source code ~/src/biogeophys/BalanceCheckMod.F90 should be enabled after skipping a time step or two after DA analysis step, because the skip_step = 2.

We currently do not have DART cesm2_2 SourceMods that override this functionality because it looked like the original CLM code was designed to be run interactively with DART without specific SourceMods.
What actually happened?

The balance checks are never enabled, because DAnstep = 0, for the entirety of the simulation, thus failure of balance checks is never enabled because the following if statement to throw mass balance error is never true, for example:

if ((errh2o_max_val > error_thresh) .and. (DAnstep > skip_steps))

When DA is turned off (running ensemble free run), the DAnstep is updated, and the checks are enabled.

Error Message

None.

Which model(s) are you working with?

ctsm-release-cesm2.2.03

Version of DART

Which version of DART are you using?

v11.4.0

Have you modified the DART code?

No

Build information

Please describe:

NSF NCAR supercomputer Derecho with intel compiler

XueliHuo commented 5 months ago

Thanks for opening this issue, Brett. I am going to add the BalanceCheckMod.F90 in the SourceMods used for SWE DA.

braczka commented 5 months ago

This same problem (DA_nstep not updated properly) also impacts the operation of the carbon and nitrogen balance checks in ~/src/biogeochem/CNBalanceCheckMod.F90. Although the CNBalanceCheckMod DART SourceMods allow for skipping of first time step through is_first_restart_step , this script is never called from ~/src/biogeochem/CNVegetationFacade.F90 because of DA_nstep problem.

Quick patch is easy enough by removing following lines from CNVegetationFacade.F90:

DA_nstep = get_nstep_since_startup_or_lastDA_restart_or_pause()if (DA_nstep <= skip_steps ) then
       if (masterproc) then
          write(iulog,*) '--WARNING-- skipping CN balance check for first timesteps after startup or data assimilation'
       end if

I will implement this quick patch for immediate solution. For longer term solution will also look into problem with initialization and update of DA_nstep and nstep variables within CTSM. Currently, these bugs are not allowing any balance checks (snow,water,energy,carbon,nitrogen) to take place when DA is turned on. Not a problem with the functioning of DART necessarily, but balance checks are a key diagnostic for model stability, which can be a problem for DA applications.

NCAR / DART