Non zeros being written to fuel_moisture_nfsc with Spitfire off

[JessicaNeedham]

I've been running FATES with Spitfire turned off but have noticed that FUEL_MOISTURE_NFSC, which should be 0s, is filled with strange values. These change depending on whether I'm running on Eddi or Lobata but for example on eddi, git hash 39d0635 I got FUEL_MOISTURE_NFSC = -8978.94, 0.8979838, 0.8979201, 30.8887, 255.9254, 0.8979838 ;

in year 11, month 8. In some years and months there are NaNfs, and following this the run will crash a few months later. In this case the NaNf was in year 11 month 5. When the run crashes I get an error message: NetCDF: Numeric conversion no representable.

If I run things in debug the FUEL_MOISTURE_NFSC is all 0s and it does not crash. If I set the variable to inactive in FatesHistoryInterfaceMod then FATES no longer crashes.

[glemieux]

Jessie and I have been talking about this for a couple days. It looks like this is probably just a problem with the values not being zero'd out in the zero_patch subroutine. FUEL_MOISTURE_NFSC (associated with hio_litter_moisture_si_fuel) is being calculated by the patch litter moisture: https://github.com/NGEET/fates/blob/39d063560153ddda887640d5bc0dc44ccd188385/main/FatesHistoryInterfaceMod.F90#L2255-L2257 A number of fire variables that are included in the history output are initialized as zero regardless of spitfire being on or off in EDPatchDynamicsMod.F90, but litter_moisture isn't included: https://github.com/NGEET/fates/blob/39d063560153ddda887640d5bc0dc44ccd188385/biogeochem/EDPatchDynamicsMod.F90#L1606-L1618

[jkshuman]

@JessicaNeedham and @glemieux I am surprised I haven't seen this error in my runs (with or without fire) using tag 1.27.2.api7.3 on Hobart and Cheyenne. I agree with @glemieux about adding it to the zero_patch subroutine. Keep me posted on if the problem continues.

[glemieux]

Me too @jkshuman. I'm wondering if its a compiler specific idiosyncrasy? Jessie's been running and eddi and lobata which default the the gnu fortran compiler.

[glemieux]

@jkshuman can you point me to an example script and run that is representative of the tests you mentioned above on cheyenne so I can compare to Jessie's script and run?

I'm now wondering if this was overlooked due to the presence of hist_empty_htapes in the setup. The standard fates regression tests set this to true turning off the default ctsm output. The fates variables are selectively added back in a subset of fates variables. Chatting briefly with @ckoven yesterday, we may need to review and expand the set of variables for the tests. That said, I'm curious if others are including this call in their scripts as well.

[jkshuman]

@glemieux are you looking for my case details? These were not specific regression tests, but optimistic science runs. I mostly run on Hobart here at NCAR - do you have access to Hobart? Will check to see how old my Cheyenne runs are...

[glemieux]

@jkshuman I don't have access to Hobart I don't think. Yes, I'm looking for the science runs so that I can get a sense of why you might not have seen the error. Specifically, do you have cases that output the fuel_moisture_nfsc variable?

[jkshuman]

@glemieux are you able to view this directory: /project/tss/jkshuman/hobart_izumi/archive/ there are 4 recent cases in there. 1deg_MoiT_C3_rm_effect_wspeed_burn-litter_1x1_08fc9481_abccb775 fire_1deg_MoiT_C3_tag1272_burn_litter_fix_1x1_08fc9481_ee583454 Fire_MoiT_C3_rm_effect_wspeed_burn-litter_1x1_08fc9481_abccb775 Yr10_Fire_Hybrid_1deg_v1_MoiT_C3_rm_effect_windspeed_burn-litter_1x1_08fc9481_abccb775 I am having trouble with ncview - but that output should be there.

[glemieux]

@jkshuman should I be able to see this directory when logged into cheyenne? If so, I can't see it unfortunately. Could I access it through a Globus end point?

[jkshuman]

@glemieux it is a CGD machine. Let me check with Mark Moore about getting you and Ryan access to Hobart so that you can view my cases there when necessary.

[jkshuman]

@JessicaNeedham and @glemieux thanks for finding and fixing this.

[glemieux]

Closing with the merge of #552. Started #551 to address why this wasn't caught earlier.