Add checks for negative precipitation coming from atmosphere

[ekluzek]

Brief summary of bug

Tony Craig reports the following problem where it seems CTSM will blow up if you give it precipitation that is roundoff level negative.

We've done some more testing and it looks like the h2osoi_liq was blowing up because WRF was sending it tiny negative values in the snowl coupling field. We are looking into what WRF is doing, but these values were typically 10-20 orders of magnitude smaller than the maximum precip values. In other words, it looks like a rounding error in repartitioning of some precip fields in WRF or something like it. We have basically corrected this in the lnd_import_export subroutine by checking for tiny negative values in precip and setting them to zero. There is almost no loss of conservation.

I think a bigger question for us is whether there is a known sensitivity in the CTSM implementation to negative precip (even if tiny). It could be that the implementation assumes it will always be greater than 0 and if it isn't, it leads to an unreasonable solution. That's what appears to be happening, although we will have to look further to confirm.

General bug information

CTSM version you are using: ?

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: Coupled to WRF with snowl negative

Details of bug

wr50a arctic regional grid with WRF and CTSM

Important output or errors that show the problem

Here's a sample error that Tony got (from either a cold start or with spunup IC)...

ERROR: In UpdateState_TopLayerFluxes, h2osoi_liq has gone significantly negative Bulk/tracer name = bulk c, lev_top(c) = 39570 0 h2osoi_liq_top_orig = 0.000000000000000E+000 h2osoi_liq = -1.972896852494521E-027 frac_sno_eff = 8.687570326868314E-002 qflx_liq_grnddtime = -2.270942022066725E-026 qflx_liqdew_to_top_layerdtime = 0.000000000000000E+000 qflx_liqevap_from_top_layer*dtime = 0.000000000000000E+000 ENDRUN: (shr_sys_abort) ERROR: In UpdateState_TopLayerFluxes, h2osoi_liq has gone significantly negative (shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping

[ekluzek]

@dlawrenncar @swensosc @olyson do you know of a problem where CTSM can't handle precipitation that is negative by roundoff? Anyone else we should ask?

[dlawrenncar]

Unless someone can provide a good rationale for why CTSM should work with negative precipitation, it seems to me that it makes more sense to prevent it at the source rather than to spend time figuring out how to accommodate it within the land model. Negative precipitation, to state the obvious, has no physical meaning

On Wed, Oct 7, 2020 at 11:12 AM Erik Kluzek notifications@github.com wrote:

@dlawrenncar https://github.com/dlawrenncar @swensosc https://github.com/swensosc @olyson https://github.com/olyson do you know of a problem where CTSM can't handle precipitation that is negative by roundoff? Anyone else we should ask?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CTSM/issues/1178#issuecomment-705075667, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFABYVE3H5E2JE7TW4O3B4DSJSOQVANCNFSM4SHVLB3A .

[swensosc]

there are already checks that solar, specific-humidity and LW downward are

= in lnd_import_export

On Wed, Oct 7, 2020 at 11:55 AM David Lawrence notifications@github.com wrote:

Unless someone can provide a good rationale for why CTSM should work with negative precipitation, it seems to me that it makes more sense to prevent it at the source rather than to spend time figuring out how to accommodate it within the land model. Negative precipitation, to state the obvious, has no physical meaning

On Wed, Oct 7, 2020 at 11:12 AM Erik Kluzek notifications@github.com wrote:

@dlawrenncar https://github.com/dlawrenncar @swensosc https://github.com/swensosc @olyson https://github.com/olyson do you know of a problem where CTSM can't handle precipitation that is negative by roundoff? Anyone else we should ask?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CTSM/issues/1178#issuecomment-705075667, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AFABYVE3H5E2JE7TW4O3B4DSJSOQVANCNFSM4SHVLB3A

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CTSM/issues/1178#issuecomment-705098397, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGRN57FGTP744Q2FJLSBBELSJSTQZANCNFSM4SHVLB3A .

[ekluzek]

Yes, it wouldn't be hard to check for negative rain and snow, so we'll plan on doing that. Some quantities the model allows to go slightly negative even if it isn't physical -- but some (like this) it doesn't. It would be good to have checks in place for as many things like this that we know of, as it would save others time.

Yes, I can't think of a reason we should handle negative precipitation. Part of my question was just had this behavior been seen before? But, yes, just aborting if we ever get negative precipitation would solve it for the future.

[negin513]

@ekluzek If they see negative values from WRF output for moisture variables they can easily fix it using WRF namelist options by setting moist_adv_opt=1. In fact, I believe this is the default namelist option in WRF.

Zeroing out negative values precipitation values from the WRF side might not be good practice, because it removed the conservation of mass and introduce errors.

[apcraig]

I'll just add a few additional things. I'm not a WRF expert. After some further investigation, It looks like WRF may be introducing negative precip as a result of the advection not being positive definite. I assume @negin513's comment about the moist_adv_opt setting addresses that issue. We'll look into that. I will say that our current namelist has moist_adv_opt = 1.

As a short term solution, we added a check in the CTSM cap where if the precip was less than zero but greater than -1.0e-16 (12 orders of magnitude smaller than typical maximum precip values), we zero it out. Otherwise, we abort. This does introduce a conservation error that is very, very small. But we're also running on regional grids so we have a lot bigger leaks to worry about. We'll look into the moist_adv_opt further. Thanks @negin513.

Mostly, I wanted to raise this issue in case the same thing happened in LILAC just as a heads up. But also to raise an awareness of the sensitivity of CTSM to the issue. I'm not going to argue that CTSM should work fine with tiny negative precip, but these kinds of issues arise all the time in coupled systems where something "slightly" non-physical happens. If you want to protect CTSM from negative precip, that sounds like a good solution. I am curious why CTSM is so sensitive to the negative precip, but I don't need to understand it.

[anewman89]

Thanks everyone. For more context, initially I had told Tony incorrectly we may have a version of WRF without positive definite advection, but I may have been getting mixed up with the monotonic advection added in V3.1. So we are using PD advection as Tony notes, but we are also using diff_opt=1 and km_opt=4, which may not be positive definite. I've also discussed with another WRF user who has seen similarly small negative precipitation values (~-1e-25 to -1e-30). So it may occur occasionally in real data cases with very light precipitation.

[ekluzek]

So is what we should do is implement Tony's solution -- that we truncate numbers slightly negative by roundoff to zero and then die for anything more negative than that? Or should we just die anytime a negative precip value is given?

[anewman89]

I'm not 100% sure yet. I think we need to dig a little more with real data cases in WRF and possibly ask Bill Skamarock and or Joe Klemp. I'm happy to do that for us.

[ekluzek]

@anewman89 OK we'll wait to hear back from you on how this should be handled then. We'll start planning on making this check happen though.

[anewman89]

I heard back from Bill and the variables CTSM is using from WRF are not impacted by the WRF advection routines, they are direct output from the microphysics scheme. In this case we are using the Morrison 2-moment scheme. Its possible this behavior will not occur with other microphysics schemes, or with other configurations using Morrison as the Morrison scheme was not necessarily designed for simulations with long atmospheric model time steps.

That being said Bill agreed the magnitude of our negative precipitation values are roundoff issues. Furthermore, @YifanCheng is able to perform year long simulations with @apcraig's fix which we're examining now. We could also examine other microphysics schemes for this behavior...

Right now I think moving towards checking for negative precipitation and setting small values to zero with a message that there are negative precipitation values will be ok.

[billsacks]

We'll add checks for negative precipitation in the coupling fields. (The atmosphere model should clean up their fields before sending them to the land model.)

[anewman89]

Thanks @billsacks. I've been discussing this with WRF help and have found the following with the help of @YifanCheng. Negative precipitation occurs with the Morrison 2-moment scheme as noted above and the WRF help team was not aware of this occurring. We've identified the problem code and have an inquiry out with the Morrison developer to see about a fix. It is possible other microphysics schemes in WRF allow for negative precipitation too, so I think this check is prudent. We do know there is no negative precipitation with the Thompson microphysics scheme and our coupled simulation looks reasonable. I can push more with the WRF folks to check all microphysics schemes.