search

NGEET / fates

repository for the Functionally Assembled Terrestrial Ecosystem Simulator (FATES)
Other
105 stars 92 forks source link

E3SM api 27 restart issue #1106

Open rgknox opened 1 year ago

rgknox commented 1 year ago

E3SM-fates does not pass restart tests with API 27, namely:

ERS_Ld20.f45_f45.IELMFATES.pm-cpu_gnu (perlmutter)

The changes are relegated to the two history diagnostics that use the heterotrophic respiration boundary condition passed in from E3SM.

I'm currently testing some fixes, the first approach is to save the column/site level heterotrophic respiration in a fates structure, instead of setting the diagnostics from the bc_in structure.

rosiealice commented 1 year ago

@mvertens will kick off a runs to spin up the model and look at the results next week (Norwegian computers just changed their configurations and @mvdebolskiy is mostly trying to fix compiling the model on that...)

glemieux commented 1 year ago

@rgknox this appears to be cropping back up in testing the expanded fates test list for https://github.com/E3SM-Project/E3SM/pull/6018 for some reason.

glemieux commented 1 year ago

I determined that this came up via https://github.com/E3SM-Project/E3SM/pull/6018 in stead of https://github.com/E3SM-Project/E3SM/pull/5555 because the former includes an update to make the test output double precision (hist_ndens=1). I confirmed this by testing E3SM-Project/E3SM#5555 (which includes the TOT_HET_RESP fix from #1114) twice, once with the default single precision and once with double precision. Sure enough the single precision passes, but the double fails. The differences I'm seeing are on the order of E-28 (normalized to E-17). This crops up with both gnu and intel compilers, regardless of debug mode.

@rgknox and I discussed this and our short term plan is to reopen this issue and revert back to single precision for https://github.com/E3SM-Project/E3SM/pull/6018 for the time being.