Recover older answers for compatibility with SPEAR

[adcroft]

Since early 2018 @wfcooke has had to revert commits 48e90d0b928457e64c80b065c32f1131cc6b6dfb and bc6c6e65d658f7cdddf0c589ae770feb40287c01 to recover solutions in the SPEAR model. The first commit increased the accuracy with which density anomalies are calculated, and the second was a bug fix changing the value of the model's time in the diabatic phase. In that last few months we've diverged so far from the 2018 code that reverting the commits is no longer practical.

The dev team already agreed that we'd recover answers but @Hallberg-NOAA and I have only just looked at the best way to do that. Proposed fixes are to:

• [x] Add an extra equation of state using the inaccurate evaluation without RHO_REF subtraction (e.g. EQN_OF_STATE="WRIGHT_LEFT");
• [x] Add a flag to optionally use the wrong time at certain places in the algorithm.

[adcroft]

Concerning 48e90d0, the change in how the reference density is used was made in the density integrals. While this used to be in MOM_EOS.F90, it did not involve a change in the equation of state codes. To recover the old answers will thus not require an alternate equation of state but just a new flag to toggle some code.

[adcroft]

@Hallberg-NOAA Should we implement this in all three variants of the PGF (PCM, PLM, PPM reconstructions) or just the PLM variant being used SPEAR? Seems like a sure way to reduce code coverage since we'll only test the one relevant combination.

[adcroft]

I believe that https://github.com/NOAA-GFDL/MOM6/commit/74dcb13706d2977d679d4fe7b45cb97f73d6b20a#diff-66ca61ccaca43a9217e857b349d1ffc3fdfcfa65b04995b6e62b0d66d1e1a16d also must be reverted to recover answers.

[adcroft]

There is also a change in MOM_neutral_diffusion.F90 that caused divergence, namely the second part of 8f4af3d9ef927dc4b99d2a44f32a1e0a3ca5c2c3. @wfcooke do you patch this file too?

[adcroft]

@wfcooke Please try branch options-to-recover-spear with four commits to see if that recovers answers for you.

[adcroft]

I should add (was distracted by some amazing poetry) that to recover answers requires four parameters:

USE_INACCURATE_PGF_RHO_ANOM=True
USE_DIABATIC_TIME_BUG=True
NDIFF_USE_UNMASKED_TRANSPORT_BUG=True
KH_BG_2D_BUG=True

[balaji-gfdl]

And here's the poem:

https://thehill.com/homenews/news/535052-read-transcript-of-amanda-gormans-inaugural-poem

[wfcooke]

Thanks for the extra flags. This is not quite there yet unfortunately. Let me try to figure out where the differences are .

[wfcooke]

@adcroft : The extra flags were helpful, but the most helpful thing was spelling the older flags correctly :) I was setting MEKE_ALPHA_RHINE instead of MEKE_ALPHA_RHINES to a non-zero value. (As an aside having MEKE_ALPHA_EADY=0.05 and MEKE_ALPHA_RHINES =0 hung my runs) Also @MJHarrison-GFDL clued me in to update the SIS2 flags as well.

So this looks like a good candidate for me. I've run several tests (different resolutions, ensembles) which reproduce our old runs.

[adcroft]

Excellent. I'll submit this to dev/gfdl.