3 month PVT FatesLUPFT test is not passing baseline comparisons on PR 1198

[rgknox]

With PR #1198 all but the new PVT test passes B4B with base.

The diffs with base are not relegated to a small set of variables, nor are they close to machine precision.

PVT_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesLUPFT.GC.0731-094347de_int

I also tried a debug mode simulation with gnu to see if I could generate any initialization failures, but nothing popped up:

PVT_D_Lm1.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_gnu.clm-FatesLUPFT

[glemieux]

I'm not quite sure what might be causing the diff here. At first blush, the restart from the spin-up portion is b4b (i.e. t_index = 1 has no RMS) and for good measure I ran a cprnc directly against the actual spinup (e.g. potveg files) in the baselines and your test and that was b4b. That said, it we don't have all possible output being reported (just whats in Fates testmod). It could be that something is actually not b4b in the spin-up. I'm running a test with a modified testmod user_nl_clm to report everything in AllVars to check this hypothesis.

[glemieux]

In trying to open up the number of history outputs per my comment above, I ran into a different issue which I've recorded in #1232.

[glemieux]

I realized that the FATES_TRANSITIONS_MATRIX_LULU isn't being reported for this test due to https://github.com/ESCOMP/CTSM/issues/2657. I've got two PVT tests running against new baselines that have this in the output: one using the FatesLUPFT testmod and one with a standard Fates testmod. My suspicion is that the transitions matrix shouldn't be at issue here given the fix with tag sci.1.77.2_api.36.0.0, but I've included it in an effort to be thorough.

To clarify, for the Fates standard testmod, I had to modify the PVT system test to avoid the land use relevant stuff. This way the test only exercises the rest of the hybrid restart infrastructure. I generated a variant baseline to test against using this modification.

[glemieux]

The comparison using Fates testmod with the variant, non-landuse PVT system test for #1198 against it's custom baseline is still turning up DIFFs. The only notable difference is that while the DIFFs still only show up on the second time step, the disturbance matrix land use history output doesn't show a diff until time step 3. This seems to suggest that the issue is not explicitly to do with land use mode.

DIFF results: /glade/u/home/glemieux/scratch/ctsm-tests/tests_0813-101139de

[glemieux]

For next tests I'm thinking of trying the following:

• [ ] manually replicating the restart for the variant PVT test by passing the restart file with finidat
• [ ] generating a new fates-sci.1.77.2_api.36.0.0 baseline using the latest ctsm tag to see if that triggers a diff