Investigate reproducibility issue with arbitrary restarts

[penguian]

| by pbd562@nci.org.au

---

---

Issue migrated from trac:311 at 2024-01-31 18:28:24 +1100

[penguian]

pbd562@nci.org.au changed status from assigned to accepted

[penguian]

pbd562@nci.org.au set owner to pbd562

[penguian]

pbd562@nci.org.au commented

---

1. ACCESS-CM2 1-month and 3-month resubmissions give different results
2. ACCESS-OM similarly does not reproduce
3. ACCESS GA7 (JULES) AMIP is ok
4. ACCESS GA7 (CABLE) AMIP to be tested

[penguian]

@martin.dix@anu.edu.au commented

---

Set up a coupled model suite that compares 1+1 and 2 day runs.

[penguian]

@martin.dix@anu.edu.au _uploaded file ocn_sst_diff.png (19.9 KiB)_

[penguian]

@martin.dix@anu.edu.au commented

---

The GA6 and 7 AMIP suites include tests for reproducibility on different decompositions, with and w/o OpenMP and also that a sequence of shorter runs matches a single longer run.

The normal test with the 360 day calendar is that 3 10 day runs matches a standard 30 day run.

I've now also checked this with the Gregorian calendar with a variant of the vn10.6 GA7.1 suite u-aj458. A sequence of 2 one month runs matches a single two month run.

Note that with UM reproducibility requires that each run write dump files at the same frequency.

The suite u-an301 compares two coupled model runs in the same way by comparing the final UM restart files. 2x1 month is different to a 2 month run.

Comparing the UM solver diagnostics from the start of the second month shows that the first difference appears at 1-02-01 03:20:00, i.e. just after the coupling step.

Ocean model from 2 month run

 From ocean_tracer_mod: tracers after explicit tendency update (taup1)

 ### Prognostic tracer checksum follows
yyyy/mm/dd hh:mm:ss =    1/ 2/ 1  0:30: 0
[chksum] temp                                      7854997267852092459

From start of second month of 2x1 month runs

 From ocean_tracer_mod: tracers after explicit tendency update (taup1)

 ### Prognostic tracer checksum follows
yyyy/mm/dd hh:mm:ss =    1/ 2/ 1  0:30: 0
[chksum] temp                                     -1392703324634173096

Comparing OCN_SST from the coupling diagnostic files at 0300 shows a coherent large scale pattern which suggests some difference in the physics rather than just random noise.

I couldn't find any per step ice diagnostics to compare. The ocean difference at 0030 could be due to ice-ocean coupling but the pattern of SST differences makes me suspect MOM is to blame.

[penguian]

@martin.dix@anu.edu.au changed _comment0 which not transferred by tractive

[penguian]

@martin.dix@anu.edu.au changed _comment1 which not transferred by tractive

[penguian]

@martin.dix@anu.edu.au commented

---

If the UM is set to save daily dump files then in AMIP mode 2x1day compares to a 2 day run. However this doesn't compare with the coupled model. The symptoms are the same as for the monthly run with the differences in the MOM checksum and a similar pattern of SST differences from the coupler diagnostics.

[penguian]

@martin.dix@anu.edu.au commented

---

u-an301 suite details. From rose-suite.conf

RESUB='P2D'
RESUB_CRUN='P1D'
RUNLEN='P2D'
TEST_CRUN=true

This does a normal run of length 2 days and another run of 2x1 day. Task names are coupled and coupled_crun. UM output goes to History_Data and History_Data_CRUN. Data is archived to share/archive/an301 and share/data/an301_CRUN rather than to the normal archive subdirectory.

[penguian]

@martin.dix@anu.edu.au uploaded file sst.png (9.8 KiB)

[penguian]

@martin.dix@anu.edu.au commented

---

Change diag_table to have

"HISTORY/ocean_daily",                30,  "minutes",  1, "days", "time",

gives SST per time step. Plotting this shows a clear jump in the SST at the start of the second day in the 2x1 run (red line).

This looks like the difference one might see using a forward time step at the start of a run rather than the normal centered time step (c.f. the flat section at the very start of the run).

[penguian]

@martin.dix@anu.edu.au changed _comment0 which not transferred by tractive