Mismatch in calendar when using a temporal perturbation on the "experiment" calendar.

[mpudig]

I am experiencing a slow-down in my ACCESS-OM2-1 simulations when applying a time-dependent, spatially constant perturbation.

However, the slow-down appears only to arise when I start from a spun-up control simulation. When I start from a cold start, my simulation runs fine (finishing 1 year in ~14 minutes wall time). Whereas when I start from a spun-up control simulation (starting at model year 2100) my simulation runs fine until it experiences a dramatic slow-down at around 2100-11-10 (finishing 1 year in ~1 hr wall time).

The cold-start run can be found at /home/561/mp2135/access-om2/1deg_jra55_ryf_wcwc_test1and the run experiencing the slow-down can be found at /home/561/mp2135/access-om2/1deg_jra55_ryf_wcwc_test2. The only difference between the two is that test2 is branched from a spin-up.

There has been some previous discussion in this thread on Slack https://arccss.slack.com/archives/C08KM5KS6/p1625456199042400.

[rmholmes]

I think we may have figured out the cause of this. Will post again shortly.

[rmholmes]

We have solved the slow-down problem. The time axis in the perturbation file did not cover the simulation time range and so there was some extrapolation going on. Really, it should spit out a warning/error if this is the case?

However, in solving that problem we have encountered another to do with leap years. We are using a "temporal", "offset" forcing perturbation with the "experiment" calendar, with an RYF simulation (so all calendars should be "noleap"). However, the applied perturbations seem to go out of sync with the output calendar by 1 day every 4 years.

I have put together a minimal example here, which applies alternating years of +20Wm-2 and -20Wm-2 shortwave anomalies. In the last cell you can see that if we read the forcing perturbation out of the forcing perturbation file we created, the transition times from +20 to -20 match up properly with the first day of each year. However, what we get out of the simulation is losing a day a year:

Forcing file perturbation transitions
[cftime.DatetimeNoLeap(1900, 1, 1, 3, 0, 0, 0)
 cftime.DatetimeNoLeap(1901, 1, 1, 3, 0, 0, 0)
 ...
 cftime.DatetimeNoLeap(1910, 1, 1, 3, 0, 0, 0)]
Output file perturbation transitions
[[cftime.DatetimeNoLeap(1901, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1902, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1903, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1904, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1905, 1, 1, 0, 0, 0, 0)]]
The output file loses a day in the leap year 1904!!

@nichannah would be really great to get some help here. I originally thought it was a problem with the way we were creating the forcing time variable. But the test above seems to show it's a libaccessom2 problem?

[rmholmes]

Just adding another note here: I thought this could be associated with how I was putting the time variable back in the netcdf file (by calculating by hand the days since 1900-01-01). However, if I instead let xarray do this by keeping the time variables as cftime_DatetimeNoLeap objects (and encoding['units'] = "days since '1900-01-01'"), I still lose a day every leap year. There must be some problem with having an experiment time calendar in the forcing file...

We could really use some help here @nichannah. Also, I wonder if @aidanheerdegen has any ideas?

[rmholmes]

We have figured out a hack workaround here by simply increasing the period of our oscillation by 365.25/365. We are no longer losing leap days, but it really is a hack. There is clearly still some bug in there as the transition times that we're putting in don't match what we get out:

Forcing file perturbation transitions
[cftime.DatetimeNoLeap(1900, 1, 1, 3, 0, 0, 0)
 cftime.DatetimeNoLeap(1901, 1, 1, 9, 0, 0, 0)
 cftime.DatetimeNoLeap(1902, 1, 1, 15, 0, 0, 0)
 cftime.DatetimeNoLeap(1903, 1, 1, 21, 0, 0, 0)
 cftime.DatetimeNoLeap(1904, 1, 2, 3, 0, 0, 0)
 cftime.DatetimeNoLeap(1905, 1, 2, 9, 0, 0, 0)
 cftime.DatetimeNoLeap(1906, 1, 2, 15, 0, 0, 0)
 cftime.DatetimeNoLeap(1907, 1, 2, 21, 0, 0, 0)
 cftime.DatetimeNoLeap(1908, 1, 3, 3, 0, 0, 0)
 cftime.DatetimeNoLeap(1909, 1, 3, 9, 0, 0, 0)
 cftime.DatetimeNoLeap(1910, 1, 3, 15, 0, 0, 0)]
Output file perturbation transitions
[[cftime.DatetimeNoLeap(1901, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1902, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1903, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1904, 1, 3, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1905, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1906, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1907, 1, 2, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1908, 1, 3, 0, 0, 0, 0)]
 [cftime.DatetimeNoLeap(1909, 1, 2, 0, 0, 0, 0)]]

But it works for our purposes...