[H2] Run ACCESS-OM2-01 RYF experiment for the first 10 yrs

[willaguiar]

The earliest saved restart for the 01deg_jra55v13_ryf9091 is at ~ year 100, so we cannot simply rerun the 10th year of that simulation to get the diagnostics we want for DSW formation. We will need this output for future comparison with 01deg_jra55v13_ryf9091_DSW5m anyways. So I am currently rerunning this RYF for 10 years, under the name 01deg_jra55v13_ryf9091_DSW with the required diagnostic outputs.

Early analysis of DSW formation in 01deg_jra55v13_ryf9091_DSW will be shown here.

Edit April 18th 2023:

• Bathymetry was edited
• Fixed non-advective cells and partial cells(April 17th)
• Run restarted after changes in the bathymetry. Currently running year 1, April

[adele-morrison]

If we're restarting from scratch anyway, should we use the new bathymetry for the control rerun also?

On Fri, 31 Mar 2023 at 14:55, Wilton Aguiar @.***> wrote:

The earliest saved restart for the 01deg_jra55v13_ryf9091 is at ~ year 100, so we cannot simply rerun the 10th year of that simulation to get the diagnostics we want for DSW formation. We will need this output for future comparison with 01deg_jra55v13_ryf9091_5m anyways. So I am currently rerunning this RYF with the required diagnostics.

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/13, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACA44U76WLCBS7QRZWZMQF3W6ZIS3ANCNFSM6AAAAAAWODXJOY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[dkhutch]

I think it depends on how long it takes to achieve spin-up of velocities. I would assume starting from scratch means setting ocean velocities to zero? Does this mean it's a 10-year minimum simulation, as opposed to a 1-year re-run?

On Fri, Mar 31, 2023 at 2:59 PM Adele Morrison @.***> wrote:

If we're restarting from scratch anyway, should we use the new bathymetry for the control rerun also?

On Fri, 31 Mar 2023 at 14:55, Wilton Aguiar @.***> wrote:

The earliest saved restart for the 01deg_jra55v13_ryf9091 is at ~ year 100, so we cannot simply rerun the 10th year of that simulation to get the diagnostics we want for DSW formation. We will need this output for future comparison with 01deg_jra55v13_ryf9091_5m anyways. So I am currently rerunning this RYF with the required diagnostics.

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/13, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACA44U76WLCBS7QRZWZMQF3W6ZIS3ANCNFSM6AAAAAAWODXJOY

. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/13#issuecomment-1491252843, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVNSK6IMLBFYPD3MGC4VLLW6ZJDRANCNFSM6AAAAAAWODXJOY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[dkhutch]

Perhaps 10 years is still ok... just thought it might be a factor.

On Fri, Mar 31, 2023 at 3:02 PM David Hutchinson < @.***> wrote:

I think it depends on how long it takes to achieve spin-up of velocities. I would assume starting from scratch means setting ocean velocities to zero? Does this mean it's a 10-year minimum simulation, as opposed to a 1-year re-run?

On Fri, Mar 31, 2023 at 2:59 PM Adele Morrison @.***> wrote:

If we're restarting from scratch anyway, should we use the new bathymetry for the control rerun also?

On Fri, 31 Mar 2023 at 14:55, Wilton Aguiar @.***> wrote:

The earliest saved restart for the 01deg_jra55v13_ryf9091 is at ~ year 100, so we cannot simply rerun the 10th year of that simulation to get the diagnostics we want for DSW formation. We will need this output for future comparison with 01deg_jra55v13_ryf9091_5m anyways. So I am currently rerunning this RYF with the required diagnostics.

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/13, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACA44U76WLCBS7QRZWZMQF3W6ZIS3ANCNFSM6AAAAAAWODXJOY

. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/13#issuecomment-1491252843, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVNSK6IMLBFYPD3MGC4VLLW6ZJDRANCNFSM6AAAAAAWODXJOY . You are receiving this because you are subscribed to this thread.Message ID: @.*** com>

[willaguiar]

Is that so we would only have a difference in the vgrid, but none in the bathymetry? If so make sense - I agree.

If we're restarting from scratch anyway, should we use the new bathymetry for the control rerun also? …

Is that so we would only have a difference in the vgrid, but none in the bathymetry? If so make sense - I agree.

[adele-morrison]

Yes.

Just to note that starting from scratch is also going to be more expensive, because we need a smaller timestep to start.

[willaguiar]

The simulation has ran for 9 months now (tstep 300). But when running the last 3-months from restart0002, the simulation gets a long time stuck in initialization (~1,5hour) with no error. After 1,5hr the simulation starts running, but then the maximum 5h walltime for the run is not enough to complete the 3 months. Does someone has a suggestion on how to solve this problem?

[adele-morrison]

Did this happen just once, or did you try again and get the same problem? gadi often has glitches that disappear when you try to rerun. If it's reproducible, I'm not sure what it could be.

On Sat, 8 Apr 2023 at 16:31, Wilton Aguiar @.***> wrote:

The simulation has ran for 9 months now (tstep 300). But when running the last 3-months from restart0002, the simulation gets a long time stuck in initialization (~1,5hour) with no error. After 1,5hr the simulation starts running, but then the maximum 5h walltime for the run is not enough to complete the 3 months. Does someone has a suggestion on how to solve this problem?

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/13#issuecomment-1500805573, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACA44U5DRDDD3IGYU3DJ4D3XAEA4LANCNFSM6AAAAAAWODXJOY . You are receiving this because you commented.Message ID: @.***>

[willaguiar]

I tried twice yesterday, and once a few hours ago changing the tstep to 600 to see if fixed the problem . I noticed Gadi has been really slow since Thursday, and I haven't been able to access it since I submitted the run earlier today so perhaps is some temporary system overload.

[adele-morrison]

You could try switching to 1 month run chunks? Though if this problem continues it's going to be an expensive run! Maybe worth waiting to see if anyone else has solutions.

[willaguiar]

The run with Tstep=600 also runs out of time. I tried finding any clue on the logs on what it could be, but can't find anything weird other the killed process error after the wall time runout (logs are very similar to the ones in .../01deg_jra55v13_ryf9091/output800/ .

[adele-morrison]

I think I've also been seeing some similar weird slow behaviour in my panan runs. Maybe it's a gadi issue. I would suggest emailing @.*** and saying your runs are suddenly slow and giving them the job numbers of the runs that ran out of time. They might be able to backtrack and figure out what the problem is.

On Sun, 9 Apr 2023 at 21:17, Wilton Aguiar @.***> wrote:

The run with Tstep=600 also runs out of time. I tried finding any clue on the logs on what it could be, but can't find anything weird other the killed process error after the wall time runout (logs are very similar to the ones in .../01deg_jra55v13_ryf9091/output800/ . I am uploading the errors here https://github.com/willaguiar/DSW-collaborative-project/tree/main/RYF_control for the case someone can find something I'm missing out.

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/13#issuecomment-1501104932, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACA44U6NFGVNLIAHYOBEVFDXAKLDJANCNFSM6AAAAAAWODXJOY . You are receiving this because you commented.Message ID: @.***>

[willaguiar]

Reporting: It seemed like it was a problem with Gadi, as today I tried again, and the simulation initialized within 8 minutes instead of 1.5hr. With 600 of timestep, the 3-months run finished within 2h and 20 min of wall time.

[willaguiar]

Update on water mass transformation on the control run ( 01deg_jra55v13_ryf9091_DSW, dzmin=1m, KDZ75 grid).

Below is the Surface water mass transformation on the shelf for the first year of the run. It seems like in the first year there is already DSW formation in the control run in the Weddell sea, Ross sea and Adelie land.

The run now is completing its third year. I wonder If we need to run for the whole 10 years period?

[PaulSpence]

Maybe need some time-series metric to decide if the DSW is in steady state to decide to stop the run? Time-series of area integrated SWMT in different regions?

[willaguiar]

Below is the SWMT for the only 3 years we have ran for the control case. It seems to have small changes tho (small increase in the density where maximum transformation occurs, but a decrease of 0.3 SV in the overall maximum rate of transformation). Unless Im missing something. After figuring out what is causing the salinity errors, perhaps we will need to run for more years to see if this is a trend.

[adele-morrison]

Is there enough output from the original RYF to check more years there? It would be useful to do two time series: 1 of the density at which the peak occurs, and

2. The transformation rate at a fixed density (e.g. 32.5 or 32.6).

On Fri, Apr 14, 2023 at 5:43 PM, Wilton Aguiar @.***> wrote:

Below is the SWMT for the only 3 years we have ran for the control case. It seems to have small changes tho (small increase in the density where maximum transformation occurs, but decrease in the overall maximum rate of transformation of 0.3 SV). Unless Im missing something. After figuring out what is causing the salinity errors, perhaps we will need to run for more years to see if this is a trend.

[image: 3years_shelf-2] https://user-images.githubusercontent.com/70033934/231977797-d4302ae1-89e9-4aa4-a0c8-73f00f82e8db.png

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/13#issuecomment-1508071662, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACA44U6IVA6N3GVESOSRSQLXBD5YVANCNFSM6AAAAAAWODXJOY . You are receiving this because you commented.Message ID: @.***>

[willaguiar]

I checked the SWMT for the first 20 years of the experiment 01deg_jra55v13_ryf9091.

Along the shelf the maximum transformation seem to occur mostly in the same density level (except on year 2)

Same can be seen in the time series of density of maximum transformation:

DSW formation rate at 32.5 kg m^{-3}, after 10 years it seems to have a small increasing trend:

DSW formation rate at 32.6 kg m^{-3}, it seems to need ~5 years for adjustment?:

Let me know what you think...