ufs-community / ufs-s2s-model

UFS sub-seasonal to seasonal forecast model. This repository was frozen in Oct 2020 and all development was moved to the ufs-weather-model repository.
GNU General Public License v3.0
8 stars 29 forks source link

ocean import fields are zero when initialized from coldstart run #86

Closed DeniseWorthen closed 4 years ago

DeniseWorthen commented 4 years ago

The current implementation of S2S uses a 1 hour cold start followed by a restart from the cold start. For ease of description, these will be referred to as 'cold1' and 'cold2' below.

Using the NEMS mediator, the cold2 run is initializing the import state for the ocean with all zero fields. This is can be verified by running ufs-s2s-model with field dumping active. For the cold2 run, all fields in the fields_ocnimport(DATE)T00:00:00.nc are zero.

This behaviour is the result of the prep_ocn phase zero-ing the accumulation fields and accumulation counters at the end of the phase. The mediator restart files produced by the cold1 run (mediator_FBaccumAtm_restart.*, mediator_FBaccumAtmOcn_restart.nc, mediator_FBaccumIce_restart.nc) and the accumulation counts (mediator_scalars_restart.txt) all contain zeros as a result.

When the cold2 run starts, the prep_ocn at the top of the run sequence averages these accumulation fields (ie, divides by the accumulation count) to create the import fields for the ocean. Since the fields are zero as well as the counts, the import fields at time 00:00:00 are also zero.

This behaviour is replicated in the current CMEPS implmentation. However, in CMEPS, the presence of a separate accum_avg phase (where the accumulations are zero'd) and restart_write phase allows the run sequence to be adjusted to produce valid (non-zero) mediator restarts for cold1. This involves placing the med_phases_restart_write before of the med_phases_prep_ocn_accum_avg in the cmeps cold1 run sequence.

For the NEMS mediator, this same change cannot be accomplished without separating out the accumulation-related calls from prep_ocn.

A secondary issue w/rt the current cold1/cold2 sequence is that cold1 is at the end of 2 ocean coupling steps. Even if the mediator restarts contain non-zero values (ie, as can be accomplished with CMEPS), this would seemingly result in import fields for the ocean at time 00:00:00 that are in some sense forward by a coupling timestep relative to the ATM and ICE.

An alternative to the cold1/cold2 sequence may be to implement an ocean lag startup. In this case, only the cold2 sequence is implemented but a special setting is used so that the ocean model does not advance on the first coupling timestep. Instead, the ocean model clock is set back to the start time. On the second coupling timestep, the ocean model advances for 2 coupling timesteps. For example, if the coupling timestep is 30 minutes and the internal ocean timesteps are 30 mins (thermo) and 15 mins (dynamic), the model will step forward 2 thermo dynamic timesteps and 4 dynamic timesteps. All subsequent coupling timesteps for the ocean model proceed as normal.

SMoorthi-emc commented 4 years ago

I don't think the cold2 step described above is true in my case as I changed the run sequence to call write_restart before prep_ocean.

On Wed, May 6, 2020 at 8:43 AM Denise Worthen notifications@github.com wrote:

The current implementation of S2S uses a 1 hour cold start followed by a restart from the cold start. For ease of description, these will be referred to as 'cold1' and 'cold2' below.

Using the NEMS mediator, the cold2 run is initializing the import state for the ocean with all zero fields. This is can be verified by running ufs-s2s-model with field dumping active. For the cold2 run, all fields in the fields_ocnimport(DATE)T00:00:00.nc are zero.

This behaviour is the result of the prep_ocn phase zero-ing the accumulation fields and accumulation counters at the end of the phase. The mediator restart files produced by the cold1 run (mediator_FBaccumAtm_restart.*, mediator_FBaccumAtmOcn_restart.nc, mediator_FBaccumIce_restart.nc) and the accumulation counts (mediator_scalars_restart.txt) all contain zeros as a result.

When the cold2 run starts, the prep_ocn at the top of the run sequence averages these accumulation fields (ie, divides by the accumulation count) to create the import fields for the ocean. Since the fields are zero as well as the counts, the import fields at time 00:00:00 are also zero.

This behaviour is replicated in the current CMEPS implmentation. However, in CMEPS, the presence of a separate accum_avg phase (where the accumulations are zero'd) and restart_write phase allows the run sequence to be adjusted to produce valid (non-zero) mediator restarts for cold1. This involves placing the med_phases_restart_write before of the med_phases_prep_ocn_accum_avg in the cmeps cold1 run sequence.

For the NEMS mediator, this same change cannot be accomplished without separating out the accumulation-related calls from prep_ocn.

A secondary issue w/rt the current cold1/cold2 sequence is that cold1 is at the end of 2 ocean coupling steps. Even if the mediator restarts contain non-zero values (ie, as can be accomplished with CMEPS), this would seemingly result in import fields for the ocean at time 00:00:00 that are in some sense forward by a coupling timestep relative to the ATM and ICE.

An alternative to the cold1/cold2 sequence is to use implement an ocean lag startup. In this case, only the cold2 sequence is implemented but a special setting is used so that the ocean model does not advance on the first coupling timestep. Instead, the ocean model clock is set back to the start time. On the second coupling timestep, the ocean model advances for 2 coupling timesteps. For example, if the coupling timestep is 30 minutes and the internal ocean timesteps are 30 mins (thermo) and 15 mins (dynamic), the model will step forward 2 thermo dynamic timesteps and 4 dynamic timesteps. All subsequent coupling timesteps for the ocean model proceed as normal.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-s2s-model/issues/86, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALLVRYWEDUQM26PTNND7BW3RQFLN7ANCNFSM4M2NDYYQ .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718

e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

SMoorthi-emc commented 4 years ago

On the other hand, it is true for cold1; I guess it is defeating the reason for cold1, which, as I understand, is to provide non zero atmospheric forcing to the ocean through the mediator restart files.

On Wed, May 6, 2020 at 10:06 AM Shrinivas Moorthi - NOAA Federal < shrinivas.moorthi@noaa.gov> wrote:

I don't think the cold2 step described above is true in my case as I changed the run sequence to call write_restart before prep_ocean.

On Wed, May 6, 2020 at 8:43 AM Denise Worthen notifications@github.com wrote:

The current implementation of S2S uses a 1 hour cold start followed by a restart from the cold start. For ease of description, these will be referred to as 'cold1' and 'cold2' below.

Using the NEMS mediator, the cold2 run is initializing the import state for the ocean with all zero fields. This is can be verified by running ufs-s2s-model with field dumping active. For the cold2 run, all fields in the fields_ocnimport(DATE)T00:00:00.nc are zero.

This behaviour is the result of the prep_ocn phase zero-ing the accumulation fields and accumulation counters at the end of the phase. The mediator restart files produced by the cold1 run (mediator_FBaccumAtm_restart.*, mediator_FBaccumAtmOcn_restart.nc, mediator_FBaccumIce_restart.nc) and the accumulation counts (mediator_scalars_restart.txt) all contain zeros as a result.

When the cold2 run starts, the prep_ocn at the top of the run sequence averages these accumulation fields (ie, divides by the accumulation count) to create the import fields for the ocean. Since the fields are zero as well as the counts, the import fields at time 00:00:00 are also zero.

This behaviour is replicated in the current CMEPS implmentation. However, in CMEPS, the presence of a separate accum_avg phase (where the accumulations are zero'd) and restart_write phase allows the run sequence to be adjusted to produce valid (non-zero) mediator restarts for cold1. This involves placing the med_phases_restart_write before of the med_phases_prep_ocn_accum_avg in the cmeps cold1 run sequence.

For the NEMS mediator, this same change cannot be accomplished without separating out the accumulation-related calls from prep_ocn.

A secondary issue w/rt the current cold1/cold2 sequence is that cold1 is at the end of 2 ocean coupling steps. Even if the mediator restarts contain non-zero values (ie, as can be accomplished with CMEPS), this would seemingly result in import fields for the ocean at time 00:00:00 that are in some sense forward by a coupling timestep relative to the ATM and ICE.

An alternative to the cold1/cold2 sequence is to use implement an ocean lag startup. In this case, only the cold2 sequence is implemented but a special setting is used so that the ocean model does not advance on the first coupling timestep. Instead, the ocean model clock is set back to the start time. On the second coupling timestep, the ocean model advances for 2 coupling timesteps. For example, if the coupling timestep is 30 minutes and the internal ocean timesteps are 30 mins (thermo) and 15 mins (dynamic), the model will step forward 2 thermo dynamic timesteps and 4 dynamic timesteps. All subsequent coupling timesteps for the ocean model proceed as normal.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-s2s-model/issues/86, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALLVRYWEDUQM26PTNND7BW3RQFLN7ANCNFSM4M2NDYYQ .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718

e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718

e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

DeniseWorthen commented 4 years ago

Update this issue to include next steps:

1) If the CMEPS mediator fields are zero, we need to fix the order of writing restart files. If CMEPS mediator has valid values (non-zero) in restart files, we need to make sure the average fields are passed to ocean correctly in the first time step.

2) If CMEPS mediator has valid values (non-zero) in restart files, we need to make sure the average fields are passed to ocean correctly in the first time step.

DeniseWorthen commented 4 years ago

I added a state_diagnose function to the mom6 nuopc cap and turned on the FB_diagnose in mep_phases_prep_ocn_avg. Using these two features, I have verified that:

1) Using the existing cold start procedure produces ocean import fields of 0.0 for the first ocean coupling timestep when restarting from coldstart.

2) Moving the restart write to earlier in the cold run sequence (as suggested by @mvertens) produces non-zero import fields for the ocean at the first coupling timestep.

3) Using the FV3 fix provided by @junwang-noaa to provide the bottom-most atm values to the ice at the first coupling timestep and using an ocean-lag for the first ocean coupling timestep allows the model to run in a concurrent run sequence from ICs with no cold start mediator file required.

Three test cases corresponding to these items were made. At hour 6, the difference in SST from the initial MOM6 SST for the three cases is shown:

sst_diffs

junwang-noaa commented 4 years ago

Thanks for testing and making these plots, Denise. Can you make a plot of fig3(ocean lag)-sst in current 2day run with mediator fields from cold start? Also will the ocean always lag one slow step when we remove the cold start?

On Mon, Aug 3, 2020 at 8:15 AM Denise Worthen notifications@github.com wrote:

I added a state_diagnose function to the mom6 nuopc cap and turned on the FB_diagnose in mep_phases_prep_ocn_avg. Using these two features, I have verified that:

1.

Using the existing cold start procedure produces ocean import fields of 0.0 for the first ocean coupling timestep when restarting from coldstart. 2.

Moving the restart write to earlier in the cold run sequence (as suggested by @mvertens https://github.com/mvertens) produces non-zero import fields for the ocean at the first coupling timestep. 3.

Using the FV3 fix provided by @junwang-noaa https://github.com/junwang-noaa to provide the bottom-most atm values to the ice at the first coupling timestep and using an ocean-lag for the first ocean coupling timestep allows the model to run in a concurrent run sequence from ICs with no cold start mediator file required.

Three test cases corresponding to these items were made. At hour 6, the difference in SST from the initial MOM6 SST for the three cases is shown:

[image: sst_diffs] https://user-images.githubusercontent.com/40498404/89181250-50102200-d561-11ea-8725-3b4f5406793e.jpg

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-s2s-model/issues/86#issuecomment-667989689, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7D6TJYLHBCWSSDRSAT2P3R62S4NANCNFSM4M2NDYYQ .

DeniseWorthen commented 4 years ago

Yes, my test cases are on Cheyenne and I ran 48 hours. Do you want the ocean lag SST (case3)- cold start SST (case1) at 48 hours?

The ocean lag is only on the first coupling timestep. On the first coupling timestep, the ocean does not advance. On the second coupling timestep, MOM6 advances by two timesteps. After that, MOM6 advances normally.

The relevant code can be seen LN 1382

junwang-noaa commented 4 years ago

I'd like to see the sst difference between no cold start and with cold start run.

From the code, MOM6 will run with 2*ocean_timestep first then go back to normal ocean timestep. Is "runtype" set to true for the first two run time steps? I seem not able to find that in the code

On Mon, Aug 3, 2020 at 9:02 AM Denise Worthen notifications@github.com wrote:

Yes, my test cases are on Cheyenne and I ran 48 hours. Do you want the ocean lag SST (case3)- cold start SST (case1) at 48 hours?

The ocean lag is only on the first coupling timestep. On the first coupling timestep, the ocean does not advance. On the second coupling timestep, MOM6 advances by two timesteps. After that, MOM6 advances normally.

The relevant code can be seen LN 1382 https://github.com/NOAA-EMC/MOM6/blob/728c429c32b49b192132e00ccd0e12c45ff0b43d/config_src/nuopc_driver/mom_cap.F90#L1382

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-s2s-model/issues/86#issuecomment-668009764, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7D6TIQWDEFC4KDTEKE3ITR62YNFANCNFSM4M2NDYYQ .

DeniseWorthen commented 4 years ago

Runtype is set in the mom_cap at InitializeAdvertise. It is read from nems.configure from the variable "start_type". For non-cesm, it will be set as "initial".

We are currently not really making use of the start_type variable since our set up uses the namelists of components to set whether we're doing an initial vs continue run. We do need to be careful during actual model restarts ("warm start") that we make consistent use of runtype.

junwang-noaa commented 4 years ago

I see, thanks

On Mon, Aug 3, 2020 at 9:44 AM Denise Worthen notifications@github.com wrote:

Runtype is set in the mom_cap at InitializeAdvertise. It is read from nems.configure from the variable "start_type". For non-cesm, it will be set as "initial".

We are currently not really making use of the start_type variable since our set up uses the namelists of components to set whether we're doing an initial vs continue run. We do need to be careful during actual model restarts ("warm start") that we make consistent use of runtype.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-s2s-model/issues/86#issuecomment-668030629, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7D6TLJDEZR4D6ADMNOS63R625J7ANCNFSM4M2NDYYQ .

DeniseWorthen commented 4 years ago

This is the SST difference at 48 hours for the 20120101 ICS between using the ocean lag and using the original cold start mediator file.

sst_diffs_48h

DeniseWorthen commented 4 years ago

Closing; this issue is resolved by the commit of the 1step coldstart PR #187