search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
74 stars 165 forks source link

Enable IAU across components in the coupled model #1776

Open CatherineThomas-NOAA opened 1 year ago

CatherineThomas-NOAA commented 1 year ago

Description IAU is currently used in the operational GFS as a form of initialization which incorporates increments gradually. Current testing with the coupled model does not include IAU due to difficulties with defining consistent model times and restart intervals (e.g. PR#1700) among components using varying definitions of IAU (4D vs 3D vs none). The desired configuration (taking into consideration current component model capabilities) would have 4DIAU for the atmosphere, 3DIAU for the ocean, and no IAU for the sea ice and waves.

This issue is a catch-all place to capture various configurations and details needed to enable IAU across components.

Requirements 4DIAU is currently operational in GFSv16 for the atmosphere and is a requirement. 3DIAU for the ocean is greatly desired, but not a strict requirement depending on the level of effort.

CatherineThomas-NOAA commented 1 year ago

@junwang-noaa:
@guillaumevernieres has offered to provide settings and dummy increments for the ocean IAU.

junwang-noaa commented 1 year ago

@CatherineThomas-NOAA Thanks. @guillaumevernieres please let me know if you have the ocean IAU dummy increments ready. We can try to set up a coupled case with IAU turned on for both atm and ocean.

aerorahul commented 1 year ago

@CatherineThomas-NOAA Thanks. @guillaumevernieres please let me know if you have the ocean IAU dummy increments ready. We can try to set up a coupled case with IAU turned on for both atm and ocean.

@junwang-noaa I have a zero increment file that @guillaumevernieres provided at 5 degree ocean resolution. He may have 0.25 degree increment file to contribute.

guillaumevernieres commented 1 year ago

@CatherineThomas-NOAA Thanks. @guillaumevernieres please let me know if you have the ocean IAU dummy increments ready. We can try to set up a coupled case with IAU turned on for both atm and ocean.

What resolution do you need @junwang-noaa ? We have 5deg and 1/4 deg increments post processed to be read by mom6's iau. The settings to use mom6 iau are in the workflow.

guillaumevernieres commented 1 year ago

@junwang-noaa : 5deg and 1/4 deg increments saved for posterity here: /scratch2/NCEPDEV/ocean/Guillaume.Vernieres/incr4jun

junwang-noaa commented 1 year ago

@guillaumevernieres The coupled IAU test with error:

960: 960: FATAL from PE 0: initialize_oda_incupd_file: Array size mismatch for oda data. 960:

With the IAU data /scratch2/NCEPDEV/ocean/Guillaume.Vernieres/incr4jun/025/gdas.t12z.ocninc.nc (/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_176039/cpld_bmark_p8_iau_intel/INPUT/mom6_increment.nc)

My run directory is at: /scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_176039/cpld_bmark_p8_iau_intel

Thanks

JessicaMeixner-NOAA commented 1 year ago

@junwang-noaa based on the name of the directory, it looks like p8 is being used as a base instead of cpld_control_gfsv17 which has slightly diverged from the p8 settings.

guillaumevernieres commented 1 year ago

I just checked your test @junwang-noaa . The iau section of the MOM_input is not correct. This section of your MOM_input

! === module MOM_oda_incupd ===
ODA_INCUPD = True               ! [Boolean] default = False
                                ! If true, oda incremental updates will be applied
                                ! everywhere in the domain.
ODA_INCUPD_FILE = "mom6_increment.nc" ! The name of the file with the T,S,h increments.

ODA_TEMPINC_VAR = "pt_inc"      ! default = "ptemp_inc"
                                ! The name of the potential temperature inc. variable in
                                ! ODA_INCUPD_FILE.
ODA_SALTINC_VAR = "s_inc"       ! default = "sal_inc"
                                ! The name of the salinity inc. variable in
                                ! ODA_INCUPD_FILE.
ODA_THK_VAR = "h_fg"            ! default = "h"
                                ! The name of the int. depth inc. variable in
                                ! ODA_INCUPD_FILE.

ODA_UINC_VAR = "u_inc"          ! default = "u_inc"
                                ! The name of the zonal vel. inc. variable in
                                ! ODA_INCUPD_UV_FILE.
ODA_VINC_VAR = "v_inc"          ! default = "v_inc"
                                ! The name of the meridional vel. inc. variable in
                                ! ODA_INCUPD_UV_FILE.
ODA_INCUPD_NHOURS = 6           ! default=3.0
                                ! Number of hours for full update (0=direct insertion).

needs to be replaced by what's currently in the global-workflow.

The name list is here https://github.com/NOAA-EMC/global-workflow/blob/develop/parm/ufs/mom6/MOM_input_template_025 and the section of interest is pasted below:

! === module MOM_oda_incupd ===
ODA_INCUPD = @[ODA_INCUPD]   ! [Boolean] default = False
                             ! If true, oda incremental updates will be applied
                             ! everywhere in the domain.
ODA_INCUPD_FILE = "mom6_increment.nc"   ! The name of the file with the T,S,h increments.

ODA_TEMPINC_VAR = "Temp"        ! default = "ptemp_inc"
                                ! The name of the potential temperature inc. variable in
                           ! ODA_INCUPD_FILE.
ODA_SALTINC_VAR = "Salt"        ! default = "sal_inc"
                                ! The name of the salinity inc. variable in
                                ! ODA_INCUPD_FILE.
ODA_THK_VAR = "h"               ! default = "h"
                                ! The name of the int. depth inc. variable in
                                ! ODA_INCUPD_FILE.
ODA_INCUPD_UV = true            !
ODA_UINC_VAR = "u"              ! default = "u_inc"
                                ! The name of the zonal vel. inc. variable in
                                ! ODA_INCUPD_UV_FILE.
ODA_VINC_VAR = "v"              ! default = "v_inc"
                                ! The name of the meridional vel. inc. variable in
                                ! ODA_INCUPD_UV_FILE.
ODA_INCUPD_NHOURS = @[ODA_INCUPD_NHOURS]            ! default=3.0
junwang-noaa commented 1 year ago

@guillaumevernieres Thanks, the test ran successfully with the changes in oda configurations.

guillaumevernieres commented 1 year ago

@guillaumevernieres Thanks, the test ran successfully with the changes in oda configurations.

Excellent, thanks for doing this @junwang-noaa .

JessicaMeixner-NOAA commented 1 year ago

The issue of writing out model output with 6 hour averaging periods starting at f000 instead of -03 for ocean/ice can be tracked in the following issues: https://github.com/NOAA-EMC/CICE/issues/66 https://github.com/ufs-community/ufs-weather-model/issues/960  https://github.com/NOAA-GFDL/FMS/issues/956
One way to get around this issue as a temporary stop-gap until these features are ready would be to use 3 hourly averaged output times, so that other work can go on towards this.   

I believe another model related issue would then be the writing out of restart files for check-point restarts.    Both MOM6 and CICE6 currently write out restarts every X hours starting from the beginning of the run, so for IAU this would be -03 hour.   One question I had is if the restarts need to be from 00 hour or if the need is simply that all the models write out a check point restart at the same time, and whether that starts at -03 or 00 doesn't matter?   I know the wave model can handle either, so it would depend on the atm component and CMEPS.    

guillaumevernieres commented 1 year ago

I'm not sure I understand what you mean @JessicaMeixner-NOAA . I have not tested the "IAU" mode yet, but this is what the cycling looks like: cycling-iau-mode @aerorahul and myself put that illustration together a few months ago so I would finally understand what the cycling is actually doing. I hope this helps! But just to make sure, we are definitely restarting with restarts that were dumped at the valid time for all the components.

JessicaMeixner-NOAA commented 1 year ago

@guillaumevernieres which part?

The restarts I'm speaking of is for the check-point restarts in the GFS forecast. This is an NCO requirement and they are not used except for to restart the model in case of say a node failure.

guillaumevernieres commented 1 year ago

@guillaumevernieres which part?

The restarts I'm speaking of is for the check-point restarts in the GFS forecast. This is an NCO requirement and they are not used except for to restart the model in case of say a node failure.

@JessicaMeixner-NOAA I see ... Never mind, ignore my amazing post above!

JessicaMeixner-NOAA commented 1 year ago

@aerorahul I think this might have gotten missed by commenting late on a Friday afternoon or perhaps there are some behind the scenes discussions, but I wanted to follow up to see if moving to 3 hourly output while the ocean/ice output would be sufficient to get this issue unblocked? Next, I do assume that restarts could potentially be an issue for the check-point and I was curious about if the restarts every x hour could start at -03 or if it has to start at 00?

aerorahul commented 1 year ago

@JessicaMeixner-NOAA I think the 3-hourly output averaging might be necessary, but not sufficient. For cycling, we are outputting the instantaneous fields from the diag_table_da. I don't think that is the blocker for IAU in the ocean/cice (unless I am missing something). I need to check what the restart interval for MOM6 and CICE is and how it is set. We need those restarts to be available at the beginning of the window and not in the middle (currently as is that case with 3DVar and not IAU). I do believe that it is not really a blocker to get ocean/cice with IAU getting ready, just a matter of understanding what is needed and how to configure it in the workflow.
I can prioritize this with @guillaumevernieres.

JessicaMeixner-NOAA commented 11 months ago

Wanted to post an update here. @guillaumevernieres showed @CatherineThomas-NOAA and myself how to run the workflow with ocean DA turned on. The first thing that was needed for running with IAU in the atm turned on (ie FHROT=3), we needed warm start ICs, so warm start ICs were created for 2021032412 and can be found here: /scratch1/NCEPDEV/climate/Jessica.Meixner/MarineGDAS/ICSDIR/C48O500/gdas.20210324 C384 ICs will be coming later this week. To run with GFS_CYC=0 we needed to update how the date for restarts were being calculated for ICE and OCN and stage one additonal file. Those changes were added to a branch here: https://github.com/JessicaMeixner-NOAA/global-workflow/tree/feature/iau This does not address output issues that will arise in the GFS forecast, but I was able to run 5.5 cycles at C48mx500 GFS_CYC=0 DOIAU=YES. Next step is to make sure we're outputting the instantaneous fields for ocean/ice for the DA (and the C384mx025 ICs).

JessicaMeixner-NOAA commented 11 months ago

@guillaumevernieres has made a PR with the updates: https://github.com/NOAA-EMC/global-workflow/pull/1944 which since last comment on this issue has since been tested with C384 and instantaneous output for MOM6.

Running a gfs forecast still needs to be tried and we know there will be issues at minimum with MOM6 time averaged output. I'm going to address a few other things first, but then I'll circle back to this and also try running S2SW instead of just S2S.

JessicaMeixner-NOAA commented 6 months ago

I have been able to run test cases with IAU. One issue that I have noted is that when running the S2S forecast, the restart files are not written at the same time between MOM6 & the atm model. Each writes out restarts every 12 hours but atm starts from f000 but the ocean model starts from -03.

JessicaMeixner-NOAA commented 6 months ago

Some more details for ATM we get: 

$ ls /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/develop20240227/C384iau04cold/TMP/RUNDIRS/cold04/fcst.127092/RESTART
20210702.090000.sfcanl_data.tile1.nc     20210704.120000.fv_srf_wnd.res.tile5.nc  20210706.120000.fv_core.res.tile3.nc
20210702.090000.sfcanl_data.tile2.nc     20210704.120000.fv_srf_wnd.res.tile6.nc  20210706.120000.fv_core.res.tile4.nc
20210702.090000.sfcanl_data.tile3.nc     20210704.120000.fv_tracer.res.tile1.nc   20210706.120000.fv_core.res.tile5.nc
20210702.090000.sfcanl_data.tile4.nc     20210704.120000.fv_tracer.res.tile2.nc   20210706.120000.fv_core.res.tile6.nc
20210702.090000.sfcanl_data.tile5.nc     20210704.120000.fv_tracer.res.tile3.nc   20210706.120000.fv_srf_wnd.res.tile1.nc
20210702.090000.sfcanl_data.tile6.nc     20210704.120000.fv_tracer.res.tile4.nc   20210706.120000.fv_srf_wnd.res.tile2.nc
20210702.120000.sfcanl_data.tile1.nc     20210704.120000.fv_tracer.res.tile5.nc   20210706.120000.fv_srf_wnd.res.tile3.nc
20210702.120000.sfcanl_data.tile2.nc     20210704.120000.fv_tracer.res.tile6.nc   20210706.120000.fv_srf_wnd.res.tile4.nc
20210702.120000.sfcanl_data.tile3.nc     20210704.120000.phy_data.tile1.nc    20210706.120000.fv_srf_wnd.res.tile5.nc
20210702.120000.sfcanl_data.tile4.nc     20210704.120000.phy_data.tile2.nc    20210706.120000.fv_srf_wnd.res.tile6.nc
20210702.120000.sfcanl_data.tile5.nc     20210704.120000.phy_data.tile3.nc    20210706.120000.fv_tracer.res.tile1.nc
20210702.120000.sfcanl_data.tile6.nc     20210704.120000.phy_data.tile4.nc    20210706.120000.fv_tracer.res.tile2.nc
20210703.000000.ca_data.tile1.nc     20210704.120000.phy_data.tile5.nc    20210706.120000.fv_tracer.res.tile3.nc
20210703.000000.ca_data.tile2.nc     20210704.120000.phy_data.tile6.nc    20210706.120000.fv_tracer.res.tile4.nc
20210703.000000.ca_data.tile3.nc     20210704.120000.sfc_data.tile1.nc    20210706.120000.fv_tracer.res.tile5.nc
20210703.000000.ca_data.tile4.nc     20210704.120000.sfc_data.tile2.nc    20210706.120000.fv_tracer.res.tile6.nc
20210703.000000.ca_data.tile5.nc     20210704.120000.sfc_data.tile3.nc    20210706.120000.phy_data.tile1.nc
20210703.000000.ca_data.tile6.nc     20210704.120000.sfc_data.tile4.nc    20210706.120000.phy_data.tile2.nc
20210703.000000.coupler.res      20210704.120000.sfc_data.tile5.nc    20210706.120000.phy_data.tile3.nc
20210703.000000.fv_core.res.nc       20210704.120000.sfc_data.tile6.nc    20210706.120000.phy_data.tile4.nc
20210703.000000.fv_core.res.tile1.nc     20210705.000000.ca_data.tile1.nc     20210706.120000.phy_data.tile5.nc
20210703.000000.fv_core.res.tile2.nc     20210705.000000.ca_data.tile2.nc     20210706.120000.phy_data.tile6.nc
20210703.000000.fv_core.res.tile3.nc     20210705.000000.ca_data.tile3.nc     20210706.120000.sfc_data.tile1.nc
20210703.000000.fv_core.res.tile4.nc     20210705.000000.ca_data.tile4.nc     20210706.120000.sfc_data.tile2.nc
20210703.000000.fv_core.res.tile5.nc     20210705.000000.ca_data.tile5.nc     20210706.120000.sfc_data.tile3.nc
20210703.000000.fv_core.res.tile6.nc     20210705.000000.ca_data.tile6.nc     20210706.120000.sfc_data.tile4.nc
20210703.000000.fv_srf_wnd.res.tile1.nc  20210705.000000.coupler.res          20210706.120000.sfc_data.tile5.nc
20210703.000000.fv_srf_wnd.res.tile2.nc  20210705.000000.fv_core.res.nc       20210706.120000.sfc_data.tile6.nc
20210703.000000.fv_srf_wnd.res.tile3.nc  20210705.000000.fv_core.res.tile1.nc     20210707.000000.ca_data.tile1.nc
20210703.000000.fv_srf_wnd.res.tile4.nc  20210705.000000.fv_core.res.tile2.nc     20210707.000000.ca_data.tile2.nc
20210703.000000.fv_srf_wnd.res.tile5.nc  20210705.000000.fv_core.res.tile3.nc     20210707.000000.ca_data.tile3.nc
20210703.000000.fv_srf_wnd.res.tile6.nc  20210705.000000.fv_core.res.tile4.nc     20210707.000000.ca_data.tile4.nc
20210703.000000.fv_tracer.res.tile1.nc   20210705.000000.fv_core.res.tile5.nc     20210707.000000.ca_data.tile5.nc
20210703.000000.fv_tracer.res.tile2.nc   20210705.000000.fv_core.res.tile6.nc     20210707.000000.ca_data.tile6.nc
20210703.000000.fv_tracer.res.tile3.nc   20210705.000000.fv_srf_wnd.res.tile1.nc  20210707.000000.coupler.res
20210703.000000.fv_tracer.res.tile4.nc   20210705.000000.fv_srf_wnd.res.tile2.nc  20210707.000000.fv_core.res.nc
20210703.000000.fv_tracer.res.tile5.nc   20210705.000000.fv_srf_wnd.res.tile3.nc  20210707.000000.fv_core.res.tile1.nc
20210703.000000.fv_tracer.res.tile6.nc   20210705.000000.fv_srf_wnd.res.tile4.nc  20210707.000000.fv_core.res.tile2.nc
20210703.000000.phy_data.tile1.nc    20210705.000000.fv_srf_wnd.res.tile5.nc  20210707.000000.fv_core.res.tile3.nc
20210703.000000.phy_data.tile2.nc    20210705.000000.fv_srf_wnd.res.tile6.nc  20210707.000000.fv_core.res.tile4.nc
20210703.000000.phy_data.tile3.nc    20210705.000000.fv_tracer.res.tile1.nc   20210707.000000.fv_core.res.tile5.nc
20210703.000000.phy_data.tile4.nc    20210705.000000.fv_tracer.res.tile2.nc   20210707.000000.fv_core.res.tile6.nc
20210703.000000.phy_data.tile5.nc    20210705.000000.fv_tracer.res.tile3.nc   20210707.000000.fv_srf_wnd.res.tile1.nc
20210703.000000.phy_data.tile6.nc    20210705.000000.fv_tracer.res.tile4.nc   20210707.000000.fv_srf_wnd.res.tile2.nc
20210703.000000.sfc_data.tile1.nc    20210705.000000.fv_tracer.res.tile5.nc   20210707.000000.fv_srf_wnd.res.tile3.nc
20210703.000000.sfc_data.tile2.nc    20210705.000000.fv_tracer.res.tile6.nc   20210707.000000.fv_srf_wnd.res.tile4.nc
20210703.000000.sfc_data.tile3.nc    20210705.000000.phy_data.tile1.nc    20210707.000000.fv_srf_wnd.res.tile5.nc
20210703.000000.sfc_data.tile4.nc    20210705.000000.phy_data.tile2.nc    20210707.000000.fv_srf_wnd.res.tile6.nc
20210703.000000.sfc_data.tile5.nc    20210705.000000.phy_data.tile3.nc    20210707.000000.fv_tracer.res.tile1.nc
20210703.000000.sfc_data.tile6.nc    20210705.000000.phy_data.tile4.nc    20210707.000000.fv_tracer.res.tile2.nc
20210703.120000.ca_data.tile1.nc     20210705.000000.phy_data.tile5.nc    20210707.000000.fv_tracer.res.tile3.nc
20210703.120000.ca_data.tile2.nc     20210705.000000.phy_data.tile6.nc    20210707.000000.fv_tracer.res.tile4.nc
20210703.120000.ca_data.tile3.nc     20210705.000000.sfc_data.tile1.nc    20210707.000000.fv_tracer.res.tile5.nc
20210703.120000.ca_data.tile4.nc     20210705.000000.sfc_data.tile2.nc    20210707.000000.fv_tracer.res.tile6.nc
20210703.120000.ca_data.tile5.nc     20210705.000000.sfc_data.tile3.nc    20210707.000000.phy_data.tile1.nc
20210703.120000.ca_data.tile6.nc     20210705.000000.sfc_data.tile4.nc    20210707.000000.phy_data.tile2.nc
20210703.120000.coupler.res      20210705.000000.sfc_data.tile5.nc    20210707.000000.phy_data.tile3.nc
20210703.120000.fv_core.res.nc       20210705.000000.sfc_data.tile6.nc    20210707.000000.phy_data.tile4.nc
20210703.120000.fv_core.res.tile1.nc     20210705.120000.ca_data.tile1.nc     20210707.000000.phy_data.tile5.nc
20210703.120000.fv_core.res.tile2.nc     20210705.120000.ca_data.tile2.nc     20210707.000000.phy_data.tile6.nc
20210703.120000.fv_core.res.tile3.nc     20210705.120000.ca_data.tile3.nc     20210707.000000.sfc_data.tile1.nc
20210703.120000.fv_core.res.tile4.nc     20210705.120000.ca_data.tile4.nc     20210707.000000.sfc_data.tile2.nc
20210703.120000.fv_core.res.tile5.nc     20210705.120000.ca_data.tile5.nc     20210707.000000.sfc_data.tile3.nc
20210703.120000.fv_core.res.tile6.nc     20210705.120000.ca_data.tile6.nc     20210707.000000.sfc_data.tile4.nc
20210703.120000.fv_srf_wnd.res.tile1.nc  20210705.120000.coupler.res          20210707.000000.sfc_data.tile5.nc
20210703.120000.fv_srf_wnd.res.tile2.nc  20210705.120000.fv_core.res.nc       20210707.000000.sfc_data.tile6.nc
20210703.120000.fv_srf_wnd.res.tile3.nc  20210705.120000.fv_core.res.tile1.nc     20210707.120000.ca_data.tile1.nc
20210703.120000.fv_srf_wnd.res.tile4.nc  20210705.120000.fv_core.res.tile2.nc     20210707.120000.ca_data.tile2.nc
20210703.120000.fv_srf_wnd.res.tile5.nc  20210705.120000.fv_core.res.tile3.nc     20210707.120000.ca_data.tile3.nc
20210703.120000.fv_srf_wnd.res.tile6.nc  20210705.120000.fv_core.res.tile4.nc     20210707.120000.ca_data.tile4.nc
20210703.120000.fv_tracer.res.tile1.nc   20210705.120000.fv_core.res.tile5.nc     20210707.120000.ca_data.tile5.nc
20210703.120000.fv_tracer.res.tile2.nc   20210705.120000.fv_core.res.tile6.nc     20210707.120000.ca_data.tile6.nc
20210703.120000.fv_tracer.res.tile3.nc   20210705.120000.fv_srf_wnd.res.tile1.nc  20210707.120000.coupler.res
20210703.120000.fv_tracer.res.tile4.nc   20210705.120000.fv_srf_wnd.res.tile2.nc  20210707.120000.fv_core.res.nc
20210703.120000.fv_tracer.res.tile5.nc   20210705.120000.fv_srf_wnd.res.tile3.nc  20210707.120000.fv_core.res.tile1.nc
20210703.120000.fv_tracer.res.tile6.nc   20210705.120000.fv_srf_wnd.res.tile4.nc  20210707.120000.fv_core.res.tile2.nc
20210703.120000.phy_data.tile1.nc    20210705.120000.fv_srf_wnd.res.tile5.nc  20210707.120000.fv_core.res.tile3.nc
20210703.120000.phy_data.tile2.nc    20210705.120000.fv_srf_wnd.res.tile6.nc  20210707.120000.fv_core.res.tile4.nc
20210703.120000.phy_data.tile3.nc    20210705.120000.fv_tracer.res.tile1.nc   20210707.120000.fv_core.res.tile5.nc
20210703.120000.phy_data.tile4.nc    20210705.120000.fv_tracer.res.tile2.nc   20210707.120000.fv_core.res.tile6.nc
20210703.120000.phy_data.tile5.nc    20210705.120000.fv_tracer.res.tile3.nc   20210707.120000.fv_srf_wnd.res.tile1.nc
20210703.120000.phy_data.tile6.nc    20210705.120000.fv_tracer.res.tile4.nc   20210707.120000.fv_srf_wnd.res.tile2.nc
20210703.120000.sfc_data.tile1.nc    20210705.120000.fv_tracer.res.tile5.nc   20210707.120000.fv_srf_wnd.res.tile3.nc
20210703.120000.sfc_data.tile2.nc    20210705.120000.fv_tracer.res.tile6.nc   20210707.120000.fv_srf_wnd.res.tile4.nc
20210703.120000.sfc_data.tile3.nc    20210705.120000.phy_data.tile1.nc    20210707.120000.fv_srf_wnd.res.tile5.nc
20210703.120000.sfc_data.tile4.nc    20210705.120000.phy_data.tile2.nc    20210707.120000.fv_srf_wnd.res.tile6.nc
20210703.120000.sfc_data.tile5.nc    20210705.120000.phy_data.tile3.nc    20210707.120000.fv_tracer.res.tile1.nc
20210703.120000.sfc_data.tile6.nc    20210705.120000.phy_data.tile4.nc    20210707.120000.fv_tracer.res.tile2.nc
20210704.000000.ca_data.tile1.nc     20210705.120000.phy_data.tile5.nc    20210707.120000.fv_tracer.res.tile3.nc
20210704.000000.ca_data.tile2.nc     20210705.120000.phy_data.tile6.nc    20210707.120000.fv_tracer.res.tile4.nc
20210704.000000.ca_data.tile3.nc     20210705.120000.sfc_data.tile1.nc    20210707.120000.fv_tracer.res.tile5.nc
20210704.000000.ca_data.tile4.nc     20210705.120000.sfc_data.tile2.nc    20210707.120000.fv_tracer.res.tile6.nc
20210704.000000.ca_data.tile5.nc     20210705.120000.sfc_data.tile3.nc    20210707.120000.phy_data.tile1.nc
20210704.000000.ca_data.tile6.nc     20210705.120000.sfc_data.tile4.nc    20210707.120000.phy_data.tile2.nc
20210704.000000.coupler.res      20210705.120000.sfc_data.tile5.nc    20210707.120000.phy_data.tile3.nc
20210704.000000.fv_core.res.nc       20210705.120000.sfc_data.tile6.nc    20210707.120000.phy_data.tile4.nc
20210704.000000.fv_core.res.tile1.nc     20210706.000000.ca_data.tile1.nc     20210707.120000.phy_data.tile5.nc
20210704.000000.fv_core.res.tile2.nc     20210706.000000.ca_data.tile2.nc     20210707.120000.phy_data.tile6.nc
20210704.000000.fv_core.res.tile3.nc     20210706.000000.ca_data.tile3.nc     20210707.120000.sfc_data.tile1.nc
20210704.000000.fv_core.res.tile4.nc     20210706.000000.ca_data.tile4.nc     20210707.120000.sfc_data.tile2.nc
20210704.000000.fv_core.res.tile5.nc     20210706.000000.ca_data.tile5.nc     20210707.120000.sfc_data.tile3.nc
20210704.000000.fv_core.res.tile6.nc     20210706.000000.ca_data.tile6.nc     20210707.120000.sfc_data.tile4.nc
20210704.000000.fv_srf_wnd.res.tile1.nc  20210706.000000.coupler.res          20210707.120000.sfc_data.tile5.nc
20210704.000000.fv_srf_wnd.res.tile2.nc  20210706.000000.fv_core.res.nc       20210707.120000.sfc_data.tile6.nc
20210704.000000.fv_srf_wnd.res.tile3.nc  20210706.000000.fv_core.res.tile1.nc     ca_data.tile1.nc
20210704.000000.fv_srf_wnd.res.tile4.nc  20210706.000000.fv_core.res.tile2.nc     ca_data.tile2.nc
20210704.000000.fv_srf_wnd.res.tile5.nc  20210706.000000.fv_core.res.tile3.nc     ca_data.tile3.nc
20210704.000000.fv_srf_wnd.res.tile6.nc  20210706.000000.fv_core.res.tile4.nc     ca_data.tile4.nc
20210704.000000.fv_tracer.res.tile1.nc   20210706.000000.fv_core.res.tile5.nc     ca_data.tile5.nc
20210704.000000.fv_tracer.res.tile2.nc   20210706.000000.fv_core.res.tile6.nc     ca_data.tile6.nc
20210704.000000.fv_tracer.res.tile3.nc   20210706.000000.fv_srf_wnd.res.tile1.nc  coupler.res
20210704.000000.fv_tracer.res.tile4.nc   20210706.000000.fv_srf_wnd.res.tile2.nc  fv_core.res.nc
20210704.000000.fv_tracer.res.tile5.nc   20210706.000000.fv_srf_wnd.res.tile3.nc  fv_core.res.tile1.nc
20210704.000000.fv_tracer.res.tile6.nc   20210706.000000.fv_srf_wnd.res.tile4.nc  fv_core.res.tile2.nc
20210704.000000.phy_data.tile1.nc    20210706.000000.fv_srf_wnd.res.tile5.nc  fv_core.res.tile3.nc
20210704.000000.phy_data.tile2.nc    20210706.000000.fv_srf_wnd.res.tile6.nc  fv_core.res.tile4.nc
20210704.000000.phy_data.tile3.nc    20210706.000000.fv_tracer.res.tile1.nc   fv_core.res.tile5.nc
20210704.000000.phy_data.tile4.nc    20210706.000000.fv_tracer.res.tile2.nc   fv_core.res.tile6.nc
20210704.000000.phy_data.tile5.nc    20210706.000000.fv_tracer.res.tile3.nc   fv_srf_wnd.res.tile1.nc
20210704.000000.phy_data.tile6.nc    20210706.000000.fv_tracer.res.tile4.nc   fv_srf_wnd.res.tile2.nc
20210704.000000.sfc_data.tile1.nc    20210706.000000.fv_tracer.res.tile5.nc   fv_srf_wnd.res.tile3.nc
20210704.000000.sfc_data.tile2.nc    20210706.000000.fv_tracer.res.tile6.nc   fv_srf_wnd.res.tile4.nc
20210704.000000.sfc_data.tile3.nc    20210706.000000.phy_data.tile1.nc    fv_srf_wnd.res.tile5.nc
20210704.000000.sfc_data.tile4.nc    20210706.000000.phy_data.tile2.nc    fv_srf_wnd.res.tile6.nc
20210704.000000.sfc_data.tile5.nc    20210706.000000.phy_data.tile3.nc    fv_tracer.res.tile1.nc
20210704.000000.sfc_data.tile6.nc    20210706.000000.phy_data.tile4.nc    fv_tracer.res.tile2.nc
20210704.120000.ca_data.tile1.nc     20210706.000000.phy_data.tile5.nc    fv_tracer.res.tile3.nc
20210704.120000.ca_data.tile2.nc     20210706.000000.phy_data.tile6.nc    fv_tracer.res.tile4.nc
20210704.120000.ca_data.tile3.nc     20210706.000000.sfc_data.tile1.nc    fv_tracer.res.tile5.nc
20210704.120000.ca_data.tile4.nc     20210706.000000.sfc_data.tile2.nc    fv_tracer.res.tile6.nc
20210704.120000.ca_data.tile5.nc     20210706.000000.sfc_data.tile3.nc    phy_data.tile1.nc
20210704.120000.ca_data.tile6.nc     20210706.000000.sfc_data.tile4.nc    phy_data.tile2.nc
20210704.120000.coupler.res      20210706.000000.sfc_data.tile5.nc    phy_data.tile3.nc
20210704.120000.fv_core.res.nc       20210706.000000.sfc_data.tile6.nc    phy_data.tile4.nc
20210704.120000.fv_core.res.tile1.nc     20210706.120000.ca_data.tile1.nc     phy_data.tile5.nc
20210704.120000.fv_core.res.tile2.nc     20210706.120000.ca_data.tile2.nc     phy_data.tile6.nc
20210704.120000.fv_core.res.tile3.nc     20210706.120000.ca_data.tile3.nc     sfc_data.tile1.nc
20210704.120000.fv_core.res.tile4.nc     20210706.120000.ca_data.tile4.nc     sfc_data.tile2.nc
20210704.120000.fv_core.res.tile5.nc     20210706.120000.ca_data.tile5.nc     sfc_data.tile3.nc
20210704.120000.fv_core.res.tile6.nc     20210706.120000.ca_data.tile6.nc     sfc_data.tile4.nc
20210704.120000.fv_srf_wnd.res.tile1.nc  20210706.120000.coupler.res          sfc_data.tile5.nc
20210704.120000.fv_srf_wnd.res.tile2.nc  20210706.120000.fv_core.res.nc       sfc_data.tile6.nc
20210704.120000.fv_srf_wnd.res.tile3.nc  20210706.120000.fv_core.res.tile1.nc
20210704.120000.fv_srf_wnd.res.tile4.nc  20210706.120000.fv_core.res.tile2.nc

For ocean we get: 

$ ls /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/develop20240227/C384iau04cold/TMP/RUNDIRS/cold04/fcst.127092/MOM6_RESTART/
20210702.210000.MOM.res_1.nc  20210704.090000.MOM.res_1.nc  20210705.210000.MOM.res_1.nc  20210707.090000.MOM.res_1.nc
20210702.210000.MOM.res_2.nc  20210704.090000.MOM.res_2.nc  20210705.210000.MOM.res_2.nc  20210707.090000.MOM.res_2.nc
20210702.210000.MOM.res_3.nc  20210704.090000.MOM.res_3.nc  20210705.210000.MOM.res_3.nc  20210707.090000.MOM.res_3.nc
20210702.210000.MOM.res.nc    20210704.090000.MOM.res.nc    20210705.210000.MOM.res.nc    20210707.090000.MOM.res.nc
20210703.090000.MOM.res_1.nc  20210704.210000.MOM.res_1.nc  20210706.090000.MOM.res_1.nc  20210707.120000.MOM.res_1.nc
20210703.090000.MOM.res_2.nc  20210704.210000.MOM.res_2.nc  20210706.090000.MOM.res_2.nc  20210707.120000.MOM.res_2.nc
20210703.090000.MOM.res_3.nc  20210704.210000.MOM.res_3.nc  20210706.090000.MOM.res_3.nc  20210707.120000.MOM.res_3.nc
20210703.090000.MOM.res.nc    20210704.210000.MOM.res.nc    20210706.090000.MOM.res.nc    20210707.120000.MOM.res_4.nc
20210703.210000.MOM.res_1.nc  20210705.090000.MOM.res_1.nc  20210706.210000.MOM.res_1.nc  20210707.120000.MOM.res.nc
20210703.210000.MOM.res_2.nc  20210705.090000.MOM.res_2.nc  20210706.210000.MOM.res_2.nc
20210703.210000.MOM.res_3.nc  20210705.090000.MOM.res_3.nc  20210706.210000.MOM.res_3.nc
20210703.210000.MOM.res.nc    20210705.090000.MOM.res.nc    20210706.210000.MOM.res.nc
JessicaMeixner-NOAA commented 6 months ago

@aerorahul my rundirectory was saved it's here: /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/develop20240227/C384iau04cold/TMP/RUNDIRS/cold04/fcst.127092

JessicaMeixner-NOAA commented 3 months ago

@aerorahul requested two directories, a S2S forecast directory before the exe was run: /scratch1/NCEPDEV/climate/Jessica.Meixner/IAUcheckpointrestarts/iau01/TMP/RUNDIRS/testiau/onlysetup.gfsfcst.2021032418/fcst.2252634

A S2S forecast directory after the exe was run: /scratch1/NCEPDEV/climate/Jessica.Meixner/IAUcheckpointrestarts/iau01/TMP/RUNDIRS/testiau/gfsfcst.2021032418/fcst.1642156

In this current configuration if you look into the /scratch1/NCEPDEV/climate/Jessica.Meixner/IAUcheckpointrestarts/iau01/TMP/RUNDIRS/testiau/gfsfcst.2021032418/fcst.1642156/RESTART directory, restarts are written at 06 and 18 hours and then in the MOM5_RESART it's 03 and 15 hours.

FYI: @junwang-noaa @KateFriedman-NOAA @CatherineThomas-NOAA