search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
70 stars 162 forks source link

missing the last ocean daily file in S2S run #2659

Closed jiandewang closed 2 weeks ago

jiandewang commented 3 weeks ago

What is wrong?

During previous HR3 and current cycled run that I am testing, we found that the last ocean daily file are not being created

What should have happened?

all ocean daily file should be created

What machines are impacted?

All or N/A

Steps to reproduce

clone lastst g-w setup S2S run check ocean daily file (the last one won't appear on disk and there is a 24hr shift between timestamp inside netcdf file and the timestamp for the file name itself, see additional information below)

Additional information

the root casue is at https://github.com/NOAA-EMC/global-workflow/blob/develop/ush/forecast_postdet.sh#L439-L444 there is a 24hr shifting while the daily file sybmol link is created.

let me use one sample file from cycled run here: /scratch1/NCEPDEV/climate/Jiande.Wang/working/g-w-cycle/cycle/C02/COMROOT/C02/gfs.20210708/00/model_data/ocean/history/gfs.ocean.t00z.daily.f024.nc

so the starting time is 0708 00Z, the time stamp inside the first daily file ( gfs.ocean.t00z.daily.f024.nc) should be 0708 12Z as we are using middle time to represent the time average.

but if you do: (1) ncks -v average_T1,average_T2,average_DT gfs.ocean.t00z.daily.f024.nc tmp-ocean-daily-024.nc (2) ncdump tmp-ocean-daily-024.nc

you get the following:

average_T1:units = "days since 2021-07-07 18:00:00" average_T1 = 1.25 ; <-- this should be 0.25 average_T2 = 2.25 ; <-- this should be 1.25

After HR3 was done, I made a test run but deleted all the daily file symbol links in rundir, that gave me all the daily files we needed with correct timestamp inside rundir. So the problem is in the symbol link creation which has a 24hr shift.

Do you have a proposed solution?

No response

JessicaMeixner-NOAA commented 3 weeks ago

Note that this also means that the gfsarchive (at least for the cycled runs using https://github.com/NOAA-EMC/global-workflow/blob/develop/ci/cases/gfsv17/C384mx025_3DVarAOWCDA.yaml) is failing.

JessicaMeixner-NOAA commented 3 weeks ago

So I think the links are not correct and shifted: 

hfe07:/scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/TMP/RUNDIRS/c48t01/gfsfcst.2021063006/fcst.203600/MOM6_OUTPUT$ ls -l *daily*
-rw-r--r-- 1 Jessica.Meixner climate 105894908 Jun  5 17:14 ocn_daily_2021_06_30.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:41 ocn_daily_2021_07_01.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f024.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:41 ocn_daily_2021_07_02.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f048.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:41 ocn_daily_2021_07_03.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f072.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:41 ocn_daily_2021_07_04.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f096.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:41 ocn_daily_2021_07_05.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f120.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:41 ocn_daily_2021_07_06.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f144.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:42 ocn_daily_2021_07_07.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f168.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:42 ocn_daily_2021_07_08.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f192.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:42 ocn_daily_2021_07_09.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f216.nc
lrwxrwxrwx 1 Jessica.Meixner climate       151 Jun  5 16:42 ocn_daily_2021_07_10.nc -> /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/COMROOT/c48t01/gfs.20210630/06//model_data/ocean/history/gfs.ocean.t06z.daily.f240.nc
hfe07:/scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/testarchive/t11/TMP/RUNDIRS/c48t01/gfsfcst.2021063006/fcst.203600/MOM6_OUTPUT$ ls *daily*
ocn_daily_2021_06_30.nc  ocn_daily_2021_07_02.nc  ocn_daily_2021_07_04.nc  ocn_daily_2021_07_06.nc  ocn_daily_2021_07_08.nc  ocn_daily_2021_07_10.nc
ocn_daily_2021_07_01.nc  ocn_daily_2021_07_03.nc  ocn_daily_2021_07_05.nc  ocn_daily_2021_07_07.nc  ocn_daily_2021_07_09.nc

Instead of: if (( fhr > 0 & fhr % 24 == 0 )); then Maybe if (( fhr < FHMAX & fhr % 24 == 0 )); then ?

I'll try this and report back... Tagging @aerorahul to see if he has any comments as this was added in https://github.com/NOAA-EMC/global-workflow/pull/2286

JessicaMeixner-NOAA commented 3 weeks ago

My fix above did not work. I'm trying this instead now: 

      # Daily output
      if (( fhr > 0 & fhr % 24 == 0 )); then
        (( fhrdaily= fhr - 24 ))
        vdate_daily=$(date --utc -d "${current_cycle:0:8} ${current_cycle:8:2} + ${fhrdaily} hours" +%Y%m%d%H)
        source_file="ocn_daily_${vdate_daily:0:4}_${vdate_daily:4:2}_${vdate_daily:6:2}.nc"
        dest_file="${RUN}.ocean.t${cyc}z.daily.f${fhr3}.nc"
        ${NLN} "${COMOUT_OCEAN_HISTORY}/${dest_file}" "${DATA}/MOM6_OUTPUT/${source_file}"
      fi

I'll keep everyone posted on how it's going...

aerorahul commented 3 weeks ago

2 diffs for your consideration. 1st one is to remove the forecast output directories that are lingering. We will handle this better in the coming days. 2nd diff to set the vdate based on the beginning day of the averaging period.

❯❯❯ git diff                                                                                            
diff --git i/scripts/exglobal_cleanup.sh w/scripts/exglobal_cleanup.sh
index cdc13c11..423296ff 100755
--- i/scripts/exglobal_cleanup.sh
+++ w/scripts/exglobal_cleanup.sh
@@ -2,6 +2,13 @@

 source "${USHgfs}/preamble.sh"

+# Remove DATAoutput from the forecast model run
+# TODO: Handle this better
+DATAfcst="${DATAROOT}/${RUN}fcst.${PDY:-}${cyc}"
+if [[ -d "${DATAfcst}" ]]; then rm -rf "${DATAfcst}"; fi
+#DATAefcs="${DATAROOT}/${RUN}efcs???${PDY:-}${cyc}"
+rm -rf "${DATAROOT}/${RUN}efcs"*"${PDY:-}${cyc}"
+
 ###############################################################
 # Clean up previous cycles; various depths
 # PRIOR CYCLE: Leave the prior cycle alone
diff --git i/ush/forecast_postdet.sh w/ush/forecast_postdet.sh
index d4d1d4ad..f4166497 100755
--- i/ush/forecast_postdet.sh
+++ w/ush/forecast_postdet.sh
@@ -438,6 +438,9 @@ MOM6_postdet() {

       # Daily output
       if (( fhr > 0 & fhr % 24 == 0 )); then
+        # For daily output, the valid date in the file is based on the beginning of avg. period
+        vdate=$(date --utc -d "${vdate:0:8} ${vdate:8:2} - 24 hours" +%Y%m%d%H)
+
         source_file="ocn_daily_${vdate:0:4}_${vdate:4:2}_${vdate:6:2}.nc"
         dest_file="${RUN}.ocean.t${cyc}z.daily.f${fhr3}.nc"
         ${NLN} "${COMOUT_OCEAN_HISTORY}/${dest_file}" "${DATA}/MOM6_OUTPUT/${source_file}"
JessicaMeixner-NOAA commented 3 weeks ago

Okay, I have pushed the first change to the PR I have open. I'm testing the second change now. Will report back. Thanks for the help with these updates @aerorahul !

JessicaMeixner-NOAA commented 3 weeks ago

@aerorahul fix worked: /scratch1/NCEPDEV/climate/Jessica.Meixner/updatemodelspackingw/test02/COMROOT/arch01/gfs.20210630/06/model_data/ocean/history we now have the last daily ocean file and the archive succeeded!!

aerorahul commented 3 weeks ago

@aerorahul fix worked: /scratch1/NCEPDEV/climate/Jessica.Meixner/updatemodelspackingw/test02/COMROOT/arch01/gfs.20210630/06/model_data/ocean/history we now have the last daily ocean file and the archive succeeded!!

Thank you for testing and confirming the change works!