search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
74 stars 167 forks source link

Restructure ROTDIR for $COMPONENT subfolders #94

Closed KateFriedman-NOAA closed 4 years ago

KateFriedman-NOAA commented 4 years ago

To facilitate a structure more conducive to UFS and merging modeling systems together (e.g. GFS + GEFS) the $CYC subfolder of COMROT/ROTDIR needs to be broken into a new $COMPONENT subfolder. This will require all components of the FV3GFS to update.

From discussions with NCO (Steven Earle):

COMOUTatmos, wave, etc = $COMROOT/$NET/$envir/$RUN.$PDY/$cyc/$COMPONENTatmos, $COMPONENTwave, etc

obs and all the components prdgen/post will go to: COMOUT = $COMROOT/$NET/$envir/$RUN.$PDY/$cyc/$COMPONENT where COMPONENT is one of obs, wave, atmos, ocean, etc.

For GFS, NET = gfs, while RUN is either gfs, gdasenkf or gdas

Will break up $CDUMP.$PDY/$CYC into atmos, wave, etc:

$CDUMP.$PDY/$CYC/wave <--move gdaswave.$PDY to this

Created branch for this work: feature/gfsv16b-restructure This branch will be merged back into feature/gfsv16b when ready. Will document that merge in issue #1

KateFriedman-NOAA commented 4 years ago

@RussTreadon-NOAA @yangfanglin @JessicaMeixner-NOAA I have committed my initial set of changes to add $COMPONENTatmos and $COMPONENTwave variables to the workflow scripts. My local clone on Mars includes modifications for the components:

/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-gfsv16b-restructure

Please feel free to review my initial changes and suggest modifications. I am running a cycled experiment with these changes and will commit further changes as needed. Will merge in recent feature/gfsv16b changes this week as well.

KateFriedman-NOAA commented 4 years ago

First half cycle run, got prep jobs at start of first full cycle to run yesterday, and am awaiting return of Mars (parallel production test) to continue test.

KateFriedman-NOAA commented 4 years ago

(below is fixed based on comment from Fanglin after this was first posted)

Final email with Steven Earle lays out the new $COMPONENT variables and COMIN/COMOUT:

  1. JJOB script for job with multiple components (e.g. forecast): 
    export COMINatmos=${COMINatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos}
export COMOUTatmos=${COMOUTatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos}
export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
  2. JJOB script for job with single component (e.g. non-forecast jobs):

for atmospheric components:

export COMPONENT=${COMPONENT:-atmos}
export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}
export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}

for wave components:

export COMPONENT=${COMPONENT:-wave}
export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}
export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}
yangfanglin commented 4 years ago

Kate,

I thought we should not use

export COMPONENTatmos=${COMPONENTatmos:-atmos} export COMPONENTwave=${COMPONENTwave:-wave}

Instead using atmos and wave directly where they are referred.

export COMINatmos=${COMINatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos} export COMOUTatmos=${COMOUTatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos} export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave} export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}

Fanglin

On Thu, Jul 16, 2020 at 12:32 PM Kate Friedman notifications@github.com wrote:

Final email with Steven Earle lays out the new $COMPONENT variables and COMIN/COMOUT:

  1. JJOB script for job with multiple components (e.g. forecast):

export COMPONENTatmos=${COMPONENTatmos:-atmos} export COMPONENTwave=${COMPONENTwave:-wave}

export COMINatmos=${COMINatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTatmos} export COMOUTatmos=${COMOUTatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTatmos} export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTwave} export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTwave}

  1. JJOB script for job with single component (e.g. non-forecast jobs):

for atmospheric components:

export COMPONENT=${COMPONENT:-atmos} export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}

for wave components:

export COMPONENT=${COMPONENT:-wave} export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/94#issuecomment-659526943, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKY5N2NQ3JYXI5LLG4NFC7DR34TQXANCNFSM4OLNIMHQ .

-- Fanglin Yang, Ph.D. Physical Scientist Environmental Modeling Center National Centers for Environmental Prediction 301-6833722; fanglin.yang@noaa.gov http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/ http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

KateFriedman-NOAA commented 4 years ago

Fanglin, you're right, let me adjust that, sorry, got my notes on that special case mixed up.

KateFriedman-NOAA commented 4 years ago

@yangfanglin fixed at 3d9f0dc

KateFriedman-NOAA commented 4 years ago

Summary of changes for FV3/ufs-weather-model component: none, all changes on global-workflow side to both JJOB and ex scripts, no changes needed in ufs-weather-model

Changes to scripts on workflow side can be seen in feature/gfsv16b-restructure branch or here:

JGLOBAL_FORECAST

[Kate.Friedman@m72a3 feature-gfsv16b-restructure]$ diff ../feature-gfsv16b/jobs/JGLOBAL_FORECAST jobs/JGLOBAL_FORECAST
72a73,80
> # Set wave variables
> if [ ${DO_WAVE:-"NO"} = "YES" ]; then
>   # WAVE component directory
>   export CDUMPwave=${CDUMPwave:-${CDUMP}wave}
>   export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
>   export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
> fi
>

exglobal_fcst_nemsfv3gfs.sh

[Kate.Friedman@m72a3 feature-gfsv16b-restructure]$ diff ../feature-gfsv16b/scripts/exglobal_fcst_nemsfv3gfs.sh scripts/exglobal_fcst_nemsfv3gfs.sh
42a43
> CDUMPwave="${CDUMP}wave"
153,157c154
<     if [ $CDUMP = "gdas" ]; then
<         RSTDIR_WAVE=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart
<     else
<         RSTDIR_WAVE=${RSTDIR_WAVE:-$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart}
<     fi
---
>     RSTDIR_WAVE=$ROTDIR/${CDUMP}.${PDY}/${cyc}/wave/restart
163c160
<     RSTDIR_ATM=${RSTDIR:-$ROTDIR}/${CDUMP}.${PDY}/${cyc}/RERUN_RESTART
---
>     RSTDIR_ATM=${RSTDIR:-$ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos/RERUN_RESTART
205c202
< memdir=$ROTDIR/${prefix}.$PDY/$cyc/$memchar
---
> memdir=$ROTDIR/${prefix}.$PDY/$cyc/atmos/$memchar
211c208
< gmemdir=$ROTDIR/${rprefix}.$gPDY/$gcyc/$memchar
---
> gmemdir=$ROTDIR/${rprefix}.$gPDY/$gcyc/atmos/$memchar
413d409
< #-------------wavewave----------------------
415c411,412
<   for file in $(ls $ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/rmp_src_to_dst_conserv_*) ; do
---
>   # Link WW3 files
>   for file in $(ls $COMINwave/rundata/rmp_src_to_dst_conserv_*) ; do
418c415
<   $NLN $ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/ww3_multi.${CDUMP}wave${WAV_MEMBER}.${cycle}.inp $DATA/ww3_multi.inp
---
>   $NLN $COMINwave/rundata/ww3_multi.${CDUMPwave}${WAV_MEMBER}.${cycle}.inp $DATA/ww3_multi.inp
419a417
>   # Check for expected wave grids for this run
421a420
>
423c422,424
<     $NLN $ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/${CDUMP}wave.mod_def.$wavGRD $DATA/mod_def.$wavGRD
---
>     # Wave IC (restart) file must exist for warm start on this cycle, if not wave model starts from flat ocean
>     # For IAU needs to use sPDY for adding IAU backup of 3h
>     $NLN $COMINwave/rundata/${CDUMPwave}.mod_def.$wavGRD $DATA/mod_def.$wavGRD
426,432c427,435
<   WAVHCYC=${WAVHCYC:-6}
<   WRDATE=`$NDATE -${WAVHCYC} $CDATE`
<   WRPDY=`echo $WRDATE | cut -c1-8`
<   WRcyc=`echo $WRDATE | cut -c9-10`
<   WRDIR=$ROTDIR/gdaswave.${WRPDY}/${WRcyc}/restart
<   datwave=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/
<   wavprfx=${CDUMP}wave${WAV_MEMBER}
---
>   # Wave IC (restart) interval assumes 4 daily cycles (restarts only written by gdas cycle)
>   # WAVHCYC needs to be consistent with restart write interval in ww3_multi.inp or will FAIL
>   export WAVHCYC=${WAVHCYC:-6}
>   export WRDATE=`$NDATE -${WAVHCYC} $CDATE`
>   export WRPDY=`echo $WRDATE | cut -c1-8`
>   export WRcyc=`echo $WRDATE | cut -c9-10`
>   export WRDIR=${ROTDIR}/${CDUMPRSTwave}.${WRPDY}/${WRcyc}/wave/restart
>   export datwave=$COMOUTwave/rundata
>   export wavprfx=${CDUMPwave}${WAV_MEMBER}
444c447
<     wavicefile=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/${CDUMP}wave.${WAVEICE_FID}.${cycle}.ice
---
>     wavicefile=$COMINwave/rundata/${CDUMPwave}.${WAVEICE_FID}.${cycle}.ice
454c457
<     wavcurfile=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/${CDUMP}wave.${WAVECUR_FID}.${cycle}.cur
---
>     wavcurfile=$COMINwave/rundata/${CDUMPwave}.${WAVECUR_FID}.${cycle}.cur
465a469,470
>
>   # Loop for gridded output (uses FHINC)
491a497
> fi #cplwav=true
493,495d498
< fi
< #-------------wavewave----------------------
<
yangfanglin commented 4 years ago

Kate,

The structure of the following block needs to be preserved for restarting wave model from a breakpoint. NCO will save breakpoint restart files for the gfs cycle in a $nwges directory , while restart files for the gdas cycle in $comrot. So in operation RSTDIR_WAVE for the gfs cycle will be defined in an ecflow suite definition file. This is also true for the atmos part.

< if [ $CDUMP = "gdas" ]; then < RSTDIR_WAVE=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart < else < RSTDIR_WAVE=${RSTDIR_WAVE:-$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart} < fi

You can change this block to

if [ $CDUMP = "gdas" ]; then RSTDIR_WAVE=$ROTDIR/${CDUMP}.${PDY}/${cyc}/wave/restart else RSTDIR_WAVE=${RSTDIR_WAVE:-$ROTDIR/${CDUMP}.${PDY}/${cyc}/wave/restart} fi

KateFriedman-NOAA commented 4 years ago

Fanglin, gotcha, changed it to the block you provided at 273c535, thanks!

KateFriedman-NOAA commented 4 years ago

Summary of changes to other FV3GFS components:

GSI:

Fork branch: release/gfsda.v16.0.0-restructure Changes committed: https://github.com/KateFriedman-NOAA/GSI/commit/f6bc0f6a010b0606bf6003f14872fb3ddc80f96f

GLDAS:

Fork branch: gldas_gfsv16_release.v1.3.0-restructure Changes committed: https://github.com/KateFriedman-NOAA/GLDAS/commit/ca6188375dfb46ac8b59346437c1b96ad50ea708

EMC_post:

Fork branch: upp_gfsv16_release.v1.0.10-restructure Changes committed: https://github.com/KateFriedman-NOAA/EMC_post/commit/9cb2723b00c276a2e004a64a4bebf1e0e3f943ca

UFS_UTILS:

Just a three line change to ush/fv3gfs_chgres.sh to add $COMPONENT. We're moving away from that script though right? If so, no changes needed to UFS_UTILS.

WAFS:

Fork branch: gfs_wafs.v5.0.11-restructure Changes committed: https://github.com/KateFriedman-NOAA/EMC_gfs_wafs/commit/52ed70d8351d186f04e25b5fe87f3bfb38b7fa5e

KateFriedman-NOAA commented 4 years ago

Continuing to let test run through additional cycles. Test on Mars:

EXPDIR: /gpfs/dell2/emc/modeling/save/Kate.Friedman/expdir/testrestrucb COMROT: /gpfs/dell3/ptmp/Kate.Friedman/comrot/testrestrucb clone: /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-gfsv16b-restructure

RussTreadon-NOAA commented 4 years ago

Quick look at GFS v16 DA changes looks good. $COMPONENT has been added to DA j-jobs and two exglobal scripts. COMPONENT defaults to "atmos" if not externally set. Seems we could pull these changes into base release/gfsda.v16.0.0 with COMPONENT="" and GFS v16 DA would work as-is in current real-time v16 parallel.

KateFriedman-NOAA commented 4 years ago

Seems we could pull these changes into base release/gfsda.v16.0.0 with COMPONENT="" and GFS v16 DA would work as-is in current real-time v16 parallel.

I agree and am suggesting that path in the email I'm drafting to everyone. The only part that will need more of a pivot is the wave part since it's moving from:

$CDUMPwave.$PDY/$cyc ...to: $CDUMP.$PDY/$CYC/wave

RussTreadon-NOAA commented 4 years ago

OK. I'll wait for and process the PR at the appropriate time.

On Thu, Jul 16, 2020 at 2:50 PM Kate Friedman notifications@github.com wrote:

Seems we could pull these changes into base release/gfsda.v16.0.0 with COMPONENT="" and GFS v16 DA would work as-is in current real-time v16 parallel.

I agree and am suggesting that path in the email I'm drafting to everyone. The only part that will need more of a pivot is the wave part since it's moving from:

$CDUMPwave.$PDY/$cyc ...to: $CDUMP.$PDY/$CYC/wave

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/94#issuecomment-659601967, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGNN63ZCRRL3KGTE2FMIC3DR35DVFANCNFSM4OLNIMHQ .

GeorgeGayno-NOAA commented 4 years ago

Summary of changes to other FV3GFS components:

GSI:

Fork branch: release/gfsda.v16.0.0-restructure Changes committed: KateFriedman-NOAA/GSI@f6bc0f6

GLDAS:

Fork branch: gldas_gfsv16_release.v1.3.0-restructure Changes committed: KateFriedman-NOAA/GLDAS@ca61883

EMC_post:

Fork branch: upp_gfsv16_release.v1.0.10-restructure Changes committed: KateFriedman-NOAA/EMC_post@9cb2723

UFS_UTILS:

Just a three line change to ush/fv3gfs_chgres.sh to add $COMPONENT. We're moving away from that script though right? If so, no changes needed to UFS_UTILS.

The ./ush/fv3gfs_chgres.sh script is used by the old chgres and is obsolete. It has been replaced by the scripts in ./util/gdas_init.

WAFS:

Fork branch: gfs_wafs.v5.0.11-restructure Changes committed: KateFriedman-NOAA/EMC_gfs_wafs@52ed70d

KateFriedman-NOAA commented 4 years ago

The ./ush/fv3gfs_chgres.sh script is used by the old chgres and is obsolete. It has been replaced by the scripts in ./util/gdas_init.

Thanks for the confirmation that fv3gfs_chgres.sh is indeed obsolete now! I'll forgo submitting changes to you for fv3gfs_chgres.sh then, thanks @GeorgeGayno-NOAA !

KateFriedman-NOAA commented 4 years ago

PR for EMC_post submitted: https://github.com/NOAA-EMC/EMC_post/pull/151

KateFriedman-NOAA commented 4 years ago

PR for GSI submitted: https://github.com/NOAA-EMC/GSI/pull/39

PR for GLDAS submitted: https://github.com/NOAA-EMC/GLDAS/pull/8

PR for WAFS submitted: https://github.com/NOAA-EMC/EMC_gfs_wafs/pull/1

KateFriedman-NOAA commented 4 years ago

EMC_post PR https://github.com/NOAA-EMC/EMC_post/pull/151 merged in.

New tag created: upp_gfsv16_release.v1.0.11

Moved feature/gfsv16b-restructure checkout.sh to use new EMC_post tag. Committed at ab89088.

KateFriedman-NOAA commented 4 years ago

GLDAS PR https://github.com/NOAA-EMC/GLDAS/pull/8 merged in.

New tag created: gldas_gfsv16_release.v1.4.0

Moved feature/gfsv16b-restructure checkout.sh to use new GLDAS tag. Committed at 95ce823.

KateFriedman-NOAA commented 4 years ago

Did another sync merge of global-workflow and GSI branches...no new updates to pull in.

Ran new ens_tracker.v.1.1.15.3 from Jiayi successfully. Sent a few small edits back to him and he is now pushing this version to GitHub. Will update on-disk clone of the tracker when on GitHub.

Waiting for WAFS v16 still but will ingest that into system when available and will move the real-time parallel before that.

Running final test with all pieces in place and turned on today (minus WAFS) on Mars. Sent update email to code managers. Putting together notes for Russ for moving the real-time parallel to the updated $COMPONENT pieces.

Need to install updated external pieces on other platforms to support this new structure there (obsproc, fit2obs, tracker).

KateFriedman-NOAA commented 4 years ago

Updated tracker in GitHub: https://github.com/NOAA-EMC/TC_tracker/tree/TC_tracker.v1.1.15.3

KateFriedman-NOAA commented 4 years ago

WCOSS outages this week will delay implementing these changes in the real-time parallel. Plan is to allow real-time parallel to catch back up to real-time over the weekend and early next week ahead of a prod switch next Tuesday. Will set up and test system on Venus this week and plan to move to new structure when the parallel moves to Venus post-prod switch.

Will resync global-workflow and GSI PR branches again before the end.

KateFriedman-NOAA commented 4 years ago

Preparing to move v16rt2 to new structure after WCOSS prod switch this Thursday. Working with Russ to set up Venus as much as we can in advance. Will submit GSI and global-workflow PRs Wednesday (if not before).

KateFriedman-NOAA commented 4 years ago

GSI PR completed: https://github.com/NOAA-EMC/GSI/pull/39

KateFriedman-NOAA commented 4 years ago

Global-workflow PR completed: https://github.com/NOAA-EMC/global-workflow/pull/109