KateFriedman-NOAA
commented
1 year ago Per a discussion in this week's global-workflow team tag-up...checked jobs to see which ones actually access COM_OBS and DMPDIR:
DMPDIR
The prep job accesses DMPDIR via ush/getdump.sh...
ush/getdump.sh:SOURCE_DIR=${3:-$DMPDIR/${CDUMP}${DUMP_SUFFIX}.${PDY}/${cyc}/${COMPONENT}}...to set up COM_OBS with the observation dump files:
jobs/rocoto/prep.sh: rm -f "${COM_OBS}/${OPREFIX}prepbufr"
jobs/rocoto/prep.sh: rm -f "${COM_OBS}/${OPREFIX}prepbufr.acft_profiles"
jobs/rocoto/prep.sh: rm -f "${COM_OBS}/${OPREFIX}nsstbufr"
jobs/rocoto/prep.sh: if [[ ${DONST} = "YES" ]]; then ${NCP} "${COM_OBSDMP}/${OPREFIX}nsstbufr" "${COM_OBS}/${OPREFIX}nsstbufr"; fi
jobs/rocoto/prep.sh: ${NCP} "${COM_OBSDMP}/${OPREFIX}prepbufr" "${COM_OBS}/${OPREFIX}prepbufr"
jobs/rocoto/prep.sh: ${NCP} "${COM_OBSDMP}/${OPREFIX}prepbufr.acft_profiles" "${COM_OBS}/${OPREFIX}prepbufr.acft_profiles"
jobs/rocoto/prep.sh: if [[ ${DONST} = "YES" ]]; then ${NCP} "${COM_OBSDMP}/${OPREFIX}nsstbufr" "${COM_OBS}/${OPREFIX}nsstbufr"; fiThe ush/global_cycle_driver.sh script also accesses directly from DMPDIR if the file variable is not set (although we don't appear to be using this script anymore? don't see reference in logs to usage):
ush/global_cycle_driver.sh:export DMPDIR=${DMPDIR:-$NWPROD}
ush/global_cycle_driver.sh:export FNTSFA=${FNTSFA:-$DMPDIR/${CDUMP}.${PDY}/${cyc}/${COMPONENT}/${CDUMP}.t${cyc}z.rtgssthr.grb}
ush/global_cycle_driver.sh:export FNSNOA=${FNSNOA:-$DMPDIR/${CDUMP}.${PDY}/${cyc}/${COMPONENT}/${CDUMP}.t${cyc}z.snogrb_t1534.3072.1536}
ush/global_cycle_driver.sh:export FNACNA=${FNACNA:-$DMPDIR/${CDUMP}.${PDY}/${cyc}/${COMPONENT}/${CDUMP}.t${cyc}z.seaice.5min.blend.grb}COM_OBS
Other jobs access COM_OBS for observation dump files that were symlinked there in the prep job: anal, eobs, ediag, esfc, sfcanl
Here are the files they access from their scripts:
kate.friedman@clogin03:/lfs/h2/emc/global/noscrub/kate.friedman/git/develop> grep COM_OBS jobs/* -r | grep OPREFIX
jobs/JGDAS_ENKF_DIAG:export PREPQC="${COM_OBS}/${OPREFIX}prepbufr"
jobs/JGDAS_ENKF_DIAG:export TCVITL="${COM_OBS}/${OPREFIX}syndata.tcvitals.tm00"
jobs/JGDAS_ENKF_DIAG: export NSSTBF="${COM_OBS}/${OPREFIX}nsstbufr"
jobs/JGDAS_ENKF_DIAG:export PREPQCPF="${COM_OBS}/${OPREFIX}prepbufr.acft_profiles"
jobs/JGDAS_ENKF_SELECT_OBS:export PREPQC="${COM_OBS}/${OPREFIX}prepbufr"
jobs/JGDAS_ENKF_SELECT_OBS: export NSSTBF="${COM_OBS}/${OPREFIX}nsstbufr"
jobs/JGDAS_ENKF_SELECT_OBS:export PREPQCPF="${COM_OBS}/${OPREFIX}prepbufr.acft_profiles"
jobs/JGLOBAL_ATMOS_ANALYSIS:export PREPQC="${COM_OBS}/${OPREFIX}prepbufr"
jobs/JGLOBAL_ATMOS_ANALYSIS:export TCVITL="${COM_OBS}/${OPREFIX}syndata.tcvitals.tm00"
jobs/JGLOBAL_ATMOS_ANALYSIS: export NSSTBF="${COM_OBS}/${OPREFIX}nsstbufr"
jobs/JGLOBAL_ATMOS_ANALYSIS:export PREPQCPF="${COM_OBS}/${OPREFIX}prepbufr.acft_profiles"
kate.friedman@clogin03:/lfs/h2/emc/global/noscrub/kate.friedman/git/develop> grep COM_OBS scripts/* -r | grep OPREFIX
scripts/exgdas_enkf_sfc.sh:export FNACNA=${FNACNA:-${COM_OBS}/${OPREFIX}seaice.5min.blend.grb}
scripts/exgdas_enkf_sfc.sh:export FNSNOA=${FNSNOA:-${COM_OBS}/${OPREFIX}snogrb_t${JCAP_CASE}.${LONB_CASE}.${LATB_CASE}}
scripts/exgdas_enkf_sfc.sh:[[ ! -f $FNSNOA ]] && export FNSNOA="${COM_OBS}/${OPREFIX}snogrb_t1534.3072.1536"
scripts/exglobal_atmos_analysis.sh:PREPQC=${PREPQC:-${COM_OBS}/${OPREFIX}prepbufr${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:PREPQCPF=${PREPQCPF:-${COM_OBS}/${OPREFIX}prepbufr.acft_profiles${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:NSSTBF=${NSSTBF:-${COM_OBS}/${OPREFIX}nsstbufr${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:SATWND=${SATWND:-${COM_OBS}/${OPREFIX}satwnd.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:OSCATBF=${OSCATBF:-${COM_OBS}/${OPREFIX}oscatw.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:RAPIDSCATBF=${RAPIDSCATBF:-${COM_OBS}/${OPREFIX}rapidscatw.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:GSNDBF=${GSNDBF:-${COM_OBS}/${OPREFIX}goesnd.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:GSNDBF1=${GSNDBF1:-${COM_OBS}/${OPREFIX}goesfv.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:B1HRS2=${B1HRS2:-${COM_OBS}/${OPREFIX}1bhrs2.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:B1MSU=${B1MSU:-${COM_OBS}/${OPREFIX}1bmsu.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:B1HRS3=${B1HRS3:-${COM_OBS}/${OPREFIX}1bhrs3.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:B1HRS4=${B1HRS4:-${COM_OBS}/${OPREFIX}1bhrs4.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:B1AMUA=${B1AMUA:-${COM_OBS}/${OPREFIX}1bamua.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:B1AMUB=${B1AMUB:-${COM_OBS}/${OPREFIX}1bamub.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:B1MHS=${B1MHS:-${COM_OBS}/${OPREFIX}1bmhs.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ESHRS3=${ESHRS3:-${COM_OBS}/${OPREFIX}eshrs3.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ESAMUA=${ESAMUA:-${COM_OBS}/${OPREFIX}esamua.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ESAMUB=${ESAMUB:-${COM_OBS}/${OPREFIX}esamub.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ESMHS=${ESMHS:-${COM_OBS}/${OPREFIX}esmhs.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:HRS3DB=${HRS3DB:-${COM_OBS}/${OPREFIX}hrs3db.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:AMUADB=${AMUADB:-${COM_OBS}/${OPREFIX}amuadb.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:AMUBDB=${AMUBDB:-${COM_OBS}/${OPREFIX}amubdb.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:MHSDB=${MHSDB:-${COM_OBS}/${OPREFIX}mhsdb.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:AIRSBF=${AIRSBF:-${COM_OBS}/${OPREFIX}airsev.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:IASIBF=${IASIBF:-${COM_OBS}/${OPREFIX}mtiasi.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ESIASI=${ESIASI:-${COM_OBS}/${OPREFIX}esiasi.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:IASIDB=${IASIDB:-${COM_OBS}/${OPREFIX}iasidb.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:AMSREBF=${AMSREBF:-${COM_OBS}/${OPREFIX}amsre.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:AMSR2BF=${AMSR2BF:-${COM_OBS}/${OPREFIX}amsr2.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:GMI1CRBF=${GMI1CRBF:-${COM_OBS}/${OPREFIX}gmi1cr.tm00.bufr_d${OSUFFIX}} # GMI temporarily disabled due to array overflow.
scripts/exglobal_atmos_analysis.sh:SAPHIRBF=${SAPHIRBF:-${COM_OBS}/${OPREFIX}saphir.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:SEVIRIBF=${SEVIRIBF:-${COM_OBS}/${OPREFIX}sevcsr.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:AHIBF=${AHIBF:-${COM_OBS}/${OPREFIX}ahicsr.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:SSTVIIRS=${SSTVIIRS:-${COM_OBS}/${OPREFIX}sstvcw.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ABIBF=${ABIBF:-${COM_OBS}/${OPREFIX}gsrcsr.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:CRISBF=${CRISBF:-${COM_OBS}/${OPREFIX}cris.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ESCRIS=${ESCRIS:-${COM_OBS}/${OPREFIX}escris.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:CRISDB=${CRISDB:-${COM_OBS}/${OPREFIX}crisdb.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:CRISFSBF=${CRISFSBF:-${COM_OBS}/${OPREFIX}crisf4.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ESCRISFS=${ESCRISFS:-${COM_OBS}/${OPREFIX}escrsf.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:CRISFSDB=${CRISFSDB:-${COM_OBS}/${OPREFIX}crsfdb.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ATMSBF=${ATMSBF:-${COM_OBS}/${OPREFIX}atms.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ESATMS=${ESATMS:-${COM_OBS}/${OPREFIX}esatms.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:ATMSDB=${ATMSDB:-${COM_OBS}/${OPREFIX}atmsdb.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:SSMITBF=${SSMITBF:-${COM_OBS}/${OPREFIX}ssmit.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:SSMISBF=${SSMISBF:-${COM_OBS}/${OPREFIX}ssmisu.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:SBUVBF=${SBUVBF:-${COM_OBS}/${OPREFIX}osbuv8.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:OMPSNPBF=${OMPSNPBF:-${COM_OBS}/${OPREFIX}ompsn8.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:OMPSTCBF=${OMPSTCBF:-${COM_OBS}/${OPREFIX}ompst8.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:OMPSLPBF=${OMPSLPBF:-${COM_OBS}/${OPREFIX}ompslp.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:GOMEBF=${GOMEBF:-${COM_OBS}/${OPREFIX}gome.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:OMIBF=${OMIBF:-${COM_OBS}/${OPREFIX}omi.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:MLSBF=${MLSBF:-${COM_OBS}/${OPREFIX}mls.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:SMIPCP=${SMIPCP:-${COM_OBS}/${OPREFIX}spssmi.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:TMIPCP=${TMIPCP:-${COM_OBS}/${OPREFIX}sptrmm.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:GPSROBF=${GPSROBF:-${COM_OBS}/${OPREFIX}gpsro.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:TCVITL=${TCVITL:-${COM_OBS}/${OPREFIX}syndata.tcvitals.tm00}
scripts/exglobal_atmos_analysis.sh:B1AVHAM=${B1AVHAM:-${COM_OBS}/${OPREFIX}avcsam.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:B1AVHPM=${B1AVHPM:-${COM_OBS}/${OPREFIX}avcspm.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_analysis.sh:HDOB=${HDOB:-${COM_OBS}/${OPREFIX}hdob.tm00.bufr_d${OSUFFIX}}
scripts/exglobal_atmos_sfcanl.sh: export NSSTBF="${COM_OBS}/${OPREFIX}nsstbufr"
scripts/exglobal_atmos_sfcanl.sh:export FNTSFA=${FNTSFA:-${COM_OBS}/${OPREFIX}rtgssthr.grb}
scripts/exglobal_atmos_sfcanl.sh:export FNACNA=${FNACNA:-${COM_OBS}/${OPREFIX}seaice.5min.blend.grb}
scripts/exglobal_atmos_sfcanl.sh:export FNSNOA=${FNSNOA:-${COM_OBS}/${OPREFIX}snogrb_t${JCAP_CASE}.${LONB_CASE}.${LATB_CASE}}
scripts/exglobal_atmos_sfcanl.sh:[[ ! -f ${FNSNOA} ]] && export FNSNOA="${COM_OBS}/${OPREFIX}snogrb_t1534.3072.1536"Note: it's possible this list is incomplete since we don't run every ops job in development nor am I accounting for the verification system and other downstream monitor/vrfy jobs.
Currently the ush/getdump.sh script does a broad sweep of files in the regular global dump archive (GDA) to pull in dump files for the GFS prep job. Recent GFSv16 testing revealed the need to make this script flexible so a user can define an experimental version of a dump file and have getdump.sh pick it up instead of the default non-experimental production dump file.
Allow getdump.sh to accept a list of variable names where each variable refers to a particular dump file. Define the file list at the configuration level and make it modifiable. Look to the GSI method for inspiration and possible variable names for each dump file for consistency.