search

NOAA-EMC / GSI

Gridpoint Statistical Interpolation
GNU Lesser General Public License v3.0
66 stars 147 forks source link

NCO Bugzilla Tickets opened with GFS v16 DA #137

Closed RussTreadon-NOAA closed 2 years ago

RussTreadon-NOAA commented 3 years ago

NCO opened numerous GFS v16 related bugzilla tickets which must be addressed in future GFS implementations. Some of NCO's bugzilla tickets are DA specific. Others include DA as one component of the larger issue raised in the bugzilla ticket. This issue is opened in NOAA-EMC/GSI to track resolution of the DA component of the following NCO bugzilla tickets:

This issue should be referred to and updated with the specific NOAA-EMC/GSI issues that address the above NCO bugzillas. The above bugzillas will be checked completed as they are resolved in other NOAA-EMC/GSI issues.

RussTreadon-NOAA commented 3 years ago

Additional details regarding the above NCO bugzillas are found in google doc GFS v16 DA bugzillas.

NOTE: Most of the information from the above google doc has been included in this issue as comments below.

RussTreadon-NOAA commented 3 years ago

Bugzilla 1196 The requested changes have been committed to release/gfsda.v16.1.0. Done at f808885 and reported in issue #120. Bugzilla 1196 may be closed upon NCO's review and approval of the change.

Bugzilla 1198 The requested changes for global DA scripts have been committed to release/gfsda.v16.1.0. Done at f808885 and reported in issue #120.

Numerous occurrences of ERRSCRIPT remain in release/gfsda.v16.1.0 as listed below:

None of the above files are used by NCEP operations. Some of the above may not even be used by developers. Obsolete files should be removed from the NOAA-EMC/GSI repository.

When creating a release branch or tag for NCO, files and directories not used by the package being implemented should be removed from the release branch or tag. This along with removing obsolete files from the NOAA-EMC/GSI repository should satisfy the requirements of bugzilla 1198.

Bugzilla 1198 may be closed upon NCO's review and approval of changes committed at f808885 along with clean up procedure outline above.

RussTreadon-NOAA commented 3 years ago

Bugzilla 1205 Examination of the segmentation fault revealed malformed format statements in read_fl_hdob.f90. The developer was contacted and confirmed that the lines in question were for debug purposes only and could be removed. Four sets of debug prints were removed from read_fl_hdob.f90. With this change the global_gsi.x executable built with -check all was able to run to completion.

The updated read_fl_hdob.f90 was committed to release/gfsda.v16.1.0 at 4fda88e and reported in issue #120. Bugzilla 1205 may be closed upon NCO's review and approval of the change.

Bugzilla 1206 Investigation the global_enkf.x segmentation fault when built with -check all shows this behavior to be limited to compilation with ips/18.0.1.163. global_enkf.x runs to completion with -check all when compiled with ips/19.1.2.254. Thus, the -check all failure is due to a specific version of the Intel fortran compiler and not the EnKF update code. As such, no changes were committed to release/gfsda.v16.1.0 in response to this bugzilla. A much longer discussion of these findings is found in Bugzilla 1206.

Bugzilla 1206 may be closed upon NCO's review and approval.

RussTreadon-NOAA commented 3 years ago

Bugzilla 1212, 1216, and 1221 Issue #123 opened to address bugzillas 1212, 1216, and 1221 as they pertain to OznMon.

RussTreadon-NOAA commented 3 years ago

Cross link with corresponding global-workflow issues

The changes requested in several GFS v16 DA bugzillas also include other GFS v16 components. These multi-component bugzillas have corresponding global-workflow issues. The table below maps bugzillas reported in this NOAA-EMC/GSI issue to corresponding global-workflow issues.

NCO bugzilla global-workflow issue
216 #292
1198 #294
1210 #289
1212 #293
1218 #300
1221 #293
1222 #298
RussTreadon-NOAA commented 3 years ago

Bugzilla 1214 DA may be done with this bugzilla. The following comment was written by NCO in bugzilla 1214.

  1. DA:"GFS v16 DA makes extensive use of dynamically allocated arrays. This is necessary since we run the package at various resolutions. Also, the nature of the observing system implies variability in the amount of data processed in any given cycle. A small number of arrays are fixed but these are for things which are truly fixed size arrays (e.g, size of tables)"

To the extent that the above response is satisfactory, bugzilla 1214 may be closed upon NCO's review and approval.

RussTreadon-NOAA commented 3 years ago

Bugzilla 216 NCO's "go to" search command found 875 "go to" in GFS v16 DA. A large number of these "go to" are in code not executed by operational GFS DA. Filtering out the non-operational "go to" left 81 occurrences of "go to". Some of the remaining 81 "go to" are in comments.

One large step toward reducing the number of "go to" in GFS DA is to remove all files from the GFS DA release branch that are not required to run GFS DA in operations. The "go to" that remain in the GFS DA release branch after this pruning should be examined and removed as possible.

RussTreadon-NOAA commented 3 years ago

Bugzilla 1210 GFS DA is not mentioned in this bugzilla. This noted, DA developers should be aware of this implementation standard and ensure continued compliance in future DA updates.

Please see global-workflow issue #289 for additional information.

RussTreadon-NOAA commented 3 years ago

Bugzilla 1212 This is a large bugzilla request. The request contains 5 sub-points. Resolution of this bugzilla requires a top down and bottom up sweep of DA jobs, scripts, and ush in conjunction with global-workflow config files to ensure the requested consistency is achieved.

Issue #123 addresses this bugzilla with regards to how it pertains to OznMon. As noted above, other pieces of GFS DA need to examined for compliance with the stated standards in bugzilla 1212.

RussTreadon-NOAA commented 3 years ago

Bugzilla 1218 This bugzilla addresses several GFS v16 components. The DA response to this bugzilla is twofold:

  1. Some DA makefiles flagged for not including a debug or clean options are not used by GFS DA. These codes should be removed from the GFS DA release branch.
  2. Makefile found in the build directory are created by cmake upon execution of ush\build_all_cmake.sh. The build directory is not required upon successful execution of cmake. Removal of build could be added to ush/build_all_cmake. It may also be possible to modify the cmake build so that intermediate directories are automatically removed upon successful installation.

These options and possibly new options, too, should be explored in future GFS DA implementations.

RussTreadon-NOAA commented 3 years ago

Bugzilla 1221 This bugzilla involves more than just GFS DA. The DA aspect of this bugzilla involves three DA scripts. The following DA scripts were flagged for including hardwired default paths for NDATE:

exgdas_atmos_verfozn.sh:export NDATE=${NDATE:-/nwprod/util/exec/ndate}
exgdas_atmos_verfrad.sh:export NDATE=${NDATE:-/nwprod/util/exec/ndate}
exgdas_atmos_vminmon.sh:export NDATE=${NDATE:-/nwprod/util/exec/ndate}

The prod_util module defines variable NDATE. As such is it not necessary to provide a default path for NDATE. Instead, developers need to ensure that the parent job loads the prod_util module, thereby defining NDATE. Given this the NDATE lines above may be removed from the scripts.

Issue #123 will resolve bugzilla 1221 for exgdas_atmos_verfozn.sh. Resolution of bugzilla 1221 for exgdas_atmos_verfrad and exgdas_atmos_vminmon.sh needs to be recorded in existing or new NOAA-EMC/GSI issue(s).

RussTreadon-NOAA commented 3 years ago

Bugzilla 1222 This bugzilla involves more than just GFS DA. The DA aspect of this bugzilla pertains to:

gsi.fd/util/Misc/Makefile:COREINC       = /scratch1/portfolios/NCEPDEV/da/save/Michael.Lueken/nwprod/incmod
gsi.fd/util/Misc/Makefile:CORELIB       = /scratch1/portfolios/NCEPDEV/da/save/Michael.Lueken/nwprod/lib

The hardwired paths for COREINC and CORELIB violate WCOSS Implementation Standards.

Note that util/Misc is not used by GFS DA. Thus, the easiest resolution of the DA part of bugzilla 1222 is to remove the util/Misc directory from the GFS DA release branch. A similar pruning exercise has been cited as a way to resolve other GFS DA bugzillas.

GFS DA release branches up to and including GFS DA v16 have only pruned fix and libsrc. More aggressive pruning should be applied to future GFS DA release branches. NCO requests this action independent of bugzilla 1222. Their preference is to only receive, install, and implement components that are executed in operations.

RussTreadon-NOAA commented 3 years ago

Changes committed to release/gfsda.v16.1.0 at 532fa23 address the following NCO GFS v16 DA bugzillas. Please see issue #120 for additional information.

Bugzilla 1221 The hardwired default path to NDATE is removed from scripts/exgdas_enkf_fcst.sh

Bugzilla 1222 util/Misc/Makefile was flagged for containing a hardwired path to a user directory. This Makefile is not used to build GFS DA apps. As such, it may be removed from the NCO GFS DA installation. Two changes committed to release/gfsda.v16.1.0 at 532fa23 achieve this end:

  1. ush/prune_4nco_global.sh is added. This script uses git rm -r to remove select directories and files from the working copy of the given cloned branch. Execution of ush/prune_4nco_global.sh removes util/Misc/Makefile, thereby addressing this aspect of Bugzilla 1222.
  2. ush/build_all_cmake.sh is modified to include a NCO build option. When NCO is passed as command line argument, script ush/prune_4nco_global.sh is executed. This script removes directories and files not used by GFS DA. The NCO flag also triggers the VERBOSE setting for make to generate additional run time output documenting the details of the NCO build. Upon successful completion of make, the cmake build directory is removed.

Bugzilla 216 Script ush/prune_4nco_global.sh addresses the GFS DA component of this bugzilla. Of the 800+ go to statements identified in GFS v16.1 DA, most are not in codes used by GFS v16.1 DA. ush/prune_4nco_global.sh removes these non-operational directories and files. The remaining code contains 80 go to. Some of these are found in comments. Filtering out the go to in comments leaves 62 go to in GFS DA code. These remaining go to will be addressed in future NOAA-EMC/GSI development.

None of the changes committed to release/gfsda.v16.1.0 at 532fa23 alter analysis results.

RussTreadon-NOAA commented 3 years ago

Bugzilla 1221 Commit 599a77c (issue #120) removes the hard coded default path for NDATE in four monitoring scripts used in GFS v16 operations. This commit addresses the DA aspect of bugzilla 1221.

RussTreadon-NOAA commented 2 years ago

Update this issue with a review of the above 12 DA related bugzillas with respect to release/gfsda.v16.3.0

Bugzilla 216 A check of release/gfsda.v16.3.0 using NCO's go to check finds 806 occurrences. Application of the prune script reduces this to 73 occurrences. Additional reduction is possible by adding to the prune script removal of non-GFS pieces of the GSI clone. In fact, updating the prune script is necessary since non-GFS pieces have been added to the GSI repository since the script was committed. These non-GFS pieces should not be part of the GFS v16.3.0 package delivered to NCO.

An updated prune_4nco_global.sh was committed to fork Russ.Treadon-NOAA/gsi/release/gfsda.v16.3.0 at 71495be. With this updated script the number of go to in gsi.fd drops to 68. Of these 68 go to 18 are found in comments or the name of a subroutine. This leaves 50 go to statements in code.

In order to activate the prune option for GFS v16.3.0, the export PRUNE_4NCO="YES" line in sorc/build_gsi.sh needs to be uncommented in g-w release/gfs.v16.3.0.

RussTreadon-NOAA commented 2 years ago

Bugzilla 1196

This bugzilla, _improve error message in gfs_atmosanalysis job for missing enkf input, has been re-examined using release/gfsda.v16.3.0. Previous statements regarding improved messaging when encountering missing or zero length input atmospheric and surface hi-res and enkf input files remain valid.

Below is a sample of tests run on WCOSS2 and the resulting error message:

missing sigf04

 READ_FILES: ***ERROR*** insufficient atm fcst for 4densvar:  PROGRAM STOPS
 READ_FILES: ***ERROR*** file sigf04 missing:  PROGRAM STOPS

missing sigf06 nid001034.cactus.wcoss2.ncep.noaa.gov 11: GESINFO: ***ERROR*** sigf06 NOT AVAILABLE: PROGRAM STOPS

zero length sigf09

 READ_FILES: ***ERROR*** insufficient atm fcst for 4densvar:  PROGRAM STOPS
 READ_FILES: ***ERROR*** file sigf09 missing:  PROGRAM STOPS

missing sfcf03

 READ_FILES: ***ERROR*** insufficient sfc fcst for 4densvar:  PROGRAM STOPS
 READ_FILES: ***ERROR*** file sfcf03 missing:  PROGRAM STOPS

missing sfcf06

nid002205.cactus.wcoss2.ncep.noaa.gov 54:  GESINFO:  ***ERROR*** sfcf06 NOT AVAILABLE: PROGRAM STOPS
nid002205.cactus.wcoss2.ncep.noaa.gov 55:  GESINFO:  ***ERROR*** sfcf06 NOT AVAILABLE: PROGRAM STOPS

zero length sfcf08

 READ_FILES: ***ERROR*** insufficient sfc fcst for 4densvar:  PROGRAM STOPS
 READ_FILES: ***ERROR*** file sfcf08 missing:  PROGRAM STOPS

missing sfcf06_anlgrid

nid001053.cactus.wcoss2.ncep.noaa.gov 0:  READ_GFSNCSFC_ANL:  ***ERROR*** sfcf06_anlgrid NOT AVAILABLE: PROGRAM STOPS                  
 ****STOP2****  ABORTING EXECUTION w/code=         999

missing sigf05_ens_mem001

nid001001.cactus.wcoss2.ncep.noaa.gov 0:   PARALLEL_READ_GFSNC_STATE:  ***ERROR*** ./ensemble_data/sigf05_ens_mem001
  NOT AVAILABLE: PROGRAM STOPS
 ****STOP2****  ABORTING EXECUTION w/code=         999

zero length sigf07_ens_mem075

nid00.569.cactus.wcoss2.ncep.noaa.gov 726:   PARALLEL_READ_GFSNC_STATE:  ***ERROR*** ./ensemble_data/sigf07_ens_mem075
  NOT AVAILABLE: PROGRAM STOPS

missing sigf09_ens_mem033

nid001193.cactus.wcoss2.ncep.noaa.gov 320:   PARALLEL_READ_GFSNC_STATE:  ***ERROR*** ./ensemble_data/sigf09_ens_mem033
  NOT AVAILABLE: PROGRAM STOPS
RussTreadon-NOAA commented 2 years ago

Bugzilla 1198

This bugzilla, _Improve error handling to use err_chk/errexit in gfs scripts, has been re-examined using release/gfsda.v16.3.0. Please note that this bugzilla refers to more than just DA components. Only the DA aspect of this bugzilla has been examined in release/gfsda.v16.3.0.

The following gfsda.v16.3.0 scripts are found to reference ERRSCRIPT

./ush/para_config.gfs_analysis:export ERRSCRIPT='eval [[ $err = 0 ]]'
./ush/para_config.gdas_analysis_high:export ERRSCRIPT='eval [[ $err = 0 ]]'
./scripts/exgdas_efsoi_update.sh:ERRSCRIPT=${ERRSCRIPT:-'eval [[ $err = 0 ]]'}
./scripts/exgdas_efsoi_update.sh:      $ERRSCRIPT || exit 3
./scripts/exgdas_efsoi_update.sh:$ERRSCRIPT || exit 2
./scripts/exgdas_efsoi.sh:ERRSCRIPT=${ERRSCRIPT:-'eval [[ $err = 0 ]]'}
./scripts/exgdas_efsoi.sh:      $ERRSCRIPT || exit 3
./scripts/exgdas_efsoi.sh:$ERRSCRIPT || exit 2
./util/global_angupdate/JGDAS_ANGUPDATE:export ERRSCRIPT=${ERRSCRIPT:-$err_chk}
./util/global_angupdate/exglobal_angupdate.sh.ecf:#     ERRSCRIPT     Error processing script
./util/global_angupdate/exglobal_angupdate.sh.ecf:#                  $ERRSCRIPT
./util/global_angupdate/exglobal_angupdate.sh.ecf:export ERRSCRIPT=${ERRSCRIPT:-'eval [[ $err = 0 ]]'}
./util/global_angupdate/exglobal_angupdate.sh.ecf:$ERRSCRIPT||exit 2

None of these scripts are used in the operational gfsda.v16.3.0 package. As such, these files may be removed during the installation of the gfs.v16.3.0 package. gdsda.v16.3.0 script ush/prune_4_nco_global.sh does this.

A check of the installed gfs.v16.3.0 package with the prune script active finds no occurrences of ERRSCRIPT in sorc/gsi.fd.

The recommended action for this bugzilla with regards to the DA component is to activate the prune option in g-w release/gfs.v16.3.0 sorc/build_gs.sh. Currently this script has

# If NCO wants to build, uncomment this line                                                                                                                
#export PRUNE_4NCO="YES"

The recommendation is to uncomment the PRUNE_4NCO line so that sorc/build_gsi.sh reads

# If NCO wants to build, uncomment this line                                                                                                                
export PRUNE_4NCO="YES"

With this change the above gfsda.v16.3.0 scripts will be removed from the gfs.v16.3.0 installation.

RussTreadon-NOAA commented 2 years ago

Bugzillas 1205 and 1206

These bugzillas,

have be re-examined in the context of release/gfsda.v16.3.0.

The following lines were added to ush/build_4nco_global.sh 

# Optionally set compiler flags
export FFLAGS="-check all,noarg_temp_created"

As explained in bugzilla 1205, the noarg_temp_created option is included to turn off warnings noting the creation of temporary storage for subroutine arguments. Absent this argument an extremely large number of warning messages is generated. As these warning messages are only informative in nature, it was decided to turn off generation of these warnings via the noarg_temp_created option.

After compiling all gfsda.v16.3.0 executables with -check all,noarg_temp_created, the g-w was used to run the 2021110806 cycle using guess files from the EMC retrospective v16.3 parallel, retro1-v16-ecf. The following DA executables failed with -check all errors:

Note that enkf.x did not fail with -check all. As such, the previously reported findings regarding bugzilla 1206 remain valid. Bugzilla 1206 may be closed upon approval by NCO.

The discussion which follows pertains to bugzilla 1205.

The errors caught by -check all fell into three categories:

One of the errors, an example of shape mismatch, is discussed at length in GSI issue #256.

Source code changes were made to correct the errors identified by compiling and running with -check all. These changes were committed to fork Russ.Treadon-NOAA/release/gfsda.v16.3.0 at 053e447

The g-w was used to run a cycled parallel from 2021110806 through 2021110900 with the updated executables. All jobs in all cycles ran to completion. The results from the test were saved and the same period cycled through again using exectuables built from the original, unaltered release/gfsda.v16.3.0. Identical results were obtained.

This test indicates the that changes to the release/gfsda.v16.3.0 source code at 053e447 to address issues identified by -check all do not alter gfs.v16.3.0 cycled results.

NCO may close bugzilla 1205 with acceptance of the changes committed at 053e447.

RussTreadon-NOAA commented 2 years ago

Bugzilla 1196

Given NCO feedback, add FATAL to error message string. The previously run tests were rerun on WCOSS2 with the revised error message. Below are the updated error messages:

missing sigf04

 READ_FILES: ***FATAL ERROR*** insufficient atm fcst for 4densvar:  PROGRAM STOPS
 READ_FILES: ***FATAL ERROR*** file sigf04 missing:  PROGRAM STOPS

missing sigf06

nid001111.cactus.wcoss2.ncep.noaa.gov 9:   GESINFO:  ***FATAL ERROR*** sigf06 NOT AVAILABLE: PROGRAM STOPS
nid001111.cactus.wcoss2.ncep.noaa.gov 2:   GESINFO:  ***FATAL ERROR*** sigf06 NOT AVAILABLE: PROGRAM STOPS

zero length sigf09

 READ_FILES: ***FATAL ERROR*** insufficient atm fcst for 4densvar:  PROGRAM STOPS
 READ_FILES: ***FATAL ERROR*** file sigf09 missing:  PROGRAM STOPS

missing sfcf03

 READ_FILES: ***FATAL ERROR*** insufficient sfc fcst for 4densvar:  PROGRAM STOPS
 READ_FILES: ***FATAL ERROR*** file sfcf03 missing:  PROGRAM STOPS

missing sfcf06

nid002602.cactus.wcoss2.ncep.noaa.gov 321:  GESINFO:  ***FATAL ERROR*** sfcf06 NOT AVAILABLE: PROGRAM STOPS
nid002608.cactus.wcoss2.ncep.noaa.gov 416:  GESINFO:  ***FATAL ERROR*** sfcf06 NOT AVAILABLE: PROGRAM STOPS
nid002621.cactus.wcoss2.ncep.noaa.gov 589:  GESINFO:  ***FATAL ERROR*** sfcf06 NOT AVAILABLE: PROGRAM STOPS
nid002622.cactus.wcoss2.ncep.noaa.gov 609:  GESINFO:  ***FATAL ERROR*** sfcf06 NOT AVAILABLE: PROGRAM STOPS

zero length sfcf08

 READ_FILES: ***FATAL ERROR*** insufficient sfc fcst for 4densvar:  PROGRAM STOPS
 READ_FILES: ***FATAL ERROR*** file sfcf08 missing:  PROGRAM STOPS

missing sfcf06_anlgrid

nid001003.cactus.wcoss2.ncep.noaa.gov 0:  READ_GFSNCSFC_ANL:  ***FATAL ERROR*** sfcf06_anlgrid NOT AVAILABLE: PROGRAM STOPS
 ****STOP2****  ABORTING EXECUTION w/code=         999

missing sigf05_ens_mem001

nid001108.cactus.wcoss2.ncep.noaa.gov 0:   PARALLEL_READ_GFSNC_STATE:  ***FATAL ERROR***
 ./ensemble_data/sigf05_ens_mem001 NOT AVAILABLE: PROGRAM STOPS
 ****STOP2****  ABORTING EXECUTION w/code=         999

zero length sigf07_ens_mem075

nid002613.cactus.wcoss2.ncep.noaa.gov 726:   PARALLEL_READ_GFSNC_STATE:  ***FATAL ERROR***
 ./ensemble_data/sigf07_ens_mem075 NOT AVAILABLE: PROGRAM STOPS

missing sigf09_ens_mem033

nid002676.cactus.wcoss2.ncep.noaa.gov 320:   PARALLEL_READ_GFSNC_STATE:  ***FATAL ERROR***
 ./ensemble_data/sigf09_ens_mem033 NOT AVAILABLE: PROGRAM STOPS

The changes required for the above changes were committed to Russ.Treadon-NOAA/release/gfsda.v16.3.0 at a1dfd2c

NCO also requests the full path and source filename be written to run time output in the event of a missing atm or sfc input file. The gsi.x does not read in or use the full path to input files. All input and output files are accessed via names local with respect to the run directory or a sub-directory created within the sub-directory.

Since the GSI supports a variety in input/output formats for atm and sfc files, gsi.x uses internally defined common names for all formats. For example, sigf06 is the name by which gsi.x accesses the 6 hour atmospheric forecast from the previous cycle regardless of the input file format (sigio, nemsio, or netcdf).

If the full path along with the source filename is necessary, extensive changes will need to be made to the GSI. If this is necessary, time should be taken to design a robust, easy to install, and easy to maintain approach. This effort should be undertaken by the DA code manager in conjunction with EIB to ensure a consistent approach across all DA and, hopefully, all GFS apps.

RussTreadon-NOAA commented 2 years ago

Bugzilla 1210

While GFS DA is not mentioned in this bugzilla, the request made in this bugzilla is good for DA developers to be aware of and adhere to.

A check of DA jobs in release/gfsda.v16.3.0 finds that compath.py is in the following jobs:

russ.treadon@clogin04:/lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/fork> grep -r compath .
./jobs/JGDAS_ENKF_DIAG:    export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$RUN.$PDY/$cyc/$COMPONENT}
./jobs/JGDAS_ENKF_DIAG:    export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$GDUMP.$gPDY/$gcyc/$COMPONENT}
./jobs/JGLOBAL_ATMOS_ANALYSIS_CALC:    export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$RUN.$PDY/$cyc/$COMPONENT}
./jobs/JGLOBAL_ATMOS_ANALYSIS_CALC:    export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$GDUMP.$gPDY/$gcyc/$COMPONENT}
./jobs/JGDAS_ATMOS_ANALYSIS_DIAG:    export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$RUN.$PDY/$cyc/$COMPONENT}
./jobs/JGDAS_ATMOS_ANALYSIS_DIAG:    export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$GDUMP.$gPDY/$gcyc/$COMPONENT}
./jobs/JGDAS_ATMOS_CHGRES_FORENKF:    export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$RUN.$PDY/$cyc/$COMPONENT}
./jobs/JGDAS_ATMOS_CHGRES_FORENKF:    export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$GDUMP.$gPDY/$gcyc/$COMPONENT}
./jobs/JGDAS_ENKF_SFC:    export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$RUN.$PDY/$cyc/$COMPONENT}
./jobs/JGDAS_ENKF_SFC:    export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$GDUMP.$gPDY/$gcyc/$COMPONENT}
./jobs/JGDAS_ENKF_ECEN:    export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$RUN.$PDY/$cyc/$COMPONENT}
./jobs/JGDAS_ENKF_ECEN:    export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$GDUMP.$gPDY/$gcyc/$COMPONENT}
./jobs/JGDAS_ENKF_SELECT_OBS:    export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$RUN.$PDY/$cyc/$COMPONENT}
./jobs/JGDAS_ENKF_SELECT_OBS:    export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$GDUMP.$gPDY/$gcyc/$COMPONENT}
./jobs/JGLOBAL_ATMOS_ANALYSIS:    export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$RUN.$PDY/$cyc/$COMPONENT}
./jobs/JGLOBAL_ATMOS_ANALYSIS:    export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$GDUMP.$gPDY/$gcyc/$COMPONENT}
./util/Ozone_Monitor/nwprod/gdas_oznmon/jobs/JGDAS_ATMOS_VERFOZN:export OZN_TANKDIR=${OZN_TANKDIR:-$(compath.py ${envir}/${NET}/${gfs_ver})}
./util/Ozone_Monitor/nwprod/gdas_oznmon/jobs/JGDAS_ATMOS_VERFOZN:export COM_IN=${COM_IN:-$(compath.py ${envir}/${NET}/${gfs_ver})}
./util/Radiance_Monitor/nwprod/gdas_radmon/jobs/JGDAS_ATMOS_VERFRAD:export TANKverf=${TANKverf:-$(compath.py ${envir}/${NET}/${gfs_ver})}
./util/Radiance_Monitor/nwprod/gdas_radmon/jobs/JGDAS_ATMOS_VERFRAD:export COM_IN=${COM_IN:-$(compath.py ${envir}/${NET}/${gfs_ver})}
./util/Radiance_Monitor/nwprod/gdas_radmon/scripts/exgdas_atmos_verfrad.sh:export COM_IN=${COM_IN:-$(compath.py ${envir}/${NET}/${gfs_ver})}
./util/Minimization_Monitor/nwprod/gdas/jobs/JGDAS_ATMOS_VMINMON:export COM_IN=${COM_IN:-$(compath.py ${envir}/${NET}/${gfs_ver})}
./util/Minimization_Monitor/nwprod/gfs/jobs/JGFS_ATMOS_VMINMON:export COM_IN=${COM_IN:-$(compath.py ${envir}/${NET}/${gfs_ver})}

The uses of compath.py in DA jobs/ JGDAS_ENKF*, JGDAS_ATMOS_ANALYSIS*, JGDAS_ATMOS_CHGRES* and JGLOBAL_ATMOS* jobs is consistent with the request in bugzilla 1210. compath.py is used to set the path to upstream model obsproc.

The use of compath.py in DA verification jobs JGDAS_ATMOS_V* and gdas_radmon/scripts/exgdas_atmos_verfrad.sh is not consistent with the request in bugzilla 1210. compath.py is used in the verification jobs to set the path to the current model, gfs. This is not the way in which compath.py is to be used.

The current NOAA-EMC/GSI develop no longer contains the DA verification package. DA verification has been moved to its own repository, NOAA-EMC/GSI-Monitor. So as to not loose track of this bugzilla and it's relationship to DA Monitoring, GSI-Monitor issue #32 has been opened. @EdwardSafford-NOAA is tagged for awareness.

Furthermore, DA jobs and scripts, including those for DA verification, are being moved out GSI based repositories and into global-workflow develop. This task has already been completed for DA jobs and scripts which formerly were in NOAA-EMC/GSI (see g-w PR #904). g-w PR #969 is currently open to move GSI-Monitor jobs and scripts into g-w develop.

As such, aligning DA verification jobs with bugzilla 1210 in a future implementation will need to be done via an issue and PR opened in NOAA-EMC/global-workflow.

Please see global-workflow issue https://github.com/NOAA-EMC/global-workflow/issues/289 for additional information on this bugzilla.

RussTreadon-NOAA commented 2 years ago

Bugzilla 1212

Resolution of this bugzilla involves more than just GFS DA. Resolution of the five items in this bugzilla requires a top down and bottom up sweep of all GFS jobs, scripts, and ush. Efforts are currently underway to bring jobs, scripts, and ush from GFS component repositories into the global-workflow repository. g-w PRs #904 and #969 are examples of this ongoing effort. Resolution of bugzilla 1212 will likely need to await completion of this reorganization and the opening of appropriate issue and PRs within g-w.

The above noted, release/gfsda.v16.3.0 was examined in light of bugzilla 1212.

1) Item 1 of bugzilla 1212 specifically mentions script exgdas_atmos_verfozn.sh. This script has been updated and no longer sets HOMEgdas_ozn, EXECoznmon, COM_IN, and COMIN. Only HOMEoznmon is referenced, not defined. The definition for HOMEoznmon is provided in JGDAS_ATMOS_VERFOZN. While this is an improvement, this leads to item 2 of bugzilla 1212.

2) Since all jobs running in the gfs suite belong to NET=gfs, variables defining paths to files in the gfs PACKAGEROOT should be consistent (e.g, HOMEgfs, EXECgfs, USHgfs, ...). This top-down consistency is not found in all jobs/, scripts/, and ush/ found in release/gfsda.v16.3.0. An example of this inconsistency is found in the use of $HOMEgfs in /util/Minimization_Monitor/nwprod/gdas/

russ.treadon@clogin04:/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/test/sorc/gsi.fd/util/Minimization_Monitor/nwprod/gdas> grep HOME jobs/JGDAS_ATMOS_VMINMON 
export HOMEgfs=${HOMEgfs:-${NWROOT}/gfs.${gfs_ver}}
export SCRgfs=${SCRgfs:-$HOMEgfs/scripts}
export M_FIXgdas=${M_FIXgdas:-$HOMEgfs/fix/gdas}
export HOMEminmon=${HOMEminmon:-${HOMEgfs}}
export EXECminmon=${EXECminmon:-$HOMEminmon/exec}
export USHminmon=${USHminmon:-$HOMEminmon/ush}
russ.treadon@clogin04:/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/test/sorc/gsi.fd/util/Minimization_Monitor/nwprod/gdas> grep minmon scripts/exgdas_atmos_vminmon.sh 
# Script name:         exgdas_vrfminmon.sh
export scr=exgdas_vrfyminmon.sh
export mm_gnormfile=${gnormfile:-${M_FIXgdas}/gdas_minmon_gnorm.txt}
export mm_costfile=${costfile:-${M_FIXgdas}/gdas_minmon_cost.txt}
export pgm=exgdas_vrfminmon.sh
   ${USHminmon}/minmon_xtrct_costs.pl ${MINMON_SUFFIX} ${PDY} ${cyc} ${gsistat} dummy
   ${USHminmon}/minmon_xtrct_gnorms.pl ${MINMON_SUFFIX} ${PDY} ${cyc} ${gsistat} dummy
   ${USHminmon}/minmon_xtrct_reduct.pl ${MINMON_SUFFIX} ${PDY} ${cyc} ${gsistat} dummy
   echo "end exgdas_vrfminmon.sh, exit value = ${err}"

Instead of setting and using *minmon variables derived from HOMEgfs, variable HOMEgfs could be directly used. Other examples can be found in release/gfsda.v16.3.0 which are not aligned with item 2 of bugzilla 1212.

3) Resolution of item 3 of bugzilla 1212 requires coordinated changes in g-w and component repositories in which jobs exist. As mentioned above efforts are underway to bring component jobs/, scripts/, and ush/ in the g-w repository. Once done, config files and jobs will reside in the same repository, thereby simplifying efforts to address this aspect of bugzilla 1212.

A check of g-w release/gfs.v16.3.0 and release/gfsda.v16.3.0 finds that config.base.nco.static only defines

config.base.nco.static:export COMINatmos=${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos
config.base.nco.static:export COMOUTatmos=${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos
config.base.nco.static:export COMINwave=${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave
config.base.nco.static:export COMOUTwave=${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave

config.wafs* refer to COMINatmos and COMOUTatmos. No files in release/gfsda.v16.3.0 reference either of these variables.

4) A check of release/gfs.v16.3.0 config files finds the following occurrences of ROTDIR in config.base.nco.static 

parm/config/config.base.nco.static:export PTMP=$ROTDIR
parm/config/config.base.nco.static:export ROTDIR="$(compath.py ${envir}/${NET}/${gfs_ver})"
parm/config/config.base.nco.static:export ROTDIR_DUMP="YES"
parm/config/config.base.nco.static:export COMINatmos=${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos
parm/config/config.base.nco.static:export COMOUTatmos=${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos
parm/config/config.base.nco.static:export COMINwave=${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave
parm/config/config.base.nco.static:export COMOUTwave=${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave

ROTDIR is used to define COMIN and COMOUT in jobs. For example, release/gfsda.v16.3.0 JGDAS_ATMOS_ANALYSIS_CALC contains

if [ $RUN_ENVIR = "nco" -o ${ROTDIR_DUMP:-NO} = "YES" ]; then
    export COMIN=${COMIN:-$ROTDIR/$RUN.$PDY/$cyc/$COMPONENT}
    export COMOUT=${COMOUT:-$ROTDIR/$RUN.$PDY/$cyc/$COMPONENT}

The appropriate default for COMIN and COMOUT could be provided in the j-job without resorting to the use of ROTDIR.

5) A check of scripts in release/gfsda.v16.3.0 indicates that references to the path for EnKF members include a COMIN or COMOUT variable reflecting the in/out nature of the given member in the given script. For example scripts/exgdas_enkf_ecen.sh contains

russ.treadon@clogin04:/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/test/sorc/gsi.fd/scripts> grep -r memchar exgdas_enkf_ecen.sh 
   memchar="mem"$(printf %03i $imem)
   $NLN $COMIN_GES_ENS/$memchar/${GPREFIX}atmf00${FHR}${ENKF_SUFFIX}$GSUFFIX ./atmges_$memchar
         $NLN $COMIN_ENS/$memchar/${APREFIX_ENKF}atmanl$ASUFFIX ./atmanl_$memchar
         $NLN $COMIN_ENS/$memchar/${APREFIX_ENKF}atma00${FHR}$ASUFFIX ./atmanl_$memchar
   mkdir -p $COMOUT_ENS/$memchar
      $NLN $COMOUT_ENS/$memchar/${APREFIX}atminc.nc ./atminc_$memchar
      $NLN $COMOUT_ENS/$memchar/${APREFIX}atmi00${FHR}.nc ./atminc_$memchar
            $NLN $COMOUT_ENS/$memchar/${APREFIX}ratmanl$ASUFFIX ./ratmanl_$memchar
            $NLN $COMOUT_ENS/$memchar/${APREFIX}ratma00${FHR}$ASUFFIX ./ratmanl_$memchar
            $NLN $COMOUT_ENS/$memchar/${APREFIX}ratminc$ASUFFIX ./ratminc_$memchar
            $NLN $COMOUT_ENS/$memchar/${APREFIX}ratmi00${FHR}$ASUFFIX ./ratminc_$memchar

The use of COMIN_GES_ENS, COMIN_ENS, and COMOUT_ENS could be simplified and standardized.

While GFS v16.3.0 represents a step in the right direction for bugzilla 1212, more work remains. While still daunting it may prove easier to tackle bugzilla 1212 once jobs, scripts, and ush for all GFS components reside in the g-w repository.

RussTreadon-NOAA commented 2 years ago

Bugzilla 1214

As mentioned above, GFS DA may be done with this bugzilla. release/gfsda.v16.3.0 continues the practice of extensively using dynamically allocated arrays in DA source code. The nature of the changing observational database from cycle to cycle necessitates this approach.

release/gfsda.v16.3.0 DA apps were built and run with -check all. No out of bounds array errors were detected. Cases of array size mismatch have been corrected. Correction of mismatched array sizes did not alter cycled GFS v16.3.0 results.

RussTreadon-NOAA commented 2 years ago

Bugzilla 1216

The issue noted in this bugzilla, erroneous missing file warnings, has been addressed via issue #123 and PR #144.

A check of jgdas_atmos_verfozn job log files from the EMC retrospective and real-time GFS v16.3.0 parallels running on WCOSS2 found that ozone satellite_sensor diagnostic files flagged as missing diag file are indeed missing.

For example, the 2022081900 run flagged the following ozone satellite_sensor diagnostic files as missing

russ.treadon@dlogin05:/lfs/h2/emc/ptmp/lin.gan/rt1-v16-ecf/para/com/gfs/v16.3/logs> grep "missing" 2022081900/gdas_atmos_verfozn_00.o16845341.log
+ 0 + echo 'missing diag file -- diag_gome_metop-c_ges.2022081900.gz not found '
+ 0 + echo 'missing diag file -- diag_ompsnp_npp_ges.2022081900.gz not found '
+ 0 + echo 'missing diag file -- diag_ompstc8_npp_ges.2022081900.gz not found '
+ 0 + echo 'missing diag file -- diag_ompslp_npp_ges.2022081900.gz not found '
+ 0 + echo 'missing diag file -- diag_ompsnp_n21_ges.2022081900.gz not found '
+ 0 + echo 'missing diag file -- diag_ompstc8_n21_ges.2022081900.gz not found '

A check of the oznstat tarball confirms this

russ.treadon@dlogin05:/lfs/h2/emc/ptmp/lin.gan/rt1-v16-ecf/para/com/gfs/v16.3/logs> tar -tvf /lfs/h2/emc/ptmp/lin.gan/rt1-v16-ecf/para/com/gfs/v16.3/gdas.20220819/00/atmos/gdas.t00z.oznstat
-rw-r--r-- lin.gan/emc   46514 2022-08-23 02:34 diag_sbuv2_n19_ges.2022081900.nc4.gz
-rw-r--r-- lin.gan/emc   69977 2022-08-23 02:34 diag_gome_metop-b_ges.2022081900.nc4.gz
-rw-r--r-- lin.gan/emc   36775 2022-08-23 02:34 diag_omi_aura_ges.2022081900.nc4.gz
-rw-r--r-- lin.gan/emc   77760 2022-08-23 02:35 diag_ompstc8_n20_ges.2022081900.nc4.gz
-rw-r--r-- lin.gan/emc  299705 2022-08-23 02:34 diag_ompsnp_n20_ges.2022081900.nc4.gz
-rw-r--r-- lin.gan/emc   46548 2022-08-23 02:34 diag_sbuv2_n19_anl.2022081900.nc4.gz
-rw-r--r-- lin.gan/emc   69932 2022-08-23 02:34 diag_gome_metop-b_anl.2022081900.nc4.gz
-rw-r--r-- lin.gan/emc   36797 2022-08-23 02:34 diag_omi_aura_anl.2022081900.nc4.gz
-rw-r--r-- lin.gan/emc   77906 2022-08-23 02:35 diag_ompstc8_n20_anl.2022081900.nc4.gz
-rw-r--r-- lin.gan/emc  300058 2022-08-23 02:34 diag_ompsnp_n20_anl.2022081900.nc4.gz

This bugzilla may be closed pending approval from NCO.

RussTreadon-NOAA commented 2 years ago

Bugzilla 1218

This bugzilla addresses the build for multiple GFS v16 applications. The initial DA response to this bugzilla found in comments earlier in this issue.

The build and installation of release/gfsda.v16.3.0 is invoked at the g-w level via sorc/build_all.sh. This script executes sorc/build_gsi.sh to build and install DA apps. Script sorc/build_gsi.sh, in turn, invokes sorc/gsi.fd/ush/build_4nco_global.sh.

release/gfsda.v16.3.0 uses CMake for build and installation of operational DA apps. The installation performs a clean by default which removes the build and install directories. Only the exec directory is retained.

debug is a build option. To build and install debug apps, the BUILD_TYPE line in sorc/gsi.fd/ush/build_4nco_global.sh is changed from

# Set NCO explicit variables                                                                                                                                    
export BUILD_TYPE="Release"

to 

# Set NCO explicit variables                                                                                                                                    
export BUILD_TYPE="Debug"

As delivered script build_4nco_global.sh is configured to build optimized apps for operation (ie, the Release build).

Bugzilla 1218 refers to Makefile and makefile in sorc/gsi.fd. Most of these references are removed in release/gfsda.v16.3.0 due to the extensive use of CMake. Listed below are the *akefile which remain in a fresh clone of release/gfsda.v16.3.0:

russ.treadon@clogin04:/lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/fork> find . -name "*akefile" -print 
./src/GSD/gsdcloud4nmmb/makefile
./util/NMC_Bkerror/sorc_aero/Makefile
./util/NMC_Bkerror/sorc/Makefile
./util/Radiance_Utilities/Fortran/RadDiag_Stats/Makefile
./util/Radiance_Utilities/diag2grads/Makefile
./util/Radiance_Utilities/satinfo_appnd/Makefile
./util/GEN_BE_V2.0/src/Makefile
./util/GEN_BE_V2.0/Makefile
./util/GEN_BE_V2.0/external/ioapi_share/makefile
./util/GEN_BE_V2.0/external/tools/Makefile
./util/GEN_BE_V2.0/external/blas/Makefile
./util/GEN_BE_V2.0/external/fftpack/fftpack5/Makefile
./util/GEN_BE_V2.0/external/io_netcdf/makefile
./util/GEN_BE_V2.0/external/Makefile
./util/GEN_BE_V2.0/external/lapack/Makefile
./util/Radiance_bias_correction_Utilities/makefile
./util/Config/GNUmakefile
./util/NCEPgsi_Coupler/GNUmakefile
./util/NCEPgsi_Coupler/Makefile
./util/MODIS_AOD/Makefile
./util/FOV_utilities/sorc/makefile
./util/global_angupdate/Makefile
./util/Misc/Makefile

When the export PRUNE_4NCO="YES" line is activated in sorc/build_gsi.sh, the above directories are removed during the build and installation so that a check of sorc/gsi.fd following execution of build_gsi.sh returns

russ.treadon@clogin04:/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/test/sorc> find gsi.fd -name "*akefile" -print                                   
russ.treadon@clogin04:/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/test/sorc>

Since bugzilla 1218 addresses more than just GFS DA components, the above changes are not sufficient to close this bugzilla. The above, however, demonstrates, that the DA component of GFS v16.3.0 is, hopefully, in alignment with the intent of bugzilla 1218. Confirmation of this must be provided by NCO.

RussTreadon-NOAA commented 2 years ago

Bugzilla 1221

As described in comments above, issue #123 addressed the DA aspect of this bugzilla.

A check of the offending DA scripts, exgdas_atmos_verf*, in release/gfsda.v16.3.0 confirms that these scripts no longer contain hard coded paths to NDATE. Each script simply refers to $NDATE with the definition provided by the prod_util module.

A check of a pruned (PRUNE_4NCO=YES) installation of release/gfsda.v16.3.0 found the following occurrences of hard coded /nwprod paths.

./util/Ozone_Monitor/data_xtrct/ush/run_gfs_gdas.sh:export NDATE=/gpfs/dell1/nco/ops/nwprod/prod_util.v1.1.1/exec/ips/ndate
./util/Ozone_Monitor/build_OznMon_cmake.sh:#  nwprod/oznmon_shared/exec and data_xtrct/exec subdirectories.
./util/Ozone_Monitor/build_OznMon_cmake.sh:#  The operational OznMon executables (in nwprod/oznmon_shared/exec) 
./util/Radiance_Monitor/build_RadMon_cmake.sh:#  nwprod/radmon_shared/exec, data_extract/exec, and image_gen/exec
./util/Radiance_Monitor/build_RadMon_cmake.sh:#  The operational RadMon executables (in nwprod/radmon_shared/exec) 
./util/Conventional_Monitor/build_ConMon_cmake.sh:#  nwprod/conmon_shared/exec, data_extract/exec, and image_gen/exec
./util/Conventional_Monitor/build_ConMon_cmake.sh:#  The operational ConMon executables (in nwprod/conmon_shared/exec) 
./util/Conventional_Monitor/image_gen/ush/grib2ctl.pl:$wgrib=(system("wgrib --v > /dev/null 2>&1") == 0) ? 'wgrib' : '/gpfs/dell1/n
co/ops/nwprod/grib_util.v1.1.1/exec/wgrib' ;  # NCEP only
./util/Conventional_Monitor/nwprod/gdas_conmon/scripts/exgdas_atmos_conmon.sh:   export NDATE=${NDATE:-/nwprod/util/exec/ndate}

The Conventional_Monitor package is not operational. GFS v16.3.0 does not add this package to operations. As such, add this directory to the list of directories ush/prune_4nco.sh removes. Done at 0b5ea6d in fork branch Russ.Treadon-NOAA/GSI/release/gfsda.v16.3.0.

With this change only the Ozone_Monitor and Radiance_Monitor occurrences of hard coded /nwprod paths remain. The two occurrences in Radiance_Monitor are in comments. No action is necessary. Two of the three occurrences in Ozone_Monitor are in comments. The remaining, a reference to NDATE may be removed. Script run_gfs_gdas.sh loads prod_util. This module defines NDATE. This change was committed to the forked release/gfsda.v16.3.0 at c6c33da.

A check for exec/ found the following hardwired defaults

./scripts/exgdas_enkf_ecen.sh:NEMSIOGET=${NEMSIOGET:-${NWPROD}/exec/nemsio_get}
./scripts/exglobal_atmos_analysis_calc.sh:export NEMSIOGET=${NEMSIOGET:-${NWPROD}/exec/nemsio_get}
./scripts/exgdas_enkf_update.sh:NEMSIOGET=${NEMSIOGET:-$NWPROD/utils/exec/nemsio_get}
./scripts/exgdas_enkf_sfc.sh:NEMSIOGET=${NEMSIOGET:-${NWPROD}/exec/nemsio_get}
./scripts/exglobal_atmos_analysis.sh:export NEMSIOGET=${NEMSIOGET:-${NWPROD}/exec/nemsio_get}
./scripts/exglobal_diag.sh:export NEMSIOGET=${NEMSIOGET:-${NWPROD}/exec/nemsio_get}
./util/Minimization_Monitor/parm/MinMon_config:   export NDATE=${NDATE:-${NWPROD}/util/exec/ndate}

The NDATE line in MinMon_config may be removed since logic in the file loads prod_util depending on the machine on which the parm is executed. This line was removed from the forked release/gfsda.v16.3.0 at 3b9c14a.

Variable NEMSIOGET is defined in config.base.nco.static via the line

export NEMSIOGET="$HOMEgfs/exec/nemsio_get"

As such, NEMSIOGET=${NEMSIOGET:-$NWPROD/utils/exec/nemsio_get} is not needed in the ex* scripts. This line was removed from the indicated ex* scripts in the forked release/gfsda.v16.3.0. This was done at 3b9c14a.

With these changes release/gfsda.v16.3.0 moves closer to the intent of bugzilla 1221. Vigilance is required to that ensure new hardwired paths do not creep into DA jobs, parm, scripts, and ush.

RussTreadon-NOAA commented 2 years ago

Bugzilla 1222

Comments pertaining to this bugzilla are found above in this issue.

release/gfsda.v16.3.0 has been examined with respect to the issues identified in bugzilla 1222. Neither strings MOD_PATH nor USE_PREINST_LIBS occur in release/gfsda.v16.3.0.
No occurrences of COREINC or CORELIB were found in the pruned directory tree.

A check of the pruned directory tree found the Hera path /NCEPDEV is several files. However, these occurrences are limited to

No action will be taken at present with regards to the above instances of NCEPDEV.

One item of note loosely associated with bugzilla 1222.
File modulefiles/gsi_wcoss2.lua contains

unload("crtm")

pushenv("HPC_OPT", "/apps/ops/para/libs")
prepend_path("MODULEPATH", "/apps/ops/para/libs/modulefiles/compiler/intel/19.1.3.304")
prepend_path("MODULEPATH", "/apps/ops/para/libs/modulefiles/mpi/intel/19.1.3.304/cray-mpich/8.1.7/")
load("crtm/2.4.0")

This section should be removed because RFC 9881 installed crtm/2.4.0 in a production location on August 22 at 1600Z. modulefiles/gsi_wcoss2.lua has been updated accordingly in the forked release/gfsda.v16.3.0. This was done at 51edc24

Execution of g-w sorc/build_gsi.sh failed, not due to crtm/2.4.0, but rather due to the removal of util/Conventional_Monitor in the pruned directory tree. The BUILD_UTIL_MON section of util/CMakeLists.txt assumes Conventional_Monitor is built along with the Ozone and Radiance monitoring packages. This is not true for current GFS operations or GFS v16.3.0. Given this, the Conventional_Monitor line in util/CMakeLists.txt was commented out. This change was committed to the forked release/gfsda.v16.3.0 at 51edc24

The changes documented above move this bugzilla closer to closure with respect to GFS DA v16.3.0 components. These changes alone do not warrant closing bugzilla 1222. This bugzilla touches more than just the GFS DA component.

RussTreadon-NOAA commented 2 years ago

PR #456 was merged into the authoritative release/gfsda.v16.3.0 at 733629. As such, this issue may be closed. Closing this issue, however, does NOT close the NCO bugzillas. EIB, the GFS v16.3.0 DA POC, and the NCO SPA working on GFS v16.3.0 must review the updated bugzillas and decide if the documented changes are sufficient for NCO to close the bugzillas.

As noted in the description of this issue, most of the 12 bugzillas listed in this issue are NOT DA-only bugzillas. The majority of the bugzillas refer to the GFS of which DA is one component. An attempt has been made to address the DA component in the GFS bugzillas. 4 of the 12 bugzillas are limited to DA alone. Changes now in the authoritative release/gfsda.v16.3.0 are hopefully sufficient for NCO to close these DA-only bugzillas.

The table below separates the 12 bugzillas mentioned in this issue into GFS and DA-only bugzillas

GFS buzillas DA-only bugzillas
216: Need to remove all "GO TO" statement in GFS source codes 1196: Improve error message in gfs_atmos_analysis job for missing enkf input
1198: Improve error handling to use err_chk/err_exit in gfs scripts 1205: exec/global_gsi.x failed in job run with "check all" compilation flag
1210: COM variables in gfs scripts 1206: exec/global_enkf.x failed in job run with "check all" compilation flag
1212: Improve Standard environment variables definition in gfs package 1216: Clear the error messages in gdas_atmos_verfozn job
1214: Review gfs source code for array allocation
1218: Add debug and clean target to gfs makefiles
1221: Remove hardcoded path in gfs scripts
1222: Remove hardcoded path in build script and makefiles in gfs source code packages
RussTreadon-NOAA commented 2 years ago

Closing this issue given the merger of PR #456 into release/gfsda.v16.3.0