search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
70 stars 161 forks source link

Update ufs model hash to 20240625 #2729

Open JessicaMeixner-NOAA opened 3 days ago

JessicaMeixner-NOAA commented 3 days ago

Description

Updates UFS weather model hash to hash from 2024-06-24 which has orion porting updates + a few namelist updates.

Type of change

Change characteristics

How has this been tested?

Tested with other chnages on hera (only a half test as things failed), currently have a C48mx500 S2SW forecast running on wcoss2. Needs full CI testing.

Checklist

emcbot commented 3 days ago 

CI Update on Wcoss2 at 06/27/24 07:09:12 PM
============================================
Cloning and Building global-workflow PR: 2729
with PID: 232730 on host: dlogin08
emcbot commented 3 days ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Thu Jun 27 19:13:23 UTC 2024 on dlogin08
---------------------------------------------------
Build: *** FAILED ***
Build: Failed at 06/27/24 07:22:47 PM
Creating logs folder
Creating /lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2729/global-workflow/exec folder
Resetting modules to system default. Reseting $MODULEPATH back to system default. All extra directories will be removed from $MODULEPATH.
Building gsi_enkf, ufs, gfs_utils, gdas, ww3prepost, ufs_utils, gsi_utils, gsi_monitor, upp
Starting build_gsi_enkf.sh
Starting build_ufs.sh
Starting build_gfs_utils.sh
Starting build_gdas.sh
Starting build_ww3prepost.sh
Starting build_ufs_utils.sh
Starting build_gsi_utils.sh
build_gdas.sh failed!  Exiting!
Check logs/build_gdas.log for details.
logs/build_gdas.log
Failed on cloning and building global-workflowi PR: 2729
CI on Wcoss2 failed to build on Thu Jun 27 19:22:47 UTC 2024 for repo git@github.com:NOAA-EMC/global-workflow.git
JessicaMeixner-NOAA commented 3 days ago

Not sure why this PR would break a gdas build as there was no chnage.

RussTreadon-NOAA commented 3 days ago

The gdas.cd hash associated with g-w develop is obsolete. GDASApp PR #1202 changed model_data to model as per NCO requirements (see g-w issue #2686). We used this opportunity to clean up the directory structure for GDASApp CI.

The GDASApp build in this PR failed because it is looking for gdasapp-fix-bda76f.tgz using a path which no longer exists.

emcbot commented 3 days ago

Checkout Failed on Hera: Could not perform submodule update

emcbot commented 3 days ago

Checkout Failed on Hera: Could not perform submodule update

RussTreadon-NOAA commented 3 days ago

As a temporary fix I added links on Dogwood to mimic the old directory structure shown below

setenv("CRTM_FIX","/lfs/h2/emc/da/noscrub/emc.da/GDASApp/fix/crtm/2.4.0")
setenv("GDASAPP_TESTDATA","/lfs/h2/emc/da/noscrub/emc.da/GDASApp/data")
setenv("GDASAPP_UNIT_TEST_DATA_PATH", "/lfs/h2/emc/da/noscrub/emc.da/GDASApp/data/test")
emcbot commented 3 days ago

Build FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2729/gfs/sorc/logs/build_gdas.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 3 days ago 

CI Update on Wcoss2 at 06/27/24 07:42:28 PM
============================================
Cloning and Building global-workflow PR: 2729
with PID: 197002 on host: dlogin08
RussTreadon-NOAA commented 3 days ago

The Hera failure is strange. Directory /scratch1/NCEPDEV/global/CI/2729/gfs/sorc/gdas.cd/sorc/vader only contains .git.

JessicaMeixner-NOAA:updateufs20240625 points at gdas.cd @ 368c9c5. This GDASApp has points at vader @ 8210ecc. This snapshot of vader contains more than just .git

aerorahul commented 3 days ago

The Hera failure is strange. Directory /scratch1/NCEPDEV/global/CI/2729/gfs/sorc/gdas.cd/sorc/vader only contains .git.

JessicaMeixner-NOAA:updateufs20240625 points at gdas.cd @ 368c9c5. This GDASApp hash points at vader @ 8210ecc. This snapshot of vader contains more than just .git

sounds like a full disk and git failed

JessicaMeixner-NOAA commented 3 days ago

Global disk space seems okay:

Project: global Directory: /scratch1/NCEPDEV/global DiskInUse=328906 GB, Quota=387000 GB, Files=42804324, FileQUota=77400000

RussTreadon-NOAA commented 3 days ago

Does automated CI have logs which can confirm that the /scratch1/NCEPDEV/global fileset is full?

I ask because I am currently in /scratch1/NCEPDEV/global/Russ.Treadon/git/global-workflow/develop. My git clone of g-w develop failed not due to a disk quota issue. Instead I see

/scratch1/NCEPDEV/global/Russ.Treadon/git/global-workflow/develop$ git clone --recursive https://github.com/NOAA-EMC/global-workflow.git .
Cloning into '.'...
remote: Enumerating objects: 61037, done.
remote: Counting objects: 100% (726/726), done.
remote: Compressing objects: 100% (440/440), done.
^Cfetch-pack: unexpected disconnect while reading sideband packet

Hera(hfe06):/scratch1/NCEPDEV/global/Russ.Treadon/git/global-workflow/develop$
emcbot commented 3 days ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Thu Jun 27 19:46:50 UTC 2024 on dlogin08
---------------------------------------------------
Build: Completed at 06/27/24 08:25:04 PM
Case setup: Completed for experiment C48_ATM_68dc5c74
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_68dc5c74
Case setup: Skipped for experiment C48_S2SWA_gefs_68dc5c74
Case setup: Completed for experiment C48_S2SW_68dc5c74
Case setup: Completed for experiment C96_atm3DVar_extended_68dc5c74
Case setup: Skipped for experiment C96_atm3DVar_68dc5c74
Case setup: Completed for experiment C96_atmaerosnowDA_68dc5c74
Case setup: Completed for experiment C96C48_hybatmDA_68dc5c74
Case setup: Completed for experiment C96C48_ufs_hybatmDA_68dc5c74
emcbot commented 3 days ago

Experiment C48_ATM_68dc5c74 SUCCESS on Wcoss2 at 06/27/24 09:36:12 PM

emcbot commented 3 days ago

Experiment C48_S2SW_68dc5c74 SUCCESS on Wcoss2 at 06/27/24 09:48:19 PM

emcbot commented 3 days ago

Build FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2729/gfs/sorc/logs/build_gdas.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 3 days ago

Experiment C96_atmaerosnowDA_68dc5c74 FAIL on Wcoss2 at 06/27/24 11:12:21 PM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2729/RUNTESTS/COMROOT/C96_atmaerosnowDA_68dc5c74/logs/2021122100/gdasaeroanlrun.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2729/RUNTESTS/COMROOT/C96_atmaerosnowDA_68dc5c74/logs/2021122100/gfsaeroanlrun.log

Follow link here to view the contents of the above file(s): (link)

WalterKolczynski-NOAA commented 3 days ago 
nid001072.dogwood.wcoss2.ncep.noaa.gov 0: Configuration input file is: /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96_atmaerosnowDA_68dc5c74/gfsaeroanl_00/gfs.t00z.aerovar.yaml
Full configuration is:YAMLConfiguration[path=/lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96_atmaerosnowDA_68dc5c74/gfsaeroanl_00/gfs.t00z.aerovar.yaml, root={cost function => {cost type => 3D-FGAT , time window => {begin => 2021-12-20T21:00:00Z , length => PT6H} , analysis variables => (mass_fraction_of_sulfate_in_air,mass_fraction_of_hydrophobic_black_carbon_in_air,mass_fraction_of_hydrophilic_black_carbon_in_air,mass_fraction_of_hydrophobic_organic_carbon_in_air,mass_fraction_of_hydrophilic_organic_carbon_in_air,mass_fraction_of_dust001_in_air,mass_fraction_of_dust002_in_air,mass_fraction_of_dust003_in_air,mass_fraction_of_dust004_in_air,mass_fraction_of_dust005_in_air,mass_fraction_of_sea_salt001_in_air,mass_fraction_of_sea_salt002_in_air,mass_fraction_of_sea_salt003_in_air,mass_fraction_of_sea_salt004_in_air) , geometry => {fms initialization => {namelist filename => ./fv3jedi/fmsmpp.nml , field table filename => ./fv3jedi/field_table} , akbk => ./fv3jedi/akbk.nc4 , layout => (8,8) , npx => 97 , npy => 97 , npz => 127 , field metadata override => ./fv3jedi/fv3jedi_fieldmetadata_restart.yaml} , model => {name => PSEUDO , datapath => ./bkg , filetype => fms restart , filename is datetime templated => true , tstep => PT3H , filename_core => %yyyy%mm%dd.%hh%MM%ss.fv_core.res.nc , filename_trcr => %yyyy%mm%dd.%hh%MM%ss.fv_tracer.res.nc , filename_cplr => %yyyy%mm%dd.%hh%MM%ss.coupler.res} , background => {datapath => ./bkg , filetype => fms restart , datetime => 2021-12-20T21:00:00Z , filename_core => 20211220.210000.fv_core.res.nc , filename_trcr => 20211220.210000.fv_tracer.res.nc , filename_cplr => 20211220.210000.coupler.res , state variables => (t,delp,sphum,so4,bc1,bc2,oc1,oc2,dust1,dust2,dust3,dust4,dust5,seas1,seas2,seas3,seas4)} , background error => {covariance model => FV3JEDI-ID , date => 2021-12-20T21:00:00Z} , observations => {observers => ({obs space => {name => viirsnid001072.dogwood.wcoss2.ncep.noaa.gov 0: _n20_aod , obsdatain => {engine => {type => H5File , obsfile => /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96_atmaerosnowDA_68dc5c74/gfsaeroanl_00/obs/gfs.t00z.viirs_n20.2021122100.nc4}} , obsdataout => {engine => {type => H5File , obsfile => /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96_atmaerosnowDA_68dc5c74/gfsaeroanl_00/diags/diag_viirs_n20_2021122100.nc4}} , io pool => {max pool size => 1} , simulated variables => (aerosolOpticalDepth) , channels => 4} , get values => {interpolation method => barycentric , time interpolation => linear} , obs operator => {name => AodCRTM , Absorbers => (H2O,O3) , obs options => {Sensor_ID => v.viirs-m_j1 , EndianType => little_endian , CoefficientPath => /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96_atmaerosnowDA_68dc5c74/gfsaeroanl_00/crtm/ , AerosolOption => aerosols_gocart_default}} , obs error => {covariance model => diagonal} , obs filters => ({filter => PreQC , maxvalue => 1},{filter => Domain Check , where => ({variable => {name => latitude@MetaData} , minvalue => -60 , maxvalue => 60})},{filter => Bounds Check , filter variables => ({name => aerosolOpticalDepth , channels => 4}) , minvalue => 0 , maxvalue => 4.9 , action => {name => reject}},{filter => Background Check , channels => 4 , threshold => 3 , action => {name => inflate error , inflation factor => 3}})},{obs space => {name => viirs_npp_aod , obsdatain => {engine => {type => H5File , obsfile => /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96_atmaerosnowDA_68dc5c74/gfsaeroanl_00/obs/gfs.t00z.viirs_npp.2021122100.nc4}} , obsdataout => {engine => {type => H5File , obsfile => /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96_atmaerosnowDA_68dc5c74/gfsaeroanl_00/diags/diag_viirs_npp_2021122100.nc4}} , io pool => {max pool size => 1} , simulated variables => (aerosolOpticalDepth) , channels => 4} , get values => {interpolation method => barycentric , time interpolation => linear} , obs operator => {name => AodCRTM , Absorbers => (H2O,O3) , obs options => {Sensor_ID => v.viirs-m_npp , EndianType => little_endian , Coeffinid001072.dogwood.wcoss2.ncep.noaa.gov 0: cientPath => /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96_atmaerosnowDA_68dc5c74/gfsaeroanl_00/crtm/ , AerosolOption => aerosols_gocart_default}} , obs error => {covariance model => diagonal} , obs filters => ({filter => PreQC , maxvalue => 1},{filter => Domain Check , where => ({variable => {name => latitude@MetaData} , minvalue => -60 , maxvalue => 60})},{filter => Bounds Check , filter variables => ({name => aerosolOpticalDepth , channels => 4}) , minvalue => 0 , maxvalue => 4.9 , action => {name => reject}},{filter => Background Check , channels => 4 , threshold => 3 , action => {name => inflate error , inflation factor => 3}})})}} , variational => {minimizer => {algorithm => DRPCG} , iterations => ({ninner => 35 , gradient norm reduction => 1e-10 , test => true , geometry => {fms initialization => {namelist filename => ./fv3jedi/fmsmpp.nml , field table filename => ./fv3jedi/field_table} , akbk => ./fv3jedi/akbk.nc4 , layout => (8,8) , npx => 97 , npy => 97 , npz => 127 , field metadata override => ./fv3jedi/fv3jedi_fieldmetadata_restart.yaml} , diagnostics => {departures => bkgmob}},{ninner => 35 , gradient norm reduction => 1e-10 , test => true , geometry => {fms initialization => {namelist filename => ./fv3jedi/fmsmpp.nml , field table filename => ./fv3jedi/field_table} , akbk => ./fv3jedi/akbk.nc4 , layout => (8,8) , npx => 97 , npy => 97 , npz => 127 , field metadata override => ./fv3jedi/fv3jedi_fieldmetadata_restart.yaml} , diagnostics => {departures => bkgmob1}})} , final => {diagnostics => {departures => anlmob} , increment => {geometry => {fms initialization => {namelist filename => ./fv3jedi/fmsmpp.nml , field table filename => ./fv3jedi/field_table} , akbk => ./fv3jedi/akbk.nc4 , layout => (8,8) , npx => 97 , npy => 97 , npz => 127 , field metadata override => ./fv3jedi/fv3jedi_fieldmetadata_restart.yaml} , output => {state component => {datapath => ./anl , prefix => aeroinc , filetype => fms restart , filename_core => 20211221.000000.fv_core.res.nc , filename_trcr => 20211221.000000.fv_tracer.nid001072.dogwood.wcoss2.ncep.noaa.gov 0: res.nc , filename_cplr => 20211221.000000.coupler.res , state variables => (t,delp,sphum,so4,bc1,bc2,oc1,oc2,dust1,dust2,dust3,dust4,dust5,seas1,seas2,seas3,seas4)}}}}}]
nid001072.dogwood.wcoss2.ncep.noaa.gov 0: OOPS_STATS ObjectCountHelper started.
nid001072.dogwood.wcoss2.ncep.noaa.gov 0: OOPS_STATS Run start                                - Runtime:      5.11 sec,  Memory: total:    76.27 Gb, per task: min =   115.64 Mb, max =   243.01 Mb
Run: Starting oops::Variational<FV3JEDI, UFO and IODA observations>
nid001072.dogwood.wcoss2.ncep.noaa.gov 0: OOPS_STATS Variational start                        - Runtime:     5.11 sec,  Local Memory:   126.60 Mb
nid001072.dogwood.wcoss2.ncep.noaa.gov 0: 
FATAL from PE     0: get_ascii_file_num_lines: File ./fv3jedi/fmsmpp.nml does not exist.

nid001072.dogwood.wcoss2.ncep.noaa.gov 0: 
FATAL from PE     0: get_ascii_file_num_lines: File ./fv3jedi/fmsmpp.nml does not exist.