search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
75 stars 168 forks source link

Support GFSv16.2.x (WCOSS2 port) on supported tier-1 platforms #665

Closed KateFriedman-NOAA closed 2 years ago

KateFriedman-NOAA commented 2 years ago

Description

This issue documents work to add support for the new GFSv16.2.x (WCOSS2 port version) on supported R&D platforms (Hera & Orion). Additional updates will also be committed to wrap up remaining WCOSS2 dev updates not required for operations (e.g. obsproc, GDA, verification).

Changes will be committed to the dev_v16.2.x branch.

Requirements

The dev_v16.2.x branch should support developers running the GFSv16.2.x package on WCOSS2, Hera, and Orion, in both forecast-only and cycled mode.

Acceptance Criteria (Definition of Done)

A user can install the dev_v16.2.x branch on WCOSS2, Hera, and/or Orion (out-of-the-box) and run either forecast-only or cycled parallels to support development for future GFS versions.

Dependencies

The following related issues:

KateFriedman-NOAA commented 2 years ago

Testing on WCOSS2 in support of developers looking to run GFSv16.3 tests in coming months

Test: Cycled test with obsproc package integrated into dev_v16 branch

Successfully completed unattended cycled test on Dogwood (no verification pieces yet): 2022032718 - 2022032912

Kate.Friedman@dlogin03:/lfs/h2/emc/global/noscrub/Kate.Friedman/expdir/devv16a> rocotostat -d devv16a.db -w devv16a.xml -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202203271800        Done    Mar 28 2022 18:03:08    Mar 29 2022 14:05:15
202203280000        Done    Mar 28 2022 18:03:08    Mar 29 2022 13:55:14
202203280600        Done    Mar 28 2022 18:03:08    Mar 29 2022 13:55:14
202203281200        Done    Mar 29 2022 13:13:14    Mar 29 2022 18:35:13
202203281800        Done    Mar 29 2022 14:00:19    Mar 29 2022 19:50:13
202203290000        Done    Mar 29 2022 14:00:19    Mar 29 2022 21:15:11
202203290600        Done    Mar 29 2022 14:10:12    Mar 29 2022 23:05:08
202203291200        Done    Mar 29 2022 18:40:17    Mar 30 2022 00:55:12

Used WCOSS1 ops GDA data pushed over to Dogwood from WCOSS-Dell.

KateFriedman-NOAA commented 2 years ago

Test: Reproduce WCOSS2 ops output using dev_v16 branch

Set up new devv16ops test on Dogwood and set test GDA to point to obsproc output in ops instead of using WCOSS1 GDA data.

Prep job output

Successfully reproduced obsproc output in gdasprep and gfsprep jobs in test:

Kate.Friedman@dlogin04:> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/gdas.20220330/00/atmos
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/obsproc/v1.0/gdas.20220330/00/atmos/gdas.t00z.prepbufr gdas.t00z.prepbufr
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/obsproc/v1.0/gdas.20220330/00/atmos/gdas.t00z.nsstbufr gdas.t00z.nsstbufr
Kate.Friedman@dlogin04:>

Also compared produced gdas.t00z.atmg*.nc files between WCOSS2 ops and test...they were successfully reproduced. Not showing lengthy netcdf file comparison output.

Kate.Friedman@dlogin04:> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/gfs.20220330/00/atmos
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/obsproc/v1.0/gfs.20220330/00/atmos/gfs.t00z.prepbufr gfs.t00z.prepbufr
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/obsproc/v1.0/gfs.20220330/00/atmos/gfs.t00z.nsstbufr gfs.t00z.nsstbufr
Kate.Friedman@dlogin04:>

Stepping through next gdas, enkf, gfs suite jobs now...

KateFriedman-NOAA commented 2 years ago

Test: Reproduce WCOSS2 ops output using dev_v16 branch

gdas[gfs]waveinit & gdas[gfs]waveprep output (using WCOSS2 operational rtofs output)

Kate.Friedman@dlogin04:/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops
Kate.Friedman@dlogin04:/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops> ll | grep rtofs
lrwxrwxrwx 1 Kate.Friedman emc   46 Mar 30 17:04 rtofs.20220329 -> /lfs/h1/ops/prod/com/rtofs/v2.2/rtofs.20220329

Note: since I am running a 00z cycle I need to only point to and use the prior day rtofs.20220329 data because the rtofs.20220330 data would not naturally be available until the 12z cycle

gdaswaveinit output matches:

Kate.Friedman@dlogin04:> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/gdas.20220330/00/wave/rundata
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.aoc_9km gdaswave.mod_def.aoc_9km
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.at_10m gdaswave.mod_def.at_10m
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.ep_10m gdaswave.mod_def.ep_10m
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.glix_10m gdaswave.mod_def.glix_10m
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.glo_15mxt gdaswave.mod_def.glo_15mxt
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.glo_30m gdaswave.mod_def.glo_30m
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.glox_10m gdaswave.mod_def.glox_10m
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.gnh_10m gdaswave.mod_def.gnh_10m
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.gsh_15m gdaswave.mod_def.gsh_15m
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.points gdaswave.mod_def.points
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.mod_def.wc_10m gdaswave.mod_def.wc_10m
Kate.Friedman@dlogin04:>

gdaswaveprep output matches:

Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/gdaswave.glix_10m.t00z.cur gdaswave.glix_10m.t00z.cur
Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wave/rundata/rmp_src_to_dst_conserv_003_001.nc rmp_src_to_dst_conserv_003_001.nc
dst_grid_frac min/max 1=0.0,1.0637187956031038 min/max 2=0.0,1.0637187956031038 max abs diff=0.0000000000
src_address min/max 1=371521,398880 min/max 2=371521,398880 max abs diff=0.0000000000
dst_address min/max 1=1,62640 min/max 2=1,62640 max abs diff=0.0000000000
remap_matrix min/max 1=-0.0008185121195211639,0.9999999559298598 min/max 2=-0.0008185121195211639,0.9999999559298598 max abs diff=0.0000000000
Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/wav
e/rundata/rmp_src_to_dst_conserv_002_001.nc rmp_src_to_dst_conserv_002_001.nc
dst_grid_frac min/max 1=-0.003972484685684787,1.0000328573419641 min/max 2=-0.003972484685684787,1.0000328573419641 max abs diff=0.0000000000
src_address min/max 1=1,1012036 min/max 2=1,1012036 max abs diff=0.0000000000
dst_address min/max 1=652590,876960 min/max 2=652590,876960 max abs diff=0.0000000000
remap_matrix min/max 1=-49.271164460122655,84.04885717741999 min/max 2=-49.271164460122655,84.04885717741999 max abs diff=0.0000000000
Kate.Friedman@dlogin04:>

gfswaveinit and gfswaveprep outputs also match WCOSS2 ops, not showing comparison output.

KateFriedman-NOAA commented 2 years ago

Test: Reproduce WCOSS2 ops output using dev_v16 branch

gdaseobs output

The output from the gdaseobs job matches WCOSS2 ops output:

Kate.Friedman@dlogin04:> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/enkfgdas.20220330/00/atmos
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/enkfgdas.20220330/00/atmos/gdas.t00z.abias_int.ensmean gdas.t00z.abias_int.ensmean
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/enkfgdas.20220330/00/atmos/gdas.t00z.abias_pc.ensmean gdas.t00z.abias_pc.ensmean
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/enkfgdas.20220330/00/atmos/gdas.t00z.abias.ensmean gdas.t00z.abias.ensmean
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/enkfgdas.20220330/00/atmos/gdas.t00z.abias_air.ensmean gdas.t00z.abias_air.ensmean
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/enkfgdas.20220330/00/atmos/gdas.t00z.gsistat.ensmean gdas.t00z.gsistat.ensmean

Spot check of contents of gdas.t00z.obsinput.ensmean tarball also shows it matches ops.

KateFriedman-NOAA commented 2 years ago

Test: Reproduce WCOSS2 ops output using dev_v16 branch

gdasanal / gdasanalcalc / gdasanaldiag output

Output from the gdasanal[calc][diag] jobs matches WCOSS2 ops output:

Kate.Friedman@dlogin04:> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/gdas.20220330/00/atmos

Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.abias_int gdas.t00z.abias_int

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.atmi003.nc gdas.t00z.atmi003.nc
lon min/max 1=0.0,359.76562 min/max 2=0.0,359.76562 max abs diff=0.0000000000
lat min/max 1=-89.82071,89.82071 min/max 2=-89.82071,89.82071 max abs diff=0.0000000000
lev min/max 1=1.0,127.0 min/max 2=1.0,127.0 max abs diff=0.0000000000
pfull min/max 1=1.0,127.0 min/max 2=1.0,127.0 max abs diff=0.0000000000
ilev min/max 1=1.0,128.0 min/max 2=1.0,128.0 max abs diff=0.0000000000
hyai min/max 1=1.0,128.0 min/max 2=1.0,128.0 max abs diff=0.0000000000
hybi min/max 1=1.0,128.0 min/max 2=1.0,128.0 max abs diff=0.0000000000
u_inc min/max 1=-26.466715,19.218096 min/max 2=-26.466715,19.218096 max abs diff=0.0000000000
v_inc min/max 1=-22.81946,36.07371 min/max 2=-22.81946,36.07371 max abs diff=0.0000000000
delp_inc min/max 1=-7.192466,4.212751 min/max 2=-7.192466,4.212751 max abs diff=0.0000000000
delz_inc min/max 1=-148.3502,237.11136 min/max 2=-148.3502,237.11136 max abs diff=0.0000000000
T_inc min/max 1=-17.768827,11.117179 min/max 2=-17.768827,11.117179 max abs diff=0.0000000000
sphum_inc min/max 1=-0.008809673,0.00636767 min/max 2=-0.008809673,0.00636767 max abs diff=0.0000000000
liq_wat_inc min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
o3mr_inc min/max 1=-2.4430403e-06,2.5684667e-06 min/max 2=-2.4430403e-06,2.5684667e-06 max abs diff=0.0000000000
icmr_inc min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.atminc.nc gdas.t00z.atminc.nc
lon min/max 1=0.0,359.76562 min/max 2=0.0,359.76562 max abs diff=0.0000000000
lat min/max 1=-89.82071,89.82071 min/max 2=-89.82071,89.82071 max abs diff=0.0000000000
lev min/max 1=1.0,127.0 min/max 2=1.0,127.0 max abs diff=0.0000000000
pfull min/max 1=1.0,127.0 min/max 2=1.0,127.0 max abs diff=0.0000000000
ilev min/max 1=1.0,128.0 min/max 2=1.0,128.0 max abs diff=0.0000000000
hyai min/max 1=1.0,128.0 min/max 2=1.0,128.0 max abs diff=0.0000000000
hybi min/max 1=1.0,128.0 min/max 2=1.0,128.0 max abs diff=0.0000000000
u_inc min/max 1=-21.754034,21.008915 min/max 2=-21.754034,21.008915 max abs diff=0.0000000000
v_inc min/max 1=-28.7007,36.604904 min/max 2=-28.7007,36.604904 max abs diff=0.0000000000
delp_inc min/max 1=-7.103228,6.8421583 min/max 2=-7.103228,6.8421583 max abs diff=0.0000000000
delz_inc min/max 1=-164.41925,170.5055 min/max 2=-164.41925,170.5055 max abs diff=0.0000000000
T_inc min/max 1=-12.777468,12.321372 min/max 2=-12.777468,12.321372 max abs diff=0.0000000000
sphum_inc min/max 1=-0.007625651,0.0065353015 min/max 2=-0.007625651,0.0065353015 max abs diff=0.0000000000
liq_wat_inc min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
o3mr_inc min/max 1=-2.269526e-06,2.4629378e-06 min/max 2=-2.269526e-06,2.4629378e-06 max abs diff=0.0000000000
icmr_inc min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.dtfanl.nc gdas.t00z.dtfanl.nc
latitude min/max 1=-90.0,90.0 min/max 2=-90.0,90.0 max abs diff=0.0000000000
longitude min/max 1=0.0,359.76562 min/max 2=0.0,359.76562 max abs diff=0.0000000000
dtf min/max 1=-10.108167193460819,4.697151391349308 min/max 2=-10.108167193460819,4.697151391349308 max abs diff=0.0000000000
msk min/max 1=0,2 min/max 2=0,2 max abs diff=0.0000000000

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.atmi009.nc gdas.t00z.atmi009.nc
lon min/max 1=0.0,359.76562 min/max 2=0.0,359.76562 max abs diff=0.0000000000
lat min/max 1=-89.82071,89.82071 min/max 2=-89.82071,89.82071 max abs diff=0.0000000000
lev min/max 1=1.0,127.0 min/max 2=1.0,127.0 max abs diff=0.0000000000
pfull min/max 1=1.0,127.0 min/max 2=1.0,127.0 max abs diff=0.0000000000
ilev min/max 1=1.0,128.0 min/max 2=1.0,128.0 max abs diff=0.0000000000
hyai min/max 1=1.0,128.0 min/max 2=1.0,128.0 max abs diff=0.0000000000
hybi min/max 1=1.0,128.0 min/max 2=1.0,128.0 max abs diff=0.0000000000
u_inc min/max 1=-23.568714,20.07954 min/max 2=-23.568714,20.07954 max abs diff=0.0000000000
v_inc min/max 1=-26.775537,24.22583 min/max 2=-26.775537,24.22583 max abs diff=0.0000000000
delp_inc min/max 1=-12.052839,6.6196475 min/max 2=-12.052839,6.6196475 max abs diff=0.0000000000
delz_inc min/max 1=-256.5796,216.22807 min/max 2=-256.5796,216.22807 max abs diff=0.0000000000
T_inc min/max 1=-16.20386,19.22775 min/max 2=-16.20386,19.22775 max abs diff=0.0000000000
sphum_inc min/max 1=-0.008641035,0.008110714 min/max 2=-0.008641035,0.008110714 max abs diff=0.0000000000
liq_wat_inc min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
o3mr_inc min/max 1=-2.2247345e-06,2.1538244e-06 min/max 2=-2.2247345e-06,2.1538244e-06 max abs diff=0.0000000000
icmr_inc min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000

Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.abias_pc gdas.t00z.abias_pc
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.abias_air gdas.t00z.abias_air
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.abias gdas.t00z.abias

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.atmanl.nc gdas.t00z.atmanl.nc
grid_xt min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
lon min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
grid_yt min/max 1=-89.91032453466268,89.91032453466268 min/max 2=-89.91032453466268,89.91032453466268 max abs diff=0.0000000000
lat min/max 1=-89.91032453466268,89.91032453466268 min/max 2=-89.91032453466268,89.91032453466268 max abs diff=0.0000000000
pfull min/max 1=0.012781459,998.7807 min/max 2=0.012781459,998.7807 max abs diff=0.0000000000
phalf min/max 1=0.00999,1000.0 min/max 2=0.00999,1000.0 max abs diff=0.0000000000
time min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
clwmr min/max 1=-8.427844e-20,0.0014964561 min/max 2=-8.427844e-20,0.0014964561 max abs diff=0.0000000000
delz min/max 1=-2998.9822,-15.131367 min/max 2=-2998.9822,-15.131367 max abs diff=0.0000000000
dpres min/max 1=0.6059999,1816.0406 min/max 2=0.6059999,1816.0406 max abs diff=0.0000000000
dzdt min/max 1=-2.0649765,5.678376 min/max 2=-2.0649765,5.678376 max abs diff=0.0000000000
grle min/max 1=-2.1546461e-19,0.009992773 min/max 2=-2.1546461e-19,0.009992773 max abs diff=0.0000000000
hgtsfc min/max 1=-379.30286,6101.7183 min/max 2=-379.30286,6101.7183 max abs diff=0.0000000000
icmr min/max 1=-1.0978739e-20,0.0014054931 min/max 2=-1.0978739e-20,0.0014054931 max abs diff=0.0000000000
o3mr min/max 1=-8.6397634e-08,1.7660843e-05 min/max 2=-8.6397634e-08,1.7660843e-05 max abs diff=0.0000000000
pressfc min/max 1=47625.207,105787.04 min/max 2=47625.207,105787.04 max abs diff=0.0000000000
rwmr min/max 1=-6.3062466e-19,0.0043598786 min/max 2=-6.3062466e-19,0.0043598786 max abs diff=0.0000000000
snmr min/max 1=-9.111516e-20,0.0018711638 min/max 2=-9.111516e-20,0.0018711638 max abs diff=0.0000000000
spfh min/max 1=-0.0017183265,0.02087891 min/max 2=-0.0017183265,0.02087891 max abs diff=0.0000000000
tmp min/max 1=162.59467,311.63965 min/max 2=162.59467,311.63965 max abs diff=0.0000000000
ugrd min/max 1=-69.32334,101.98713 min/max 2=-69.32334,101.98713 max abs diff=0.0000000000
vgrd min/max 1=-77.42704,81.244225 min/max 2=-77.42704,81.244225 max abs diff=0.0000000000

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.sfcanl.nc gdas.t00z.sfcanl.nc
grid_xt min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
lon min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
grid_yt min/max 1=-89.90933990478516,89.90933990478516 min/max 2=-89.90933990478516,89.90933990478516 max abs diff=0.0000000000
lat min/max 1=-89.90933990478516,89.90933990478516 min/max 2=-89.90933990478516,89.90933990478516 max abs diff=0.0000000000
time min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
alnsf min/max 1=0.014551381,0.67130923 min/max 2=0.014551381,0.67130923 max abs diff=0.0000000000
alnwf min/max 1=0.013771121,0.66712266 min/max 2=0.013771121,0.66712266 max abs diff=0.0000000000
alvsf min/max 1=0.011501236,0.7448065 min/max 2=0.011501236,0.7448065 max abs diff=0.0000000000
alvwf min/max 1=0.011267143,0.7279612 min/max 2=0.011267143,0.7279612 max abs diff=0.0000000000
cnwat min/max 1=0.0,0.5 min/max 2=0.0,0.5 max abs diff=0.0000000000
crain min/max 1=0.0,1.0 min/max 2=0.0,1.0 max abs diff=0.0000000000
f10m min/max 1=0.9312268,1.5200102 min/max 2=0.9312268,1.5200102 max abs diff=0.0000000000
facsf min/max 1=0.0,1.0 min/max 2=0.0,1.0 max abs diff=0.0000000000
facwf min/max 1=-4.2632563e-16,1.0 min/max 2=-4.2632563e-16,1.0 max abs diff=0.0000000000
ffhh min/max 1=1.1871055,54.36069 min/max 2=1.1871055,54.36069 max abs diff=0.0000000000
ffmm min/max 1=1.3616332,56.513565 min/max 2=1.3616332,56.513565 max abs diff=0.0000000000
fricv min/max 1=0.0070779463,1.9398302 min/max 2=0.0070779463,1.9398302 max abs diff=0.0000000000
icec min/max 1=0.0,1.0 min/max 2=0.0,1.0 max abs diff=0.0000000000
icetk min/max 1=0.0,5.0009995 min/max 2=0.0,5.0009995 max abs diff=0.0000000000
land min/max 1=0.0,2.0 min/max 2=0.0,2.0 max abs diff=0.0000000000
orog min/max 1=-410.4434,6343.5 min/max 2=-410.4434,6343.5 max abs diff=0.0000000000
sfcr min/max 1=1e-07,2.653 min/max 2=1e-07,2.653 max abs diff=0.0000000000
shdmax min/max 1=0.0,0.99 min/max 2=0.0,0.99 max abs diff=0.0000000000
shdmin min/max 1=0.0,0.88987714 min/max 2=0.0,0.88987714 max abs diff=0.0000000000
sltyp min/max 1=0.0,9.0 min/max 2=0.0,9.0 max abs diff=0.0000000000
snoalb min/max 1=0.0,0.84 min/max 2=0.0,0.84 max abs diff=0.0000000000
snod min/max 1=0.0,4.8426147 min/max 2=0.0,4.8426147 max abs diff=0.0000000000
soill1 min/max 1=0.0,0.465 min/max 2=0.0,0.465 max abs diff=0.0000000000
soill2 min/max 1=0.0,0.4454564 min/max 2=0.0,0.4454564 max abs diff=0.0000000000
soill3 min/max 1=0.0,0.43812716 min/max 2=0.0,0.43812716 max abs diff=0.0000000000
soill4 min/max 1=0.0,0.4367312 min/max 2=0.0,0.4367312 max abs diff=0.0000000000
soilt1 min/max 1=221.19437,313.09464 min/max 2=221.19437,313.09464 max abs diff=0.0000000000
soilt2 min/max 1=221.92006,313.66183 min/max 2=221.92006,313.66183 max abs diff=0.0000000000
soilt3 min/max 1=221.13837,312.59158 min/max 2=221.13837,312.59158 max abs diff=0.0000000000
soilt4 min/max 1=219.41208,310.09665 min/max 2=219.41208,310.09665 max abs diff=0.0000000000
soilw1 min/max 1=0.02,1.0 min/max 2=0.02,1.0 max abs diff=0.0000000000
soilw2 min/max 1=0.023297971,1.0 min/max 2=0.023297971,1.0 max abs diff=0.0000000000
soilw3 min/max 1=0.023275392,1.0 min/max 2=0.023275392,1.0 max abs diff=0.0000000000
soilw4 min/max 1=0.027600043,1.0 min/max 2=0.027600043,1.0 max abs diff=0.0000000000
sotyp min/max 1=0.0,16.0 min/max 2=0.0,16.0 max abs diff=0.0000000000
spfh2m min/max 1=4.7579983e-06,0.021767922 min/max 2=4.7579983e-06,0.021767922 max abs diff=0.0000000000
tg3 min/max 1=215.4809,304.41296 min/max 2=215.4809,304.41296 max abs diff=0.0000000000
tisfc min/max 1=201.7845,317.4675 min/max 2=201.7845,317.4675 max abs diff=0.0000000000
tmp2m min/max 1=207.24977,312.1273 min/max 2=207.24977,312.1273 max abs diff=0.0000000000
tmpsfc min/max 1=201.7845,317.4675 min/max 2=201.7845,317.4675 max abs diff=0.0000000000
tprcp min/max 1=0.0,0.004687114 min/max 2=0.0,0.004687114 max abs diff=0.0000000000
veg min/max 1=0.0,97.49154 min/max 2=0.0,97.49154 max abs diff=0.0000000000
vtype min/max 1=0.0,20.0 min/max 2=0.0,20.0 max abs diff=0.0000000000
weasd min/max 1=0.0,518.67633 min/max 2=0.0,518.67633 max abs diff=0.0000000000
c0 min/max 1=0.0,0.08514695 min/max 2=0.0,0.08514695 max abs diff=0.0000000000
cd min/max 1=-216.82819,0.0 min/max 2=-216.82819,0.0 max abs diff=0.0000000000
dconv min/max 1=0.0,6.6856475 min/max 2=0.0,6.6856475 max abs diff=0.0000000000
dtcool min/max 1=0.0,1.8446044 min/max 2=0.0,1.8446044 max abs diff=0.0000000000
qrain min/max 1=-32.954117,65.05079 min/max 2=-32.954117,65.05079 max abs diff=0.0000000000
tref min/max 1=213.06844,337.0954 min/max 2=213.06844,337.0954 max abs diff=0.0000000000
w0 min/max 1=-102.98286,5.6856933 min/max 2=-102.98286,5.6856933 max abs diff=0.0000000000
wd min/max 1=-3.043744,121.84093 min/max 2=-3.043744,121.84093 max abs diff=0.0000000000
xs min/max 1=-0.33111978,0.10088074 min/max 2=-0.33111978,0.10088074 max abs diff=0.0000000000
xt min/max 1=0.0,4.2146363 min/max 2=0.0,4.2146363 max abs diff=0.0000000000
xtts min/max 1=-40.43849,37.60145 min/max 2=-40.43849,37.60145 max abs diff=0.0000000000
xu min/max 1=-2.3518295,1.9517245 min/max 2=-2.3518295,1.9517245 max abs diff=0.0000000000
xv min/max 1=-2.104838,2.541633 min/max 2=-2.104838,2.541633 max abs diff=0.0000000000
xz min/max 1=0.2,30.0 min/max 2=0.2,30.0 max abs diff=0.0000000000
xzts min/max 1=-147337.47,17137524.0 min/max 2=-147337.47,17137524.0 max abs diff=0.0000000000
zc min/max 1=0.0,0.0056810207 min/max 2=0.0,0.0056810207 max abs diff=0.0000000000

Spot check of radstat tarball contents show it matches. Spot check of RESTART files (compared to _bfgldas versions in ops) show they match. Will run gdasgldas and compare RESTART files again.

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.atma003.ensres.nc gdas.t00z.atma003.ensres.nc
grid_xt min/max 1=0.0,359.765625 min/max 2=0.0,359.765625 max abs diff=0.0000000000
lon min/max 1=0.0,359.765625 min/max 2=0.0,359.765625 max abs diff=0.0000000000
grid_yt min/max 1=-89.82070742329198,89.82070742329198 min/max 2=-89.82070742329198,89.82070742329198 max abs diff=0.0000000000
lat min/max 1=-89.82070742329198,89.82070742329198 min/max 2=-89.82070742329198,89.82070742329198 max abs diff=0.0000000000
pfull min/max 1=0.012781459,998.7807 min/max 2=0.012781459,998.7807 max abs diff=0.0000000000
phalf min/max 1=0.00999,1000.0 min/max 2=0.00999,1000.0 max abs diff=0.0000000000
time min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
clwmr min/max 1=-8.477363e-20,0.0013122681 min/max 2=-8.477363e-20,0.0013122681 max abs diff=0.0000000000
delz min/max 1=-2990.0808,-15.186266 min/max 2=-2990.0808,-15.186266 max abs diff=0.0000000000
dpres min/max 1=0.606,1769.8995 min/max 2=0.606,1769.8995 max abs diff=0.0000000000
dzdt min/max 1=-1.7614557,6.089975 min/max 2=-1.7614557,6.089975 max abs diff=0.0000000000
grle min/max 1=-8.247967e-20,0.009453941 min/max 2=-8.247967e-20,0.009453941 max abs diff=0.0000000000
hgtsfc min/max 1=-119.835884,5786.296 min/max 2=-119.835884,5786.296 max abs diff=0.0000000000
icmr min/max 1=-1.0205619e-20,0.0011841579 min/max 2=-1.0205619e-20,0.0011841579 max abs diff=0.0000000000
o3mr min/max 1=-9.3227186e-08,1.7603e-05 min/max 2=-9.3227186e-08,1.7603e-05 max abs diff=0.0000000000
pressfc min/max 1=49624.977,103704.06 min/max 2=49624.977,103704.06 max abs diff=0.0000000000
rwmr min/max 1=-6.0916987e-19,0.0049264026 min/max 2=-6.0916987e-19,0.0049264026 max abs diff=0.0000000000
snmr min/max 1=-6.9446446e-20,0.0018121777 min/max 2=-6.9446446e-20,0.0018121777 max abs diff=0.0000000000
spfh min/max 1=-0.00085838325,0.02147977 min/max 2=-0.00085838325,0.02147977 max abs diff=0.0000000000
tmp min/max 1=161.20972,312.07715 min/max 2=161.20972,312.07715 max abs diff=0.0000000000
ugrd min/max 1=-73.133354,104.9381 min/max 2=-73.133354,104.9381 max abs diff=0.0000000000
vgrd min/max 1=-72.00801,86.377304 min/max 2=-72.00801,86.377304 max abs diff=0.0000000000

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.atmanl.ensres.nc gdas.t00z.atmanl.ensres.nc
grid_xt min/max 1=0.0,359.765625 min/max 2=0.0,359.765625 max abs diff=0.0000000000
lon min/max 1=0.0,359.765625 min/max 2=0.0,359.765625 max abs diff=0.0000000000
grid_yt min/max 1=-89.82070742329198,89.82070742329198 min/max 2=-89.82070742329198,89.82070742329198 max abs diff=0.0000000000
lat min/max 1=-89.82070742329198,89.82070742329198 min/max 2=-89.82070742329198,89.82070742329198 max abs diff=0.0000000000
pfull min/max 1=0.012781459,998.7807 min/max 2=0.012781459,998.7807 max abs diff=0.0000000000
phalf min/max 1=0.00999,1000.0 min/max 2=0.00999,1000.0 max abs diff=0.0000000000
time min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
clwmr min/max 1=-8.427846e-20,0.0013056577 min/max 2=-8.427846e-20,0.0013056577 max abs diff=0.0000000000
delz min/max 1=-2998.2173,-15.18072 min/max 2=-2998.2173,-15.18072 max abs diff=0.0000000000
dpres min/max 1=0.606,1776.3214 min/max 2=0.606,1776.3214 max abs diff=0.0000000000
dzdt min/max 1=-1.8827755,4.926549 min/max 2=-1.8827755,4.926549 max abs diff=0.0000000000
grle min/max 1=-2.1546465e-19,0.008091853 min/max 2=-2.1546465e-19,0.008091853 max abs diff=0.0000000000
hgtsfc min/max 1=-119.835884,5786.296 min/max 2=-119.835884,5786.296 max abs diff=0.0000000000
icmr min/max 1=-1.0978742e-20,0.0011152404 min/max 2=-1.0978742e-20,0.0011152404 max abs diff=0.0000000000
o3mr min/max 1=-7.8560326e-08,1.7659686e-05 min/max 2=-7.8560326e-08,1.7659686e-05 max abs diff=0.0000000000
pressfc min/max 1=49693.574,103994.63 min/max 2=49693.574,103994.63 max abs diff=0.0000000000
rwmr min/max 1=-6.306248e-19,0.0037962685 min/max 2=-6.306248e-19,0.0037962685 max abs diff=0.0000000000
snmr min/max 1=-9.111518e-20,0.0017715709 min/max 2=-9.111518e-20,0.0017715709 max abs diff=0.0000000000
spfh min/max 1=-0.0009801429,0.021138854 min/max 2=-0.0009801429,0.021138854 max abs diff=0.0000000000
tmp min/max 1=162.57936,311.14023 min/max 2=162.57936,311.14023 max abs diff=0.0000000000
ugrd min/max 1=-69.39376,102.00431 min/max 2=-69.39376,102.00431 max abs diff=0.0000000000
vgrd min/max 1=-77.417595,81.25825 min/max 2=-77.417595,81.25825 max abs diff=0.0000000000

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.atma009.ensres.nc gdas.t00z.atma009.ensres.nc
grid_xt min/max 1=0.0,359.765625 min/max 2=0.0,359.765625 max abs diff=0.0000000000
lon min/max 1=0.0,359.765625 min/max 2=0.0,359.765625 max abs diff=0.0000000000
grid_yt min/max 1=-89.82070742329198,89.82070742329198 min/max 2=-89.82070742329198,89.82070742329198 max abs diff=0.0000000000
lat min/max 1=-89.82070742329198,89.82070742329198 min/max 2=-89.82070742329198,89.82070742329198 max abs diff=0.0000000000
pfull min/max 1=0.012781459,998.7807 min/max 2=0.012781459,998.7807 max abs diff=0.0000000000
phalf min/max 1=0.00999,1000.0 min/max 2=0.00999,1000.0 max abs diff=0.0000000000
time min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
clwmr min/max 1=-8.258654e-20,0.0012861144 min/max 2=-8.258654e-20,0.0012861144 max abs diff=0.0000000000
delz min/max 1=-2997.2996,-15.241574 min/max 2=-2997.2996,-15.241574 max abs diff=0.0000000000
dpres min/max 1=0.606,1785.652 min/max 2=0.606,1785.652 max abs diff=0.0000000000
dzdt min/max 1=-1.3455415,3.991395 min/max 2=-1.3455415,3.991395 max abs diff=0.0000000000
grle min/max 1=-7.30128e-20,0.008249128 min/max 2=-7.30128e-20,0.008249128 max abs diff=0.0000000000
hgtsfc min/max 1=-119.835884,5786.296 min/max 2=-119.835884,5786.296 max abs diff=0.0000000000
icmr min/max 1=-1.351192e-20,0.0011348715 min/max 2=-1.351192e-20,0.0011348715 max abs diff=0.0000000000
o3mr min/max 1=-7.767932e-08,1.7676442e-05 min/max 2=-7.767932e-08,1.7676442e-05 max abs diff=0.0000000000
pressfc min/max 1=49800.977,104414.54 min/max 2=49800.977,104414.54 max abs diff=0.0000000000
rwmr min/max 1=-4.457228e-19,0.0037036564 min/max 2=-4.457228e-19,0.0037036564 max abs diff=0.0000000000
snmr min/max 1=-7.351055e-20,0.0017131253 min/max 2=-7.351055e-20,0.0017131253 max abs diff=0.0000000000
spfh min/max 1=-0.0012101512,0.021259304 min/max 2=-0.0012101512,0.021259304 max abs diff=0.0000000000
tmp min/max 1=162.72733,309.68723 min/max 2=162.72733,309.68723 max abs diff=0.0000000000
ugrd min/max 1=-84.348114,94.160995 min/max 2=-84.348114,94.160995 max abs diff=0.0000000000
vgrd min/max 1=-73.25235,79.37716 min/max 2=-73.25235,79.37716 max abs diff=0.0000000000
KateFriedman-NOAA commented 2 years ago

Test: Reproduce WCOSS2 ops output using dev_v16 branch

gdasgldas output

The updated RESTART files produced by the gdasgldas job match WCOSS2 ops. Showing the comparison for one of the six tile files:

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/RESTART/20220329.210000.sfcanl_data.tile1.nc RESTART/20220329.210000.sfcanl_data.tile1.nc
xaxis_1 min/max 1=1.0,768.0 min/max 2=1.0,768.0 max abs diff=0.0000000000
yaxis_1 min/max 1=1.0,768.0 min/max 2=1.0,768.0 max abs diff=0.0000000000
zaxis_1 min/max 1=1.0,4.0 min/max 2=1.0,4.0 max abs diff=0.0000000000
Time min/max 1=1.0,1.0 min/max 2=1.0,1.0 max abs diff=0.0000000000
slmsk min/max 1=0.0,1.0 min/max 2=0.0,1.0 max abs diff=0.0000000000
tsea min/max 1=263.76942549818534,308.75915471819656 min/max 2=263.76942549818534,308.75915471819656 max abs diff=0.0000000000
sheleg min/max 1=0.0,64.21526559712935 min/max 2=0.0,64.21526559712935 max abs diff=0.0000000000
tg3 min/max 1=281.8829098448854,303.72505006563256 min/max 2=281.8829098448854,303.72505006563256 max abs diff=0.0000000000
zorl min/max 1=9.999999999999999e-06,265.3 min/max 2=9.999999999999999e-06,265.3 max abs diff=0.0000000000
alvsf min/max 1=0.01704918032786885,0.4174951778740374 min/max 2=0.01704918032786885,0.4174951778740374 max abs diff=0.0000000000
alvwf min/max 1=0.016726287874619866,0.4123224970480478 min/max 2=0.016726287874619866,0.4123224970480478 max abs diff=0.0000000000
alnsf min/max 1=0.042974080152084614,0.6713092076709293 min/max 2=0.042974080152084614,0.6713092076709293 max abs diff=0.0000000000
alnwf min/max 1=0.04219951328992869,0.6671226530329384 min/max 2=0.04219951328992869,0.6671226530329384 max abs diff=0.0000000000
facsf min/max 1=0.0,1.0000000000000004 min/max 2=0.0,1.0000000000000004 max abs diff=0.0000000000
facwf min/max 1=-4.263256414560601e-16,1.0 min/max 2=-4.263256414560601e-16,1.0 max abs diff=0.0000000000
vfrac min/max 1=0.0,0.9749153776687314 min/max 2=0.0,0.9749153776687314 max abs diff=0.0000000000
canopy min/max 1=0.0,0.5 min/max 2=0.0,0.5 max abs diff=0.0000000000
f10m min/max 1=0.9356037416118544,1.2019804169035568 min/max 2=0.9356037416118544,1.2019804169035568 max abs diff=0.0000000000
t2m min/max 1=268.6984876762248,310.75398951928497 min/max 2=268.6984876762248,310.75398951928497 max abs diff=0.0000000000
q2m min/max 1=0.0005350092310907976,0.02051786772873326 min/max 2=0.0005350092310907976,0.02051786772873326 max abs diff=0.0000000000
vtype min/max 1=0.0,16.0 min/max 2=0.0,16.0 max abs diff=0.0000000000
stype min/max 1=0.0,15.0 min/max 2=0.0,15.0 max abs diff=0.0000000000
uustar min/max 1=0.007066188131337192,1.4265273092676327 min/max 2=0.007066188131337192,1.4265273092676327 max abs diff=0.0000000000
ffmm min/max 1=1.498413194141384,56.60760689714396 min/max 2=1.498413194141384,56.60760689714396 max abs diff=0.0000000000
ffhh min/max 1=1.4690141278203748,54.454765940090965 min/max 2=1.4690141278203748,54.454765940090965 max abs diff=0.0000000000
hice min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
fice min/max 1=0.0,0.0 min/max 2=0.0,0.0 max abs diff=0.0000000000
tisfc min/max 1=263.76942549818534,308.75915471819656 min/max 2=263.76942549818534,308.75915471819656 max abs diff=0.0000000000
tprcp min/max 1=0.0,0.007393124224232047 min/max 2=0.0,0.007393124224232047 max abs diff=0.0000000000
srflag min/max 1=0.0,1.0 min/max 2=0.0,1.0 max abs diff=0.0000000000
snwdph min/max 1=0.0,500.94871597938726 min/max 2=0.0,500.94871597938726 max abs diff=0.0000000000
shdmin min/max 1=0.0,0.8354866204198486 min/max 2=0.0,0.8354866204198486 max abs diff=0.0000000000
shdmax min/max 1=0.0,0.9900000000000004 min/max 2=0.0,0.9900000000000004 max abs diff=0.0000000000
slope min/max 1=0.0,7.0 min/max 2=0.0,7.0 max abs diff=0.0000000000
snoalb min/max 1=0.0,0.8300000000000003 min/max 2=0.0,0.8300000000000003 max abs diff=0.0000000000
tref min/max 1=272.6720275878906,309.0691223144531 min/max 2=272.6720275878906,309.0691223144531 max abs diff=0.0000000000
z_c min/max 1=0.0,0.005340186684247355 min/max 2=0.0,0.005340186684247355 max abs diff=0.0000000000
c_0 min/max 1=0.0,0.07360508346849962 min/max 2=0.0,0.07360508346849962 max abs diff=0.0000000000
c_d min/max 1=-212.9880706828275,0.0 min/max 2=-212.9880706828275,0.0 max abs diff=0.0000000000
w_0 min/max 1=-5.152063762781449,0.14600952132945555 min/max 2=-5.152063762781449,0.14600952132945555 max abs diff=0.0000000000
w_d min/max 1=-0.01573076534374443,4.578985220708044 min/max 2=-0.01573076534374443,4.578985220708044 max abs diff=0.0000000000
xt min/max 1=0.0,3.487305494840218 min/max 2=0.0,3.487305494840218 max abs diff=0.0000000000
xs min/max 1=-0.13128204694505569,0.08539447495157564 min/max 2=-0.13128204694505569,0.08539447495157564 max abs diff=0.0000000000
xu min/max 1=-0.9602265009878532,1.8456233757208316 min/max 2=-0.9602265009878532,1.8456233757208316 max abs diff=0.0000000000
xv min/max 1=-1.4665190600979536,1.1668684489884842 min/max 2=-1.4665190600979536,1.1668684489884842 max abs diff=0.0000000000
xz min/max 1=0.47444194686445446,30.0 min/max 2=0.47444194686445446,30.0 max abs diff=0.0000000000
zm min/max 1=0.0,22.87152651978927 min/max 2=0.0,22.87152651978927 max abs diff=0.0000000000
xtts min/max 1=-1.872591241157932,2.420976750933093 min/max 2=-1.872591241157932,2.420976750933093 max abs diff=0.0000000000
xzts min/max 1=-1527.521553323999,179912.74820863627 min/max 2=-1527.521553323999,179912.74820863627 max abs diff=0.0000000000
d_conv min/max 1=0.0,3.9904417116041575 min/max 2=0.0,3.9904417116041575 max abs diff=0.0000000000
ifd min/max 1=0.0,1.0 min/max 2=0.0,1.0 max abs diff=0.0000000000
dt_cool min/max 1=0.0,0.6744665018260807 min/max 2=0.0,0.6744665018260807 max abs diff=0.0000000000
qrain min/max 1=-1.5683806205489517,105.23804297541086 min/max 2=-1.5683806205489517,105.23804297541086 max abs diff=0.0000000000
tfinc min/max 1=-0.5375568165733853,0.830027836018985 min/max 2=-0.5375568165733853,0.830027836018985 max abs diff=0.0000000000
stc min/max 1=272.6064758300781,315.4602966308594 min/max 2=272.6064758300781,315.4602966308594 max abs diff=0.0000000000
smc min/max 1=0.011955256573855877,1.0 min/max 2=0.011955256573855877,1.0 max abs diff=0.0000000000
slc min/max 1=0.011955256573855877,1.0 min/max 2=0.011955256573855877,1.0 max abs diff=0.0000000000
KateFriedman-NOAA commented 2 years ago

Test: Reproduce WCOSS2 ops output using dev_v16 branch

gdasfcst and gdaspost* output

The output produced by the gdasfcst and gdaspost* jobs matches WCOSS2 ops output. Showing just the atmf009.nc and pgrb2.0p25.f009 file comparisons below.

Kate.Friedman@dlogin04:> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.atmf009.nc gdas.t00z.atmf009.nc
grid_xt min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
lon min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
grid_yt min/max 1=-89.91032453466268,89.91032453466268 min/max 2=-89.91032453466268,89.91032453466268 max abs diff=0.0000000000
lat min/max 1=-89.91032453466268,89.91032453466268 min/max 2=-89.91032453466268,89.91032453466268 max abs diff=0.0000000000
pfull min/max 1=0.012781459,998.7807 min/max 2=0.012781459,998.7807 max abs diff=0.0000000000
phalf min/max 1=0.00999,1000.0 min/max 2=0.00999,1000.0 max abs diff=0.0000000000
time min/max 1=9.0,9.0 min/max 2=9.0,9.0 max abs diff=0.0000000000
clwmr min/max 1=-8.936655e-20,0.0018938886 min/max 2=-8.936655e-20,0.0018938886 max abs diff=0.0000000000
delz min/max 1=-3002.3032,-15.381293 min/max 2=-3002.3032,-15.381293 max abs diff=0.0000000000
dpres min/max 1=0.6059999,1823.4427 min/max 2=0.6059999,1823.4427 max abs diff=0.0000000000
dzdt min/max 1=-2.612571,7.2027 min/max 2=-2.612571,7.2027 max abs diff=0.0000000000
grle min/max 1=-1.0875053e-19,0.009410312 min/max 2=-1.0875053e-19,0.009410312 max abs diff=0.0000000000
hgtsfc min/max 1=-379.30286,6101.7183 min/max 2=-379.30286,6101.7183 max abs diff=0.0000000000
icmr min/max 1=-1.24286e-20,0.001388567 min/max 2=-1.24286e-20,0.001388567 max abs diff=0.0000000000
o3mr min/max 1=2.9325656e-10,1.7591701e-05 min/max 2=2.9325656e-10,1.7591701e-05 max abs diff=0.0000000000
pressfc min/max 1=47704.547,106120.16 min/max 2=47704.547,106120.16 max abs diff=0.0000000000
rwmr min/max 1=-6.816188e-19,0.0054538916 min/max 2=-6.816188e-19,0.0054538916 max abs diff=0.0000000000
snmr min/max 1=-1.04265864e-19,0.0025203314 min/max 2=-1.04265864e-19,0.0025203314 max abs diff=0.0000000000
spfh min/max 1=1e-09,0.020775383 min/max 2=1e-09,0.020775383 max abs diff=0.0000000000
tmp min/max 1=160.70917,314.9357 min/max 2=160.70917,314.9357 max abs diff=0.0000000000
ugrd min/max 1=-92.12336,86.08903 min/max 2=-92.12336,86.08903 max abs diff=0.0000000000
vgrd min/max 1=-70.63443,82.11716 min/max 2=-70.63443,82.11716 max abs diff=0.0000000000
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/00/atmos/gdas.t00z.pgrb2.0p25.f009 gdas.t00z.pgrb2.0p25.f009

Will let the test run the rest of the enkf/gfs suites and then progress into the next cycle to check reproducibility continues.

KateFriedman-NOAA commented 2 years ago

Test: Reproduce WCOSS2 ops output using dev_v16 branch

The prep jobs in the next cycle in the test (2022033006) also reproduced WCOSS2 ops output:

Kate.Friedman@dlogin04:> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/gdas.20220330/06/atmos
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/obsproc/v1.0/gdas.20220330/06/atmos/gdas.t06z.prepbufr gdas.t06z.prepbufr
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/obsproc/v1.0/gdas.20220330/06/atmos/gdas.t06z.nsstbufr gdas.t06z.nsstbufr 
Kate.Friedman@dlogin04:> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/gfs.20220330/06/atmos
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/obsproc/v1.0/gfs.20220330/06/atmos/gfs.t06z.prepbufr gfs.t06z.prepbufr
Kate.Friedman@dlogin04:> cmp /lfs/h1/ops/prod/com/obsproc/v1.0/gfs.20220330/06/atmos/gfs.t06z.nsstbufr gfs.t06z.nsstbufr

These jobs used WCOSS2 ops obsproc output and prior cycle forecast output produced by this test.

Turning on cron for this test so rocoto let's it finish up overnight. Will compare additional output when it completes to confirm it continues to reproduce WCOSS2 ops output.

KateFriedman-NOAA commented 2 years ago

Test: Reproduce WCOSS2 ops output using dev_v16 branch

After sorting out a small error in my experiment setup that impacted the enkf_forecast jobs, I have rerun the rest of the reproducibility test and confirmed I can reproduce WCOSS2 ops output using the dev_v16 branch.

The 2nd cycle (2022033006) outputs match WCOSS2 ops:

06z enkf post output for f009:

Kate.Friedman@dlogin03> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/enkfgdas.20220330/06/atmos
Kate.Friedman@dlogin03> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/enkfgdas.20220330/06/at
mos/gdas.t06z.atmf009.ensmean.nc gdas.t06z.atmf009.ensmean.nc
grid_xt min/max 1=0.0,359.765625 min/max 2=0.0,359.765625 max abs diff=0.0000000000
lon min/max 1=0.0,359.765625 min/max 2=0.0,359.765625 max abs diff=0.0000000000
grid_yt min/max 1=-89.82070742329198,89.82070742329198 min/max 2=-89.82070742329198,89.82070742329198 max abs diff=0.0000000000
lat min/max 1=-89.82070742329198,89.82070742329198 min/max 2=-89.82070742329198,89.82070742329198 max abs diff=0.0000000000
pfull min/max 1=0.012781459,998.7807 min/max 2=0.012781459,998.7807 max abs diff=0.0000000000
phalf min/max 1=0.00999,1000.0 min/max 2=0.00999,1000.0 max abs diff=0.0000000000
time min/max 1=9.0,9.0 min/max 2=9.0,9.0 max abs diff=0.0000000000
clwmr min/max 1=-7.215729e-20,0.001700558 min/max 2=-7.215729e-20,0.001700558 max abs diff=0.0000000000
delz min/max 1=-2966.4565,-15.288778 min/max 2=-2966.4565,-15.288778 max abs diff=0.0000000000
dpres min/max 1=0.6059998,1803.5619 min/max 2=0.6059998,1803.5619 max abs diff=0.0000000000
dzdt min/max 1=-0.6196008,0.64451474 min/max 2=-0.6196008,0.64451474 max abs diff=0.0000000000
grle min/max 1=-3.4804316e-20,0.0010132423 min/max 2=-3.4804316e-20,0.0010132423 max abs diff=0.0000000000
hgtsfc min/max 1=-119.835884,5786.296 min/max 2=-119.835884,5786.296 max abs diff=0.0000000000
icmr min/max 1=-8.8497614e-21,0.00041328426 min/max 2=-8.8497614e-21,0.00041328426 max abs diff=0.0000000000
o3mr min/max 1=1.0197246e-08,1.7348355e-05 min/max 2=1.0197246e-08,1.7348355e-05 max abs diff=0.0000000000
pressfc min/max 1=49940.82,105221.86 min/max 2=49940.82,105221.86 max abs diff=0.0000000000
rwmr min/max 1=-3.0641215e-19,0.00087187346 min/max 2=-3.0641215e-19,0.00087187346 max abs diff=0.0000000000
snmr min/max 1=-6.332252e-20,0.00080350286 min/max 2=-6.332252e-20,0.00080350286 max abs diff=0.0000000000
spfh min/max 1=1.0041117e-09,0.020204144 min/max 2=1.0041117e-09,0.020204144 max abs diff=0.0000000000
tmp min/max 1=165.3218,315.61142 min/max 2=165.3218,315.61142 max abs diff=0.0000000000
ugrd min/max 1=-72.52525,81.7354 min/max 2=-72.52525,81.7354 max abs diff=0.0000000000
vgrd min/max 1=-67.97119,86.682655 min/max 2=-67.97119,86.682655 max abs diff=0.0000000000

06z gdas deterministic f009 output:

Kate.Friedman@dlogin03> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/gdas.20220330/06/atmos
Kate.Friedman@dlogin03> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gdas.20220330/06/atmos/gdas.t06z.atmf009.nc gdas.t06z.atmf009.nc
grid_xt min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
lon min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
grid_yt min/max 1=-89.91032453466268,89.91032453466268 min/max 2=-89.91032453466268,89.91032453466268 max abs diff=0.0000000000
lat min/max 1=-89.91032453466268,89.91032453466268 min/max 2=-89.91032453466268,89.91032453466268 max abs diff=0.0000000000
pfull min/max 1=0.012781459,998.7807 min/max 2=0.012781459,998.7807 max abs diff=0.0000000000
phalf min/max 1=0.00999,1000.0 min/max 2=0.00999,1000.0 max abs diff=0.0000000000
time min/max 1=9.0,9.0 min/max 2=9.0,9.0 max abs diff=0.0000000000
clwmr min/max 1=-1.0204776e-19,0.0016611789 min/max 2=-1.0204776e-19,0.0016611789 max abs diff=0.0000000000
delz min/max 1=-3035.7993,-15.224168 min/max 2=-3035.7993,-15.224168 max abs diff=0.0000000000
dpres min/max 1=0.6059999,1819.4849 min/max 2=0.6059999,1819.4849 max abs diff=0.0000000000
dzdt min/max 1=-1.7201232,7.426127 min/max 2=-1.7201232,7.426127 max abs diff=0.0000000000
grle min/max 1=-7.145054e-20,0.009798346 min/max 2=-7.145054e-20,0.009798346 max abs diff=0.0000000000
hgtsfc min/max 1=-379.30286,6101.7183 min/max 2=-379.30286,6101.7183 max abs diff=0.0000000000
icmr min/max 1=-9.918754e-21,0.0013575014 min/max 2=-9.918754e-21,0.0013575014 max abs diff=0.0000000000
o3mr min/max 1=1.3570168e-09,1.7514541e-05 min/max 2=1.3570168e-09,1.7514541e-05 max abs diff=0.0000000000
pressfc min/max 1=47846.504,105941.26 min/max 2=47846.504,105941.26 max abs diff=0.0000000000
rwmr min/max 1=-5.346603e-19,0.005395917 min/max 2=-5.346603e-19,0.005395917 max abs diff=0.0000000000
snmr min/max 1=-8.789506e-20,0.0020873265 min/max 2=-8.789506e-20,0.0020873265 max abs diff=0.0000000000
spfh min/max 1=1.1660929e-09,0.020585036 min/max 2=1.1660929e-09,0.020585036 max abs diff=0.0000000000
tmp min/max 1=157.89745,315.28204 min/max 2=157.89745,315.28204 max abs diff=0.0000000000
ugrd min/max 1=-80.64036,88.60494 min/max 2=-80.64036,88.60494 max abs diff=0.0000000000
vgrd min/max 1=-86.521904,89.668144 min/max 2=-86.521904,89.668144 max abs diff=0.0000000000

06z gfs deterministic f384 output:

Kate.Friedman@dlogin03> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/gfs.20220330/06/atmos
Kate.Friedman@dlogin03> compare_ncfile.py /lfs/h1/ops/prod/com/gfs/v16.2/gfs.20220330/06/atmos/gfs.t06z.atmf384.nc gfs.t06z.atmf384.nc
grid_xt min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
lon min/max 1=0.0,359.8828125 min/max 2=0.0,359.8828125 max abs diff=0.0000000000
grid_yt min/max 1=-89.91032453466268,89.91032453466268 min/max 2=-89.91032453466268,89.91032453466268 max abs diff=0.0000000000
lat min/max 1=-89.91032453466268,89.91032453466268 min/max 2=-89.91032453466268,89.91032453466268 max abs diff=0.0000000000
pfull min/max 1=0.012781459,998.7807 min/max 2=0.012781459,998.7807 max abs diff=0.0000000000
phalf min/max 1=0.00999,1000.0 min/max 2=0.00999,1000.0 max abs diff=0.0000000000
time min/max 1=384.0,384.0 min/max 2=384.0,384.0 max abs diff=0.0000000000
cld_amt min/max 1=0.0,1.0000001 min/max 2=0.0,1.0000001 max abs diff=0.0000000000
clwmr min/max 1=-1.0440907e-19,0.0017215309 min/max 2=-1.0440907e-19,0.0017215309 max abs diff=0.0000000000
delz min/max 1=-3119.6829,-14.860414 min/max 2=-3119.6829,-14.860414 max abs diff=0.0000000000
dpres min/max 1=0.6059999,1815.2563 min/max 2=0.6059999,1815.2563 max abs diff=0.0000000000
dzdt min/max 1=-6.8455834,7.1265707 min/max 2=-6.8455834,7.1265707 max abs diff=0.0000000000
grle min/max 1=-5.0632123e-20,0.011119593 min/max 2=-5.0632123e-20,0.011119593 max abs diff=0.0000000000
hgtsfc min/max 1=-379.30286,6101.7183 min/max 2=-379.30286,6101.7183 max abs diff=0.0000000000
icmr min/max 1=-1.1106019e-20,0.0011855357 min/max 2=-1.1106019e-20,0.0011855357 max abs diff=0.0000000000
o3mr min/max 1=1.7707457e-08,1.7583568e-05 min/max 2=1.7707457e-08,1.7583568e-05 max abs diff=0.0000000000
pressfc min/max 1=47788.145,105750.26 min/max 2=47788.145,105750.26 max abs diff=0.0000000000
rwmr min/max 1=-5.980235e-19,0.004645411 min/max 2=-5.980235e-19,0.004645411 max abs diff=0.0000000000
snmr min/max 1=-7.7506933e-20,0.0020104484 min/max 2=-7.7506933e-20,0.0020104484 max abs diff=0.0000000000
spfh min/max 1=2.5909117e-09,0.022440564 min/max 2=2.5909117e-09,0.022440564 max abs diff=0.0000000000
tmp min/max 1=165.812,318.27298 min/max 2=165.812,318.27298 max abs diff=0.0000000000
ugrd min/max 1=-78.2065,124.05551 min/max 2=-78.2065,124.05551 max abs diff=0.0000000000
vgrd min/max 1=-81.814804,77.85331 min/max 2=-81.814804,77.85331 max abs diff=0.0000000000

06z sample wave output:

Kate.Friedman@dlogin03> pwd
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/devv16ops/gfs.20220330/06/wave/rundata
Kate.Friedman@dlogin03> cmp /lfs/h1/ops/prod/com/gfs/v16.2/gfs.20220330/06/wave/rundata/gfswave.out_grd.gsh_15m.20220415.060000 gfswave.out_grd.gsh_15m.20220415.060000
Kate.Friedman@dlogin03>
AndrewEichmann-NOAA commented 2 years ago

Is dev_v16 expected to work on Orion at this point? When building, gldas fails, and sorc/gldas.fd/sorc/logs/build_gdas2gldas.log has the following:

mpiifort -O3 -fp-model precise -g -traceback -r8 -i4 -qopenmp -convert big_endian -assume byterecl -c model_grid.F90 model_grid.F90(3): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [ESMF] use esmf -----^ model_grid.F90(11): error #6457: This derived type name has not been declared. [ESMF_GRID] type(esmf_grid), public :: input_grid ------^ model_grid.F90(12): error #6457: This derived type name has not been declared. [ESMF_GRID] type(esmf_grid), public :: target_grid ------^ model_grid.F90(14): error #6457: This derived type name has not been declared. [ESMF_FIELD] type(esmf_field), public :: latitude_target_grid ------^ model_grid.F90(15): error #6457: This derived type name has not been declared. [ESMF_FIELD] type(esmf_field), public :: longitude_target_grid ------^

and continues to fail on ESMF variables.

The build is under /work/noaa/da/aeichman/fv3gfs/dev_v16/global-workflow

KateFriedman-NOAA commented 2 years ago

@AndrewEichmann-NOAA Work on dev_v16 is not yet complete and we are not yet supporting it anywhere. An announcement will be made when it's ready for use.

AndrewEichmann-NOAA commented 2 years ago

@KateFriedman-NOAA Ok, thank you

KateFriedman-NOAA commented 2 years ago

Released the GFSv16.2.x package to users: https://github.com/NOAA-EMC/global-workflow/releases/tag/gfs.v16.2.2

Also announced it to the glopara-announce and fv3-announce listservs.