search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
74 stars 167 forks source link

Accelerated RFC to place PlanetiQ RO data in monitor mode #2951

Open RussTreadon-NOAA opened 4 days ago

RussTreadon-NOAA commented 4 days ago

Description

PlanetiQ data has added noise. This issue issue is opened in case we decide to place this data in monitor mode until we more fully examine it.

Target version

v16.3.19

Expected workflow changes

The only file changed is global_convinfo.txt. This requires a new GSI tag and a new GFS tag.

Tasks

KateFriedman-NOAA commented 4 days ago

Shuffled other current release branches around and recut release/gfs.v16.3.19 from dev/gfs.v16 for this ARFC.

KateFriedman-NOAA commented 4 days ago

@RussTreadon-NOAA sounds like this may or may not happen? Let me know the status when you do.

Also, let me know the new GSI tag name when/if created. I will prep release notes and be ready to update the GSI tag in the release branch when/if available. I'll ask you to review a PR into the release branch from my fork once ready.

FYI, we will be handing off a new GFS version that is a companion to the obsproc/v1.3 update that is being handed off in early October. Hopefully this issue can be decided on before we need to hand-off for that.

KateFriedman-NOAA commented 4 days ago

Additional update from @RussTreadon-NOAA in #2591 (https://github.com/NOAA-EMC/global-workflow/issues/2591#issuecomment-2371619577):

DA team members are meeting this afternoon to make a final decision as to what course of action to take. 
Submitting an ARFC to place PlanetiQ data in monitor mode is one option. Another option is to let the 
PlanetiQ data flow into and be assimilated by the GFS. This later option requires no EIB action.

This issue is opened as a "heads up" to EIB in case the DA team goes with the first option.

Thanks for the update @RussTreadon-NOAA ! Will see what the DA team decides and go from there.

RussTreadon-NOAA commented 4 days ago

DA team decided to move forward with ARFC to place PlanetiQ gpsro data in monitor mode. This is a one line change to global_convinfo.txt

diff -r /lfs/h1/ops/prod/packages/gfs.v16.3.18/fix/fix_gsi/global_convinfo.txt fix/global_convinfo.txt
260c260
<  gps      267    0    1     3.0      0      0      0  10.0  10.0   1.0  10.0  0.000000     0    0.     0.      0    0.     0.    0    0
---
>  gps      267    0   -1     3.0      0      0      0  10.0  10.0   1.0  10.0  0.000000     0    0.     0.      0    0.     0.    0    0

fix submodule updated in GSI branch release/gfsda.v16. Done at 7aabf37

RussTreadon-NOAA commented 4 days ago

GSI branch release/gfsda.v16 at 7aabf37 tagged as gfsda.v16.3.19

RussTreadon-NOAA commented 4 days ago

@KateFriedman-NOAA , let me know what you need from me and I'll work to get it to you as quickly as possible.

KateFriedman-NOAA commented 4 days ago

@KateFriedman-NOAA , let me know what you need from me and I'll work to get it to you as quickly as possible.

@RussTreadon-NOAA please review PR #2952. Once the release branch in the auth repo is ready I can cut a hand-off tag. Who is submitting the CDF?

RussTreadon-NOAA commented 4 days ago

I'm fine with submitting the ARFC but don't know what has been current practice. How have we been handling the CDF for recent implementations?

KateFriedman-NOAA commented 4 days ago

@RussTreadon-NOAA Tag cut for CDF: EMC-v16.3.19

I'm fine with submitting the ARFC but don't know what has been current practice. How have we been handling the CDF for recent implementations?

The last few upgrades were sudden or upstream so we didn't do CDFs and NCO manually made the new version installs with local edits. The upgrades before that had CDFs...either me or the component POC (e.g. Andrew for a few). I'd do a regular CDF since we're initiating it and choose the "Accelerated change" option. Let me know if I can assist with the CDF.

RussTreadon-NOAA commented 4 days ago

@KateFriedman-NOAA , I'm touching base with Daryl to see how he would like to proceed since we are in CWD.

RussTreadon-NOAA commented 4 days ago

As I wait for his reply (he is on travel), I am stepping through the Implementation Instructions in Release_Notes.md on my directory on Cactus. It's been a long time since I built a gfs.v16 package.

KateFriedman-NOAA commented 4 days ago

Sounds good @RussTreadon-NOAA, thanks! I have to step away from my desk now but will check back periodically. Let me know if you need anything else from me for this stage. I'll take care of the last six checklist items above once this goes into ops.

RussTreadon-NOAA commented 4 days ago

Thank you @KateFriedman-NOAA . I appreciate your quick action this afternoon. The ball is on my side of the court. I'll keep you in the loop as things progress.

RussTreadon-NOAA commented 3 days ago

@KateFriedman-NOAA: tag EMC-v16.3.19 has been installed on Cactus in /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/gfs.v16.3.19. A diff of directories in gfs.v16.3.19 and /lfs/h1/ops/prod/packages/gfs.v16.3.18 finds unexpected differences.

The unexpected diffs are related to WAFS.

ecf/

russ.treadon@clogin08:/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/gfs.v16.3.19> diff -r /lfs/h1/ops/prod/packages/gfs.v16.3.18/ecf ecf |head
Only in /lfs/h1/ops/prod/packages/gfs.v16.3.18/ecf/scripts/enkfgdas/post: jenkfgdas_post_f003.ecf
Only in /lfs/h1/ops/prod/packages/gfs.v16.3.18/ecf/scripts/enkfgdas/post: jenkfgdas_post_f004.ecf
Only in /lfs/h1/ops/prod/packages/gfs.v16.3.18/ecf/scripts/enkfgdas/post: jenkfgdas_post_f005.ecf
Only in /lfs/h1/ops/prod/packages/gfs.v16.3.18/ecf/scripts/enkfgdas/post: jenkfgdas_post_f006.ecf

jobs/

diff -r /lfs/h1/ops/prod/packages/gfs.v16.3.18/jobs/JGFS_ATMOS_WAFS_BLENDING_0P25 jobs/JGFS_ATMOS_WAFS_BLENDING_0P25
97c97
< export SLEEP_TIME=1500  
---
> export SLEEP_TIME=1500

The diff in JGFS_ATMOS_WAFS_BLENDING_0P25 is extra white space on the end of the SLEEP_TIME in the operational gfs.v16.3.18 file.

scripts/

diff -r /lfs/h1/ops/prod/packages/gfs.v16.3.18/scripts/exgfs_atmos_wafs_blending_0p25.sh scripts/exgfs_atmos_wafs_blending_0p25.sh
73c73
<           echo "UK WAFS GRIB2 file  $COMINuk/EGRR_WAFS_0p25_*_unblended_${PDY}_${cyc}z_t${ffhr}.grib2  not found" 
---
>           echo "UK WAFS GRIB2 file  $COMINuk/EGRR_WAFS_0p25_*_unblended_${PDY}_${cyc}z_t${ffhr}.grib2 not found"
311c311,313
<   cat $COMOUT/${RUN}.t${cyc}z.wafs_blend_0p25_usonly.emailbody | mail.py -s "$subject" $maillist -v
---
>   cat $COMOUT/${RUN}.t${cyc}z.wafs_blend_0p25_usonly.emailbody | mail.py -s "$subject" $maillis
> t -v
>

Does tag EMC-v16.3.19 contain changes from g-w issue #2591? Do we want WAFS changes in tag EMC-v16.3.19?

RussTreadon-NOAA commented 3 days ago

Also see difference in prepobs_run_ver when comparing operational gfs.v16.3.18 and tag EMC-v16.3.19

russ.treadon@clogin08:/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/gfs.v16.3.19> diff /lfs/h1/ops/prod/packages/gfs.v16.3.18/versions/ versions/
diff /lfs/h1/ops/prod/packages/gfs.v16.3.18/versions/hera.ver versions/hera.ver
6c6
< export prepobs_run_ver=1.0.1
---
> export prepobs_run_ver=1.1.0
diff /lfs/h1/ops/prod/packages/gfs.v16.3.18/versions/orion.ver versions/orion.ver
6c6
< export prepobs_run_ver=1.0.1
---
> export prepobs_run_ver=1.1.0
diff /lfs/h1/ops/prod/packages/gfs.v16.3.18/versions/run.ver versions/run.ver
1,2c1,2
< export version=v16.3.16
< export gfs_ver=v16.3.16
---
> export version=v16.3.19
> export gfs_ver=v16.3.19
Only in /lfs/h1/ops/prod/packages/gfs.v16.3.18/versions/: run.ver.para
diff /lfs/h1/ops/prod/packages/gfs.v16.3.18/versions/wcoss2.ver versions/wcoss2.ver
6c6
< export prepobs_run_ver=1.0.1
---
> export prepobs_run_ver=1.1.0

EMC-v16.3.19 sets prepobs_run_ver to 1.1.0 whereas operations has 1.0.1.

KateFriedman-NOAA commented 3 days ago

@RussTreadon-NOAA those are all expected differences due to how NCO made those installs with manual local updates.

Does tag EMC-v16.3.19 contain changes from g-w issue https://github.com/NOAA-EMC/global-workflow/issues/2591? Do we want WAFS changes in tag EMC-v16.3.19?

No, you're seeing the sudden update in ops early last week to adjust the wait time for UKMet data and update an email message in WAFS within the GFS package, resulting in v16.3.18. See https://github.com/NOAA-EMC/global-workflow/releases/tag/gfs.v16.3.18 and https://github.com/NOAA-EMC/WAFS/compare/gfs_wafs.v6.3.2...gfs_wafs.v6.3.3

Good catch, looks like we have an extra line return when we shouldn't though in the EMC WAFS copy but NCO has it correctly so that's what matters. We are going to remove WAFS from the GFS soon but I will fix that extra line return and recut the WAFS tag so if NCO does a full install it's ok.

EMC-v16.3.19 sets prepobs_run_ver to 1.1.0 whereas operations has 1.0.1.

Our prepobs_run_ver is only used in dev mode, obsproc sets and uses prepobs internally. You can see that we updated the version for our use but NCO didn't since the OPS GFS doesn't use that variable.

RussTreadon-NOAA commented 3 days ago

Thank you @KateFriedman-NOAA. Good to know that EMC-v16.3.19 differences noted above with respect to /lfs/h1/ops/prod/packages/gfs.v16.3.8 can be ignored.

KateFriedman-NOAA commented 3 days ago

@YaliMao-NOAA FYI I have fixed the email line that Russ noticed during a compare and recut the latest WAFS tag. I did it quickly so the WAFS tag is good in case this ARFC or following implementation pre-WAFS-removal does a full fresh install in ops. Apologies if I overstepped in the WAFS repo. I re-released the latest tag with the fix: https://github.com/NOAA-EMC/WAFS/releases/tag/gfs_wafs.v6.3.3

RussTreadon-NOAA commented 3 days ago

WCOSS2 (Cactus) tests

Use operational gfs.v16.3.8 and EMC-v16.3.19 in stand-alone GSI run script. Run the script four times

  1. use gfs.v16.3.18 for gdas 20240923 12Z with operational gpsro dump
  2. use EMC-v16.3.19 for gdas 20240923 12Z with operational gpsro dump
  3. use gfs.v16.3.18 for gdas 20240917 00Z with test gpsro dump containing PlanetiQ along with other operational gpsro data
  4. use gfs.v16.3.19 for gdas 20240917 00Z with test gpsro dump containing PlanetiQ along with other operational gpsro data

The analysis results from 1 and 2 are bitwise identical. This is expected because PlanetiQ gpsro data is not in the operational gpsro dump file,

The analysis results from 3 and 4 differ. This is expected because PlanetiQ gpsro data is in the test gpsro dump file. gfs.v16.3.18 assimilates PlanetiQ (type 267) gpsro data.

 o-g 01     gps asm 267 0000 count         24        779       1388       4965       7367       4774       3172       3807       4543       5821       8821      84281
 o-g 01     gps asm 267 0000  bias  0.133E+01  0.235E+00 -0.354E+00 -0.328E+00 -0.227E+00 -0.150E+00 -0.254E+00 -0.167E+00 -0.539E-01 -0.999E-01  0.337E-01 -0.833E-01
 o-g 01     gps asm 267 0000   rms  0.519E+01  0.471E+01  0.470E+01  0.465E+01  0.294E+01  0.149E+01  0.136E+01  0.138E+01  0.139E+01  0.155E+01  0.170E+01  0.220E+01
 o-g 01     gps asm 267 0000  cpen  0.714E+00  0.759E+00  0.855E+00  0.115E+01  0.126E+01  0.911E+00  0.837E+00  0.820E+00  0.816E+00  0.845E+00  0.884E+00  0.631E+00
 o-g 01     gps asm 267 0000 qcpen  0.714E+00  0.759E+00  0.855E+00  0.115E+01  0.126E+01  0.911E+00  0.837E+00  0.820E+00  0.816E+00  0.845E+00  0.884E+00  0.631E+00

whereas EMC-v16.3.19 monitors PlanetiQ (type 267) gpsro data

 o-g 01     gps mon 267 0000 count         24        779       1388       4965       7367       4774       3172       3807       4543       5821       8821      84281
 o-g 01     gps mon 267 0000  bias  0.133E+01  0.235E+00 -0.354E+00 -0.328E+00 -0.227E+00 -0.150E+00 -0.254E+00 -0.167E+00 -0.539E-01 -0.999E-01  0.337E-01 -0.833E-01
 o-g 01     gps mon 267 0000   rms  0.519E+01  0.471E+01  0.470E+01  0.465E+01  0.294E+01  0.149E+01  0.136E+01  0.138E+01  0.139E+01  0.155E+01  0.170E+01  0.220E+01
 o-g 01     gps mon 267 0000  cpen  0.714E+00  0.759E+00  0.855E+00  0.115E+01  0.126E+01  0.911E+00  0.837E+00  0.820E+00  0.816E+00  0.845E+00  0.884E+00  0.631E+00
 o-g 01     gps mon 267 0000 qcpen  0.714E+00  0.759E+00  0.855E+00  0.115E+01  0.126E+01  0.911E+00  0.837E+00  0.820E+00  0.816E+00  0.845E+00  0.884E+00  0.631E+00

EMC-v16.3.19 is the behavior we want in operations.

Tagging @XuanliLi-NOAA, @HaixiaLiu-NOAA , and @dtkleist for awareness.

KateFriedman-NOAA commented 2 days ago

@RussTreadon-NOAA checking in, has the CDF for this ARFC been submitted yet?

RussTreadon-NOAA commented 2 days ago

Sorry for not notifiying you, @KateFriedman-NOAA . Yes, the ARFC was submitted Tuesday, 9/24/2024, afternoon. NCO reached out to Daryl and me for more information. The ARFC has been assigned to Wei Wei. An implementation date has not yet been set but it could be implemented as early as Monday, 9/30/2024.

KateFriedman-NOAA commented 2 days ago

Sounds good, no worries, thanks @RussTreadon-NOAA! Please let me know if the implementation date changes, thanks!