search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
75 stars 168 forks source link

GFSv16.3.20 - OBSPROCv1.3 and updated USAF snow filename #2913

Open KateFriedman-NOAA opened 2 months ago

KateFriedman-NOAA commented 2 months ago

Description

The snow data file provided by the USAF has changed names and the new name will go into affect with obsproc.v1.3. The GFS will need to update to use the new obsproc.v1.3 and the new snow filename. Additionally, the GSI will require a change in the GSI code (read_prepbufr.f) for the saildrone data.

The prepobs package is also updating to v1.2.0 with obsproc/v1.3.0. This version is internal to the obsproc package but we'll want to update both obsproc and prepobs versions in develop for GFSv17. Will handle that separately.

FYI @ilianagenkova @GeorgeGayno-NOAA @ADCollard

Update 10/21/24: NCO pulled the new CO2 fix files for 2025 into the gfs.v16.3.20 para install. Part of issue #2558 will be resolved via this issue. Update 11/5/24: PlanetIQ GPSRO data is now switched on for active assimilation (https://github.com/NOAA-EMC/GSI/pull/803). The gfsda.v16.3.20 tag was updated to reflect this change.

Target version

v16.3.20

Expected workflow changes

The updated UFS_UTILS release tag is being worked under: Issue #973

The test branch is here: release/ops-gfsv16.3.1

Will need to coordinate to pass in the global afwa file from the JJOB to ./scripts/exemcsfc_global_sfc_prep.sh

KateFriedman-NOAA commented 2 months ago

Release branch for this created: https://github.com/NOAA-EMC/global-workflow/tree/release/gfs.v16.3_obsproc_snow

Will rename release branch to version once the version is known.

GeorgeGayno-NOAA commented 2 months ago

The new snow file will need to be exported from here. https://github.com/NOAA-EMC/global-workflow/blob/release/gfs.v16.3.20/jobs/JGLOBAL_ATMOS_EMCSFC_SFC_PREP#L67

I have been using AFWA_GLOBAL_FILE. The AFWA_NH_FILE and AFWA_SH_FILE variables can be removed.

How can we test this?

KateFriedman-NOAA commented 2 months ago

How can we test this?

@ilianagenkova Do you have a para COM for obsproc.v1.3 that we could point to for testing this?

ilianagenkova commented 2 months ago

On Dogwood: /lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3/g//atmos/usaf

-rw-r--r-- 1 iliana.genkova emc 42315832 Sep 12 05:40 gdas.20240912/00/atmos/gdas.t00z.snow.usaf.grib2 -rw-r--r-- 1 iliana.genkova emc 42315832 Sep 12 11:40 gdas.20240912/06/atmos/gdas.t06z.snow.usaf.grib2 -rw-r--r-- 1 iliana.genkova emc 42315832 Sep 12 17:40 gdas.20240912/12/atmos/gdas.t12z.snow.usaf.grib2 -rw-r--r-- 1 iliana.genkova emc 42315832 Sep 12 23:40 gdas.20240912/18/atmos/gdas.t18z.snow.usaf.grib2

KateFriedman-NOAA commented 1 month ago

Branch renamed and recut based on release/gfs.v16.3.19 state: https://github.com/NOAA-EMC/global-workflow/tree/release/gfs.v16.3.20

KateFriedman-NOAA commented 1 month ago

On Dogwood: /lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3/g//atmos/usaf

@ilianagenkova Is this available on Cactus as well?

KateFriedman-NOAA commented 1 month ago

@ilianagenkova Do you have a copy of the new obsproc release/obsproc.v1.3.0 branch on Cactus that I can point to for testing the GFS? Or shall I build my own copy for testing?

KateFriedman-NOAA commented 1 month ago

/lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3/g//atmos/usaf

Nevermind, I see it now, thanks!

@ilianagenkova Do you have a copy of the new obsproc release/obsproc.v1.3.0 branch on Cactus that I can point to for testing the GFS? Or shall I build my own copy for testing?

I've gone ahead and installed copies of both the obsproc and prepobs release branches for this upgrade.

I will use the provided COM and installs to test the GFS on Cactus.

KateFriedman-NOAA commented 1 month ago

@ilianagenkova I ran the GFSv16 workflow with the new obsproc/v1.3, prepobs/v1.2, USAF snow file from your COM, and UFS_UTILS release branch with snow filename updates. I ran the emcsfc_sfc_prep job that produces the GFS snogrb files and which uses the new USAF snow filename. I then ran the prep job to create a prepbufr file using the generated snogrb files, new obsproc, and new prepobs. @GeorgeGayno-NOAA has checked the emcsfc_sfc_prep job and confirmed it ran successfully.

Please take a look at the following log and output on WCOSS2-Cactus and let me know if the prep job ran as expected. Thanks!

output: /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/gdas.20240924/00/atmos/
log: /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/logs/2024092400/gdasprep.log
ilianagenkova commented 1 month ago

@KateFriedman-NOAA , thanks for testing the processing end-to-end. The prep job output looks good! 

[clogin06 /lfs/h2/emc/ptmp/ashley.stanfield/CRON]$ ll /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/gdas.20240924/00/atmos/*prepbufr*
-rw-r----- 1 kate.friedman rstprod 115100896 Sep 24 18:01 /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/gdas.20240924/00/atmos/gdas.t00z.prepbufr
-rw-r----- 1 kate.friedman rstprod  21224440 Sep 24 18:01 /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/gdas.20240924/00/atmos/gdas.t00z.prepbufr.acft_profiles
lrwxrwxrwx 1 kate.friedman emc           102 Sep 24 17:51 lrwxrwxrwx 1 kate.friedman emc           106 Sep 24 17:51 
[clogin06 /lfs/h2/emc/stmp/iliana.genkova]$ ll  CRON/R13/com/obsproc/v1.3//gdas.20240924/00/atmos/*prepbufr*                                                                                                                                                                            
-rw-r----- 1 iliana.genkova rstprod 115100896 Sep 24 06:26 CRON/R13/com/obsproc/v1.3//gdas.20240924/00/atmos/gdas.t00z.prepbufr
-rw-r----- 1 iliana.genkova rstprod  21224440 Sep 24 06:26 CRON/R13/com/obsproc/v1.3//gdas.20240924/00/atmos/gdas.t00z.prepbufr.acft_profiles
KateFriedman-NOAA commented 1 month ago

@ilianagenkova Excellent! I see I even reproduced your prepbufr file:

kate.friedman@clogin01:/lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13> cmp gdas.20240924/00/atmos/gdas.t00z.prepbufr /lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3//gdas.20240924/00/atmos/gdas.t00z.prepbufr
kate.friedman@clogin01:/lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13>

Thanks for checking and confirming!

KateFriedman-NOAA commented 1 month ago

@GeorgeGayno-NOAA I think we're ready for a UFS_UTILS tag to include in the release branch for this update.

GeorgeGayno-NOAA commented 1 month ago

@GeorgeGayno-NOAA I think we're ready for a UFS_UTILS tag to include in the release branch for this update.

See: https://github.com/ufs-community/UFS_UTILS/releases/tag/ops-gfsv16.3.20

KateFriedman-NOAA commented 1 month ago

Once the ARFC in issue #2951 goes in I will sync those updates into the release branch for this.

Only remaining question is whether a GSI update to handle saildrone data in the new obsproc/v1.3 should be included in this or possibly held for the upcoming GFSv16.4 DA upgrade. @RussTreadon-NOAA @ilianagenkova do we know what the status of that change is?

KateFriedman-NOAA commented 1 month ago

Synced dev/gfs.v16 into the release/gfs.v16.3.20 branch and reverted the commit to remove WAFS (this branch was originally cut from the WAFS removal branch).

KateFriedman-NOAA commented 1 month ago

@ADCollard I added you as an assignee as a formality since the GSI is updating for this version.

KateFriedman-NOAA commented 1 month ago

Hand-off tag EMC-v16.3.20 has been cut and the CDF for GFSv16.3.20 has been submitted.

ADCollard commented 1 week ago

PlanetIQ GPSRO data is now switched on for active assimilation (GSI PR #803). The gfsda.v16.3.20 tag was updated to reflect this change.

KateFriedman-NOAA commented 1 week ago

Recut hand-off tag (EMC-v16.3.20) after merging PR #3070.

KateFriedman-NOAA commented 1 week ago

Recut hand-off tag (EMC-v16.3.20) after merging PR #3072.