Replace satellite_sensor based UFSDA radiance bias correction files with tarball

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)

https://global-workflow.readthedocs.io/en/latest

GNU Lesser General Public License v3.0

75 stars 168 forks source link

Replace satellite_sensor based UFSDA radiance bias correction files with tarball #2869

Closed RussTreadon-NOAA closed 1 month ago

RussTreadon-NOAA commented 2 months ago

Target fix directory

NEW

Type of change

[ ] Update existing files
[ ] Remove files
[X] Add new files to existing directory
[ ] Add new fix directory

Any workflow changes needed?

No change needed

Related issues

NOAA-EMC g-w #2862 NOAA-EMC GDASApp #1264

Pickup location

Hera: /scratch1/NCEPDEV/stmp2/Russ.Treadon/COMROOT/prjedi/gdas.20240223/18/analysis/atmos

Size

0.1 Mb

RussTreadon-NOAA commented 2 months ago

This issue is opened to request that UFSDA radiance bias correction file gdas.t18z.radbcor be copied on Hera from

/scratch1/NCEPDEV/stmp2/Russ.Treadon/COMROOT/prjedi/gdas.20240223/18/analysis/atmos

/scratch1/NCEPDEV/global/glopara/data/ICSDIR/C96C48/gdas.20240223/18/analysis/atmos

Once gdas.t18z.radbcor is in the Hera CI ICSDIR_ROOT, we need to rsync file gdas.t18z.radbcor to $ICSDIR_ROOT/C96C48/gdas.20240223/18/analysis/atmos on other supported platforms.

The gdas.t18z.radbcor tarball needs to be in place before a PR can be opened for g-w issue #2862. Changes in the PR associated with g-w issue #2862 will assume that file gdas.t18z.radbcor exists in the g-w CI ICSROOT_DIR on various platforms.

RussTreadon-NOAA commented 2 months ago

@KateFriedman-NOAA and @WalterKolczynski-NOAA , I am a member of both WCOSS2 emc.global and Hera role.glopara. With your permission I can log into either / both accounts and create the gdas.t18z.radbcor tarball. May I do so?

RussTreadon-NOAA commented 2 months ago

While the offer to help create gdas.t18z.radbcor on WCOSS2 and Hera still stands, I copied the glopara ICSDIR_ROOT to my space, created the tarball, and am now testing changes for g-w issue #2862.

RussTreadon-NOAA commented 2 months ago

Repeat comment from g-w PR #2875

@KateFriedman-NOAA and @WalterKolczynski-NOAA , please pause any work planned for g-w issue https://github.com/NOAA-EMC/global-workflow/issues/2869. The DA team has not reached consensus on the name to use for the radiance bias correction tarball.

RussTreadon-NOAA commented 2 months ago

Based on conversation in g-w PR #2875, rename gdas.t18z.radbcor as gdas.t18z.rad_varbc_params.tar.

The renamed file can be picked up from Hera /scratch1/NCEPDEV/global/Russ.Treadon/data/ICSDIR/C96C48/20240610/gdas.20240223/18/analysis/atmos

Hera(hfe12):/scratch1/NCEPDEV/global/Russ.Treadon/data/ICSDIR/C96C48/20240610/gdas.20240223/18/analysis/atmos$ ls -l gdas.t18z.rad_varbc_params.tar
-rw-r--r-- 1 Russ.Treadon global 102400 Aug 28 15:40 gdas.t18z.rad_varbc_params.tar

Hera(hfe12):/scratch1/NCEPDEV/global/Russ.Treadon/data/ICSDIR/C96C48/20240610/gdas.20240223/18/analysis/atmos$ tar -tvf gdas.t18z.rad_varbc_params.tar
-rw-r--r-- Russ.Treadon/global 22260 2024-08-05 18:08 gdas.t18z.atms_n20.satbias.nc
-rw-r--r-- Russ.Treadon/global 22260 2024-08-05 18:08 gdas.t18z.atms_n20.satbias_cov.nc
-rw-r--r-- Russ.Treadon/global   547 2024-08-05 18:08 gdas.t18z.atms_n20.tlapse.txt
-rw-r--r-- Russ.Treadon/global 22260 2024-08-05 18:08 gdas.t18z.atms_npp.satbias.nc
-rw-r--r-- Russ.Treadon/global 22260 2024-08-05 18:08 gdas.t18z.atms_npp.satbias_cov.nc
-rw-r--r-- Russ.Treadon/global   547 2024-08-05 18:08 gdas.t18z.atms_npp.tlapse.txt

KateFriedman-NOAA commented 2 months ago

@RussTreadon-NOAA I have copied the new tarball into ICSDIR on Hera:

[role.glopara@hfe12 ICSDIR]$ pwd
/scratch1/NCEPDEV/global/glopara/data/ICSDIR
[role.glopara@hfe12 ICSDIR]$ ll C96C48/20240610/gdas.20240223/18/analysis/atmos/
total 82520
-rw-r--r-- 1 role.glopara global 84291229 Mar  5 12:13 gdas.t18z.atminc.nc
-rw-r--r-- 1 role.glopara global    22260 Aug  5 18:08 gdas.t18z.atms_n20.satbias.nc
-rw-r--r-- 1 role.glopara global    22260 Aug  5 18:08 gdas.t18z.atms_n20.satbias_cov.nc
-rw-r--r-- 1 role.glopara global      547 Aug  5 18:08 gdas.t18z.atms_n20.tlapse.txt
-rw-r--r-- 1 role.glopara global    22260 Aug  5 18:08 gdas.t18z.atms_npp.satbias.nc
-rw-r--r-- 1 role.glopara global    22260 Aug  5 18:08 gdas.t18z.atms_npp.satbias_cov.nc
-rw-r--r-- 1 role.glopara global      547 Aug  5 18:08 gdas.t18z.atms_npp.tlapse.txt
-rw-r--r-- 1 role.glopara global   102400 Aug 28 15:40 gdas.t18z.rad_varbc_params.tar

Please confirm it's as requested and we can sync it to the other platforms. There will be other new ICs going into ICSDIR today so I may wait to rsync until those have been pulled in.

RussTreadon-NOAA commented 2 months ago

Thank you @KateFriedman-NOAA for copying the new JEDI radiance bias correction tarball to the ICSDIR_ROOT for C96C48_ufs_hybatmDA CI. I set up and triggered C96C48_ufs_hybatmDA CI in my workspace on Hera. The 20240223 18 gdasstage_ic correctly copied gdas.t18z.rad_varbc_params.tar to my COMROOT.

We can certainly wait until latter to rsync file gdas.t18z.rad_varbc_params.tar along with other new IC's to other platforms. I'm not in a rush. I can continue development and testing on Hera.

KateFriedman-NOAA commented 2 months ago

@RussTreadon-NOAA I have synced this tarball to WCOSS2 and Jet. @WalterKolczynski-NOAA please sync the ../ICSDIR/C96C48 set to MSU, thanks!

RussTreadon-NOAA commented 1 month ago

@KateFriedman-NOAA , g-w PR #2875 has successfully used the new radiance bias correction tarball via g-w CI _C96C48_ufshybatmDA on Hera, Hercules, and WCOSS2 (Dogwood). This issue may be closed.

KateFriedman-NOAA commented 1 month ago

Thanks @RussTreadon-NOAA !