search

NOAA-EMC / graphcast

GraphCastGFS
https://graphcastgfs.readthedocs.io/en/latest/index.html
Apache License 2.0
17 stars 6 forks source link

Added capability of using pygrib to extract variables #20

Closed LinlinCui-NOAA closed 5 months ago

LinlinCui-NOAA commented 5 months ago

Description

This PR provides updates on the gdas_utility.py script to process grib2 files using pygrib library which is much faster than wgrib2. We still maintained wgrib2 as an option for the user.

Linked Issues

Blocking Dependencies

Anticipated Changes

Input data

Needed libraries

LinlinCui-NOAA commented 5 months ago

Tested with 10 days data, pygrib (15 mins) is much faster than wgrib2 (37 mins). With wgrib2, merging grib2 files is a major bottleneck. 

wgrib2 took 37.61423015197118 mins to process data 2024010100 to 2024011100
pygrib took 15.084020833174387 mins to process data 2024-01-01 00:00:00 to 2024-01-11 00:00:00

The results (averaged over time and/or level) show consistency between two methods: averaged_2m_temperature averaged_10m_u_component_of_wind averaged_10m_v_component_of_wind averaged_geopotential averaged_mean_sea_level_pressure averaged_specific_humidity averaged_temperature averaged_total_precipitation_6hr averaged_u_component_of_wind averaged_v_component_of_wind averaged_vertical_velocity

SadeghTabas-NOAA commented 5 months ago

@LinlinCui-NOAA Thank you for the great job, that's good that we know pygrib is much faster. I merged a PR on EMC/graphcast. Please resolve the conflicts, then we will review it. Thanks

SadeghTabas-NOAA commented 5 months ago

@LinlinCui-NOAA Once you addressed my comments, please add a screenshot of a sample pygrib output opened in Python (or jupyter) using the xarray library, here.

LinlinCui-NOAA commented 5 months ago

When using 37 levels, pygrib only returns 31 levels (missing levels 125, 175, 225, 775, 825, 875):

Total number of pressure levels is 37
Start extracting variables and associated levels from grib2 files:

Get variable 2t from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
Get variable prmsl from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
Get variable 10u from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
Get variable 10v from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
Get variable w from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
99:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 100 Pa:fcst time 0 hrs:from 202401090000
109:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 200 Pa:fcst time 0 hrs:from 202401090000
119:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 300 Pa:fcst time 0 hrs:from 202401090000
129:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 500 Pa:fcst time 0 hrs:from 202401090000
139:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 700 Pa:fcst time 0 hrs:from 202401090000
149:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 1000 Pa:fcst time 0 hrs:from 202401090000
169:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 2000 Pa:fcst time 0 hrs:from 202401090000
179:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 3000 Pa:fcst time 0 hrs:from 202401090000
200:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 5000 Pa:fcst time 0 hrs:from 202401090000
215:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 7000 Pa:fcst time 0 hrs:from 202401090000
226:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 10000 Pa:fcst time 0 hrs:from 202401090000
242:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 15000 Pa:fcst time 0 hrs:from 202401090000
258:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 20000 Pa:fcst time 0 hrs:from 202401090000
274:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 25000 Pa:fcst time 0 hrs:from 202401090000
290:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 30000 Pa:fcst time 0 hrs:from 202401090000
306:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 35000 Pa:fcst time 0 hrs:from 202401090000
322:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 40000 Pa:fcst time 0 hrs:from 202401090000
338:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 45000 Pa:fcst time 0 hrs:from 202401090000
354:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 50000 Pa:fcst time 0 hrs:from 202401090000
370:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 55000 Pa:fcst time 0 hrs:from 202401090000
386:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 60000 Pa:fcst time 0 hrs:from 202401090000
402:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 65000 Pa:fcst time 0 hrs:from 202401090000
418:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 70000 Pa:fcst time 0 hrs:from 202401090000
434:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 75000 Pa:fcst time 0 hrs:from 202401090000
450:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 80000 Pa:fcst time 0 hrs:from 202401090000
466:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 85000 Pa:fcst time 0 hrs:from 202401090000
482:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 90000 Pa:fcst time 0 hrs:from 202401090000
498:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 92500 Pa:fcst time 0 hrs:from 202401090000
514:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 95000 Pa:fcst time 0 hrs:from 202401090000
531:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 97500 Pa:fcst time 0 hrs:from 202401090000
546:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 100000 Pa:fcst time 0 hrs:from 202401090000
SadeghTabas-NOAA commented 5 months ago

When using 37 levels, pygrib only returns 31 levels (missing levels 125, 175, 225, 775, 825, 875):

Total number of pressure levels is 37
Start extracting variables and associated levels from grib2 files:

Get variable 2t from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
Get variable prmsl from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
Get variable 10u from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
Get variable 10v from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
Get variable w from file 1 ./noaa-gfs-bdp-pds-data/20240109/00/gdas.t00z.pgrb2.0p25.f000:
99:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 100 Pa:fcst time 0 hrs:from 202401090000
109:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 200 Pa:fcst time 0 hrs:from 202401090000
119:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 300 Pa:fcst time 0 hrs:from 202401090000
129:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 500 Pa:fcst time 0 hrs:from 202401090000
139:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 700 Pa:fcst time 0 hrs:from 202401090000
149:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 1000 Pa:fcst time 0 hrs:from 202401090000
169:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 2000 Pa:fcst time 0 hrs:from 202401090000
179:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 3000 Pa:fcst time 0 hrs:from 202401090000
200:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 5000 Pa:fcst time 0 hrs:from 202401090000
215:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 7000 Pa:fcst time 0 hrs:from 202401090000
226:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 10000 Pa:fcst time 0 hrs:from 202401090000
242:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 15000 Pa:fcst time 0 hrs:from 202401090000
258:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 20000 Pa:fcst time 0 hrs:from 202401090000
274:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 25000 Pa:fcst time 0 hrs:from 202401090000
290:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 30000 Pa:fcst time 0 hrs:from 202401090000
306:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 35000 Pa:fcst time 0 hrs:from 202401090000
322:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 40000 Pa:fcst time 0 hrs:from 202401090000
338:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 45000 Pa:fcst time 0 hrs:from 202401090000
354:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 50000 Pa:fcst time 0 hrs:from 202401090000
370:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 55000 Pa:fcst time 0 hrs:from 202401090000
386:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 60000 Pa:fcst time 0 hrs:from 202401090000
402:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 65000 Pa:fcst time 0 hrs:from 202401090000
418:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 70000 Pa:fcst time 0 hrs:from 202401090000
434:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 75000 Pa:fcst time 0 hrs:from 202401090000
450:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 80000 Pa:fcst time 0 hrs:from 202401090000
466:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 85000 Pa:fcst time 0 hrs:from 202401090000
482:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 90000 Pa:fcst time 0 hrs:from 202401090000
498:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 92500 Pa:fcst time 0 hrs:from 202401090000
514:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 95000 Pa:fcst time 0 hrs:from 202401090000
531:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 97500 Pa:fcst time 0 hrs:from 202401090000
546:Vertical velocity:Pa s**-1 (instant):regular_ll:isobaricInhPa:level 100000 Pa:fcst time 0 hrs:from 202401090000

I know that, it's because the grib2 files currently do not account for some of the pressure levels included in 37 PLs. so for now just add the ability to extract, we may update the grib2 files in the future.

LinlinCui-NOAA commented 5 months ago

Okay, I see. I realized that wgrib2 did not throw an error when a level is not available. But pygrib did.

SadeghTabas-NOAA commented 5 months ago

Okay, I see. I realized that wgrib2 did not throw an error when a level is not available. But pygrib did.

yes, it does not give error but generates negative levels which are meaningless.

LinlinCui-NOAA commented 5 months ago

@SadeghTabas-NOAA Please review new commits. Here are screenshots of output nc file from pygrib:

s1 s2

SadeghTabas-NOAA commented 5 months ago

Could you please address and resolve my other comments as well? Thanks

SadeghTabas-NOAA commented 5 months ago

@LinlinCui-NOAA I did compare two nc files from pygrib and wgrib2:

(mlwp) [Sadegh.Tabas@hfe02 NCEP]$ nccmp -d source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,54] : VALUES : 251.71 <> 0

let me check if this is the only one

SadeghTabas-NOAA commented 5 months ago

looks like there are others as well: (mlwp) [Sadegh.Tabas@hfe02 NCEP]$ nccmp -d -v geopotential source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : geopotential : POSITION : [0,0,0,0,1237] : VALUES : 197977 <> 0

(mlwp) [Sadegh.Tabas@hfe02 NCEP]$ nccmp -d -x 2m_temperature source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : mean_sea_level_pressure : POSITION : [0,0,1,51] : VALUES : 99257.4 <> 0

@junwang-noaa What do you suggest on this? I was wondering if we can ignore this? Thanks

SadeghTabas-NOAA commented 5 months ago

Could you please address and resolve my other comments as well? Thanks

@LinlinCui-NOAA Thanks for the updates

LinlinCui-NOAA commented 5 months ago

looks like there are others as well: (mlwp) [Sadegh.Tabas@hfe02 NCEP]$ nccmp -d -v geopotential source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : geopotential : POSITION : [0,0,0,0,1237] : VALUES : 197977 <> 0

(mlwp) [Sadegh.Tabas@hfe02 NCEP]$ nccmp -d -x 2m_temperature source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : mean_sea_level_pressure : POSITION : [0,0,1,51] : VALUES : 99257.4 <> 0

The first value is from pygrib and the second one from wgrib2, correct?

SadeghTabas-NOAA commented 5 months ago

looks like there are others as well: (mlwp) [Sadegh.Tabas@hfe02 NCEP]$ nccmp -d -v geopotential source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : geopotential : POSITION : [0,0,0,0,1237] : VALUES : 197977 <> 0 (mlwp) [Sadegh.Tabas@hfe02 NCEP]$ nccmp -d -x 2m_temperature source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : mean_sea_level_pressure : POSITION : [0,0,1,51] : VALUES : 99257.4 <> 0

The first value is from pygrib and the second one from wgrib2, correct?

Yes, its correct.

SadeghTabas-NOAA commented 5 months ago

looks like there are others as well: (mlwp) [Sadegh.Tabas@hfe02 NCEP]$ nccmp -d -v geopotential source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : geopotential : POSITION : [0,0,0,0,1237] : VALUES : 197977 <> 0

(mlwp) [Sadegh.Tabas@hfe02 NCEP]$ nccmp -d -x 2m_temperature source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : mean_sea_level_pressure : POSITION : [0,0,1,51] : VALUES : 99257.4 <> 0

@junwang-noaa What do you suggest on this? I was wondering if we can ignore this? Thanks

looks like there is lots of diffs: nccmp -d -f source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,54] : VALUES : 251.71 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,55] : VALUES : 251.71 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,56] : VALUES : 251.7 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,57] : VALUES : 251.7 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,58] : VALUES : 251.7 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,59] : VALUES : 251.7 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,60] : VALUES : 251.69 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,61] : VALUES : 251.69 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,62] : VALUES : 251.69 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,63] : VALUES : 251.68 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,64] : VALUES : 251.68 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,65] : VALUES : 251.68 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,66] : VALUES : 251.68 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,67] : VALUES : 251.68 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,68] : VALUES : 251.67 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,69] : VALUES : 251.67 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,70] : VALUES : 251.67 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,71] : VALUES : 251.67 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,72] : VALUES : 251.67 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,73] : VALUES : 251.66 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,74] : VALUES : 251.66 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,75] : VALUES : 251.66 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,76] : VALUES : 251.66 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,77] : VALUES : 251.66 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,78] : VALUES : 251.65 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,79] : VALUES : 251.65 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,80] : VALUES : 251.65 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,81] : VALUES : 251.65 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,82] : VALUES : 251.64 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,83] : VALUES : 251.64 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,84] : VALUES : 251.64 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,85] : VALUES : 251.63 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,86] : VALUES : 251.63 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,87] : VALUES : 251.63 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,88] : VALUES : 251.63 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,89] : VALUES : 251.62 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,90] : VALUES : 251.62 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,91] : VALUES : 251.62 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,92] : VALUES : 251.62 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,93] : VALUES : 251.62 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,94] : VALUES : 251.62 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,95] : VALUES : 251.61 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,96] : VALUES : 251.61 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,97] : VALUES : 251.61 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,98] : VALUES : 251.6 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,99] : VALUES : 251.6 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,100] : VALUES : 251.6 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,101] : VALUES : 251.6 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,102] : VALUES : 251.59 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,103] : VALUES : 251.59 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,104] : VALUES : 251.59 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,105] : VALUES : 251.59 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,106] : VALUES : 251.59 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,107] : VALUES : 251.59 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,108] : VALUES : 251.58 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,109] : VALUES : 251.58 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,110] : VALUES : 251.58 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,111] : VALUES : 251.58 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,112] : VALUES : 251.58 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,113] : VALUES : 251.58 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,114] : VALUES : 251.57 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,115] : VALUES : 251.57 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,116] : VALUES : 251.57 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,117] : VALUES : 251.56 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,118] : VALUES : 251.56 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,119] : VALUES : 251.56 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,120] : VALUES : 251.56 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,121] : VALUES : 251.56 <> 0 DIFFER : VARIABLE : 2m_temperature : POSITION : [0,0,1,122] : VALUES : 251.55 <> 0

@junwang-noaa I'm not sure why these are different, to have the complete difference list on hera please run the following command: cd /home/Sadegh.Tabas/pygrib/graphcast/NCEP/

nccmp -d -f source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc

LinlinCui-NOAA commented 5 months ago

@SadeghTabas-NOAA Seems something wrong with wgrib2 generated file: image

Maximum values for temperature and specific humidity are 20194, which is wrong.

junwang-noaa commented 5 months ago

@SadeghTabas-NOAA @LinlinCui-NOAA I saw this in the file header:

    float \2m_temperature(batch, time, lat, lon) ;
            \2m_temperature:_FillValue = NaNf ;
            \2m_temperature:short_name = "TMP_2maboveground" ;
            \2m_temperature:long_name = "Temperature" ;
            \2m_temperature:level = "2 m above ground" ;
            \2m_temperature:units = "K" ;
            \2m_temperature:coordinates = "datetime" ;
    float mean_sea_level_pressure(batch, time, lat, lon) ;
            mean_sea_level_pressure:_FillValue = NaNf ;
            mean_sea_level_pressure:short_name = "PRMSL_meansealevel" ;
            mean_sea_level_pressure:long_name = "Pressure Reduced to MSL" ;
            mean_sea_level_pressure:level = "mean sea level" ;
            mean_sea_level_pressure:units = "Pa" ;
            mean_sea_level_pressure:coordinates = "datetime" ;
    float \10m_u_component_of_wind(batch, time, lat, lon) ;
            \10m_u_component_of_wind:_FillValue = NaNf ;
            \10m_u_component_of_wind:short_name = "UGRD_10maboveground" ;
            \10m_u_component_of_wind:long_name = "U-Component of Wind" ;
            \10m_u_component_of_wind:level = "10 m above ground" ;
            \10m_u_component_of_wind:units = "m/s" ;
            \10m_u_component_of_wind:coordinates = "datetime" ;
    float \10m_v_component_of_wind(batch, time, lat, lon) ;
            \10m_v_component_of_wind:_FillValue = NaNf ;
            \10m_v_component_of_wind:short_name = "VGRD_10maboveground" ;
            \10m_v_component_of_wind:long_name = "V-Component of Wind" ;
            \10m_v_component_of_wind:level = "10 m above ground" ;
            \10m_v_component_of_wind:units = "m/s" ;
            \10m_v_component_of_wind:coordinates = "datetime" ;

Is this expected?

LinlinCui-NOAA commented 5 months ago

@SadeghTabas-NOAA @junwang-noaa Is that because of version of wgrib? I saw on hera it is wgrib/1.8.1.0b. On Orion, there are three different versions: wgrib/1.8.0b wgrib/2.0.8 wgrib2/3.0.2

SadeghTabas-NOAA commented 5 months ago

@SadeghTabas-NOAA Seems something wrong with wgrib2 generated file: image

Maximum values for temperature and specific humidity are 20194, which is wrong.

@LinlinCui-NOAA thank you for finding this. I don’t think that is because of wgrib2 version, I guess there is a bug in the merging process. Give me some time I’m currently working on it. I’ll update you. Thanks

SadeghTabas-NOAA commented 5 months ago

@SadeghTabas-NOAA @LinlinCui-NOAA I saw this in the file header:

    float \2m_temperature(batch, time, lat, lon) ;
            \2m_temperature:_FillValue = NaNf ;
            \2m_temperature:short_name = "TMP_2maboveground" ;
            \2m_temperature:long_name = "Temperature" ;
            \2m_temperature:level = "2 m above ground" ;
            \2m_temperature:units = "K" ;
            \2m_temperature:coordinates = "datetime" ;
    float mean_sea_level_pressure(batch, time, lat, lon) ;
            mean_sea_level_pressure:_FillValue = NaNf ;
            mean_sea_level_pressure:short_name = "PRMSL_meansealevel" ;
            mean_sea_level_pressure:long_name = "Pressure Reduced to MSL" ;
            mean_sea_level_pressure:level = "mean sea level" ;
            mean_sea_level_pressure:units = "Pa" ;
            mean_sea_level_pressure:coordinates = "datetime" ;
    float \10m_u_component_of_wind(batch, time, lat, lon) ;
            \10m_u_component_of_wind:_FillValue = NaNf ;
            \10m_u_component_of_wind:short_name = "UGRD_10maboveground" ;
            \10m_u_component_of_wind:long_name = "U-Component of Wind" ;
            \10m_u_component_of_wind:level = "10 m above ground" ;
            \10m_u_component_of_wind:units = "m/s" ;
            \10m_u_component_of_wind:coordinates = "datetime" ;
    float \10m_v_component_of_wind(batch, time, lat, lon) ;
            \10m_v_component_of_wind:_FillValue = NaNf ;
            \10m_v_component_of_wind:short_name = "VGRD_10maboveground" ;
            \10m_v_component_of_wind:long_name = "V-Component of Wind" ;
            \10m_v_component_of_wind:level = "10 m above ground" ;
            \10m_v_component_of_wind:units = "m/s" ;
            \10m_v_component_of_wind:coordinates = "datetime" ;

Is this expected?

yes, it looks fine to me.

SadeghTabas-NOAA commented 5 months ago

@SadeghTabas-NOAA Seems something wrong with wgrib2 generated file: image Maximum values for temperature and specific humidity are 20194, which is wrong.

@LinlinCui-NOAA thank you for finding this. I don’t think that is because of wgrib2 version, I guess there is a bug in the merging process. Give me some time I’m currently working on it. I’ll update you. Thanks

@LinlinCui-NOAA I've updated the code and resolved the bug (it will be merged by my next PR), now most of the differences are resolved. The only issue is the total_precipitation_6hr. you need to divide it by 1000 for unit conversion (from (kg/m^2) to (m) --> by dividing it by 1000kg/m³). please look at the wgrib2 part of the code.

DIFFER : VARIABLE : total_precipitation_6hr : POSITION : [0,1,115,101] : VALUES : 0.5 <> 0.0005

I will approve it as soon as you update this. Thanks

P.S. in case you want to check yourself, please run the following on hera:

cd /home/Sadegh.Tabas/pygrib/graphcast/NCEP/

nccmp -d -f source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc

SadeghTabas-NOAA commented 5 months ago

@SadeghTabas-NOAA Seems something wrong with wgrib2 generated file: image Maximum values for temperature and specific humidity are 20194, which is wrong.

@LinlinCui-NOAA thank you for finding this. I don’t think that is because of wgrib2 version, I guess there is a bug in the merging process. Give me some time I’m currently working on it. I’ll update you. Thanks

@LinlinCui-NOAA I've updated the code and resolved the bug (it will be merged by my next PR), now most of the differences are resolved. The only issue is the total_precipitation_6hr. you need to divide it by 1000 for unit conversion (from (kg/m^2) to (m) --> by dividing it by 1000kg/m³). please look at the wgrib2 part of the code.

DIFFER : VARIABLE : total_precipitation_6hr : POSITION : [0,1,115,101] : VALUES : 0.5 <> 0.0005

I will approve it as soon as you update this. Thanks

P.S. in case you want to check yourself, please run the following on hera:

cd /home/Sadegh.Tabas/pygrib/graphcast/NCEP/

nccmp -d -f source-gdas_date-2022010106_res-0.25_levels-13_steps-0.nc wgrib2/source-gdas_date-2022010106_res-0.25_levels-13_steps-2.nc

Please also update the level dtype to int32, currently it is int64. Also, the time variable should show time deltas not datetime, that needs to be updated as well. Thanks