search

jswhit / pygrib

Python interface for reading and writing GRIB data
https://jswhit.github.io/pygrib
MIT License
327 stars 97 forks source link

Problems reading NGAC grib files #89

Open bbakernoaa opened 6 years ago

bbakernoaa commented 6 years ago

I'm trying to read the NGAC grib files http://www.ftp.ncep.noaa.gov/data/nccf/com/ngac/prod/ngac.20180918/

It seems that the way that NCEP is storing the data is a little incompatable with pygrib. Is it an issue of using the correct grib table to define these? Can the eccodes backend handle that? 

In [1]: import pygrib

In [2]: f = pygrib.open('ngac.t00z.a2df00.grib2')

In [3]: f.read()
Out[3]: 
[1:104:104 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 2:103:103 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 3:102:102 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 4:102:102 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 5:102:102 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 6:102:102 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 7:102:102 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 8:102:102 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 9:Mass mixing ratio:mass fraction in air (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 10:11:11 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 11:12:12 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 12:Surface dry deposition mass flux into stomata   kg.m-2.s-1:unknown (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 13:10:10 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 14:Mass mixing ratio:mass fraction in air (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 15:11:11 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 16:12:12 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 17:Surface dry deposition mass flux into stomata   kg.m-2.s-1:unknown (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 18:10:10 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 19:Mass mixing ratio:mass fraction in air (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 20:11:11 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 21:12:12 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 22:Surface dry deposition mass flux into stomata   kg.m-2.s-1:unknown (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 23:10:10 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 24:Mass mixing ratio:mass fraction in air (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 25:11:11 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 26:12:12 (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 27:Surface dry deposition mass flux into stomata   kg.m-2.s-1:unknown (instant):regular_ll:unknown:level 0:fcst time 0 hrs:from 201809160000,
 28:10:10 (instant):regular_ll:unknown:level 0:fcst time 0 hrs:from 201809160000,
 29:Mass density:concentration (instant):regular_ll:surface:level 0:fcst time 0 hrs:from 201809160000,
 30:Mass density:concentration (instant):regular_ll:surface:level 0:fcst time 0 hrs:from 201809160000,
 31:Total column:integrated mass density (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000,
 32:Total column:integrated mass density (instant):regular_ll:atmosphere:level 0 -:fcst time 0 hrs:from 201809160000]

Notice that this doesn't give the short names correctly but rather the variable long name description as the name. This is almost indeciferable. wgrib2 output looks way different. 

1:0:d=2018091200:ASYSFK:entire atmosphere:105 hour fcst:aerosol=Total Aerosol:aerosol_size <2e-05:aerosol_wavelength >=3.38e-07,<=3.42e-07
2:88995:d=2018091200:SSALBK:entire atmosphere:105 hour fcst:aerosol=Total Aerosol:aerosol_size <2e-05:aerosol_wavelength >=3.38e-07,<=3.42e-07
3:173734:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Total Aerosol:aerosol_size <2e-05:aerosol_wavelength >=5.45e-07,<=5.65e-07
4:257495:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:aerosol_wavelength >=5.45e-07,<=5.65e-07
5:308070:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:aerosol_wavelength >=5.45e-07,<=5.65e-07
6:381840:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Sulphate Dry:aerosol_size <7e-07:aerosol_wavelength >=5.45e-07,<=5.65e-07
7:434792:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:aerosol_wavelength >=5.45e-07,<=5.65e-07
8:496510:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:aerosol_wavelength >=5.45e-07,<=5.65e-07
9:551432:d=2018091200:AEMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
10:559057:d=2018091200:SEDMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
11:598927:d=2018091200:DDMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
12:646148:d=2018091200:WLSMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
13:684570:d=2018091200:WDCPMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
14:718763:d=2018091200:AEMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
15:766737:d=2018091200:SEDMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
16:830919:d=2018091200:DDMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
17:897610:d=2018091200:WLSMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
18:952870:d=2018091200:WDCPMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
19:1000818:d=2018091200:AEMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <2.36e-08:
20:1020808:d=2018091200:SEDMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:
21:1066505:d=2018091200:DDMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:
22:1104627:d=2018091200:WLSMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:
23:1140259:d=2018091200:WDCPMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:
24:1168276:d=2018091200:AEMFLX:entire atmosphere:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
25:1186046:d=2018091200:SEDMFLX:entire atmosphere:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
26:1231200:d=2018091200:DDMFLX:entire atmosphere:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
27:1280346:d=2018091200:WLSMFLX:no_level:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
28:1327429:d=2018091200:WDCPMFLX:no_level:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
29:1359326:d=2018091200:MASSDEN:surface:105 hour fcst:aerosol=Dust Dry:aerosol_size <1e-05:
30:1391056:d=2018091200:MASSDEN:surface:105 hour fcst:aerosol=Dust Dry:aerosol_size <2.5e-06:
31:1427968:d=2018091200:COLMD:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <1e-05:
32:1468168:d=2018091200:COLMD:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2.5e-06:
bbakernoaa commented 6 years ago

I realize this is a little different time from the wgrib2 output but the file contents are the same other than a different time

jswhit commented 6 years ago

Looks like the ecCodes does not populate the shortName key, only the parameterName. Don't know if this is an issue with the file or the library.

bbakernoaa commented 6 years ago

@jswhit is it that eccodes does not read table 4 template 48? It appears that this is how the files are encoded.

jswhit commented 6 years ago

I have no idea - would have to dig into the ecCodes source code to figure that out. It's here (https://confluence.ecmwf.int//display/ECC/Releases) if you want to take a look.

Specifically,

eccodes-2.8.2-Source/definitions/grib2/template.4.48.def

bbakernoaa commented 6 years ago

It is strange. The table is there but when you go to http://apps.ecmwf.int/codes/grib/format/grib2/templates/4/ it doesn't show template 4.48