search

Unidata / thredds

THREDDS Data Server v4.6
https://www.unidata.ucar.edu/software/tds/v4.6/index.html
266 stars 179 forks source link

Support GRIB2 PDS Template 32 #846

Open lesserwhirls opened 7 years ago

lesserwhirls commented 7 years ago

We are missing Product Definition Section (PDS) Template 32:

template description
32 Analysis or forecast at a horizontal level or in a horizontal layer at a point in time for simulate (synthetic) staellite data (see Template 4.32)

ucar.nc2.grib.grib2.Grib2Pds will need to be extended. The sample file is not licensed for redistribution (thus not attached), so will need to find a public dataset to use for tests.

keithto commented 7 years ago

We are seeking a sample grib2 file that can be distributed to public. At the same time, we might send a sample privately to who would like to help. Please contact me. Thanks.

cofinoa commented 6 years ago

@lesserwhirls and @keithto

The HWRF data from NCEP could be a good test:

from OSGeo/gdal#249

Sample GRIB2: http://nomads.ncep.noaa.gov/pub/data/nccf/com/hur/prod/ (Pick any HWRF directory, download any "hwrfsat" file.)

BTW, the template 4.32 is quite similar to 4.31 .

@lesserwhirls I think I can add the 4.32 very easily do you want me to try?

lesserwhirls commented 6 years ago

Sure, that would be awesome!

cofinoa commented 6 years ago

@lesserwhirls I have hit a wall. The data contains info about satellite id, sensor id and band been use, but it's been defined as additional octets in the PDS template. The problem it's how to map those information... to a parameter id? to a new level type? to a new band coordinate? all of them?

In the above example all data corresponds to the same grib2 parameter :Grib2_Parameter = 0, 5, 7; :long_name = "Brightness temperature";). For example looking at: http://nomads.ncep.noaa.gov/pub/data/nccf/com/hur/prod/hwrf.2018012412/invest99s.2018012412.hwrfsat.core.0p02.f036.grb2.idx

you can see that 18 GRIB2 messages have the same parameter and forecast step:

1:0:d=2018012412:BRTEMP:no_level:36 hour fcst:DMSP 17 SSMIS 183.31 GHz H pol.
2:180743:d=2018012412:BRTEMP:no_level:36 hour fcst:DMSP 17 SSMIS 19.35 GHz H pol.
3:442498:d=2018012412:BRTEMP:no_level:36 hour fcst:DMSP 17 SSMIS 19.35 GHz V pol.
4:679632:d=2018012412:BRTEMP:no_level:36 hour fcst:DMSP 17 SSMIS 37.00 GHz H pol.
5:955770:d=2018012412:BRTEMP:no_level:36 hour fcst:DMSP 17 SSMIS 37.00 GHz V pol.
6:1203762:d=2018012412:BRTEMP:no_level:36 hour fcst:DMSP 17 SSMIS 91.65 GHz V pol.
7:1428495:d=2018012412:BRTEMP:no_level:36 hour fcst:DMSP 17 SSMIS 91.65 GHz H pol.
8:1685709:d=2018012412:BRTEMP:no_level:36 hour fcst:METEOSAT 10 SEVIRI 6.27 um 
9:1968662:d=2018012412:BRTEMP:no_level:36 hour fcst:METEOSAT 10 SEVIRI 7.35 um 
10:2264792:d=2018012412:BRTEMP:no_level:36 hour fcst:METEOSAT 10 SEVIRI 8.71 um 
11:2506341:d=2018012412:BRTEMP:no_level:36 hour fcst:METEOSAT 10 SEVIRI 9.66 um 
12:2744063:d=2018012412:BRTEMP:no_level:36 hour fcst:METEOSAT 10 SEVIRI 10.77 um 
13:2986938:d=2018012412:BRTEMP:no_level:36 hour fcst:METEOSAT 10 SEVIRI 11.93 um 
14:3223885:d=2018012412:BRTEMP:no_level:36 hour fcst:METEOSAT 10 SEVIRI 13.34 um 
15:3544530:d=2018012412:BRTEMP:no_level:36 hour fcst:INSAT 3D VHRR 3.92 um 
16:3797267:d=2018012412:BRTEMP:no_level:36 hour fcst:INSAT 3D VHRR 6.88 um 
17:4087431:d=2018012412:BRTEMP:no_level:36 hour fcst:INSAT 3D VHRR 10.81 um 
18:4330519:d=2018012412:BRTEMP:no_level:36 hour fcst:INSAT 3D VHRR 11.94 um

but they differ on platform/source, sensor/instrument, band and polarization. I have seen some examples where the band it's been used as a new coordinate ... any ideas how to map this? there is any examples for templates 4.30 or 4.31?

lesserwhirls commented 6 years ago

HI @cofinoa! Thanks for taking a look at this. Supporting this kind of GRIB2 collection will be a quite a bit of work in the code. It's not just supporting the PDS, but we will need to extend the GRID code to include a wavelength dimension, much like we do with the ensemble dimension. Then, we will need to upgrade the GRIB variable naming code to take into account, say, platform name. Between those two enhancements, I think we will be able to uniquely identify the grids in this kind of file.

This might also require a change to the GRIB index files, which means we would need to tackle this in v5.0 of the code, rather than 4.6, as we've already bumped the version of the index files there.

ethanrd commented 6 years ago

@lesserwhirls Is a wavelength dimension required? Couldn't the different wavelengths be kept as separate variables? Not as elegant as a new dimension but much less work.

lesserwhirls commented 6 years ago

True - the wavelengths could be included in the variable name to separate the grids.

ColemanTom commented 4 years ago

@lesserwhirls here is a sample file like discussed in https://github.com/Unidata/netcdf-java/pull/285#issuecomment-627675941

It has 3 grib messages. Two have the same wavelength but different satellite identifiers (message 1 and 2). Two have the same satellite identifiers but different wavelengths (message 1 and 3). The values for message 2 and 3 are identical. It is a 22 by 34 grid (approximately 3.7 by 3.7 degrees). Forecast hour 43 for the 12Z 12 May 2020 model basetime.

synsat.BOM.modded.2020051212_043.zip

lesserwhirls commented 4 years ago

Excellent - thank you @ColemanTom! I'll take a look and see where things stand.