NDFD 7-day forecast data retrieval

[alexander-petkov]

This is a subtask for #7

Configure data retrieval routine (Similar to #10 ) for 7-day NDFD forecast data.

Questions:

1. Is ~this online archive~ the correct location for the data we need? I can see a historical archive, which doesn't list data with a timestamp sometime in the near future.

2. Which variables do we need? I think this might have already been answered, so I will look back through correspondence.

[alexander-petkov]

NDFD Data Directory structure documentation, and data access portal:

https://www.weather.gov/mdl/ndfd_data_grid

For 7-day forecast, it looks like we need the files from the 001-003 and 004-007 directories:

https://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/

[alexander-petkov]

There seems to be a limit to the number of bands (32) in the Grib plugin for Geoserver...

For example, this file https://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003/ds.rhm.bin has 44 bands. Configuring a raster layer from that file throws an error.

If I subset that file to 32 bands (using cdo), I can configure a layer without errors in Geoserver.

./cdo -seltimestep,1/32 ~/ds.rhm.bin ~/ds.rhm.bin.grb

This means that downloaded data might require some processing, as in slice the files into manageable chunks.

[wmjolly]

Well that's kind of a bummer. I wonder if it would be easier to just use the floating point files I make?

MJ

On Fri, Jun 28, 2019 at 3:50 AM alexander-petkov notifications@github.com wrote:

There seems to be a limit to the number of bands (32) in the Grib plugin for Geoserver...

For example, this file

https://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003/ds.rhm.bin has 44 bands. Configuring a raster layer from that file throws an error.

If I subset that file to 32 bands (using cdo https://code.mpimet.mpg.de/projects/cdo, I can configure a layer without errors in Geoserver.

This means that downloaded data might require some processing, as in slice the files into manageable chunks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/11?email_source=notifications&email_token=AA4G3D7IWAH5Y5ATDB5WG6TP4XNHDA5CNFSM4H3VG5OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYZUFLY#issuecomment-506675887, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4G3D4R5KW47W52ANIF5YDP4XNHDANCNFSM4H3VG5OA .

[alexander-petkov]

Well that's kind of a bummer. I wonder if it would be easier to just use the floating point files I make? MJ

I was hoping to keep the data original.

[alexander-petkov]

Online NDFD archive seems to be inaccessible from wfas.net or the Openstack cluster today.

[alexander-petkov]

There seems to be a problem in Geotools regarding the 32-band "limitation" issue. If a multidimensional file has more than 32 bands, the retrieved imageIndex is like this: [32 33 34 ... n 0 1 2 3 ... 31]

Adding a Collections.sort() call to the getImageIndex method in GeoSpatialImageReader.java sorts the index and fixes this error:

  public List<Integer> getImageIndex(Query filterQuery) throws IOException {
        List<CoverageSlice> descs = slicesCatalog.getGranules(filterQuery);
        List<Integer> indexes = new ArrayList<Integer>();
        for (CoverageSlice desc : descs) {
            Integer index =
                    (Integer) desc.getOriginator().getAttribute(CoverageSlice.Attributes.INDEX);
            indexes.add(index);
        }
        Collections.sort(indexes);
        return indexes;
    }

[alexander-petkov]

Submitted a patch to Geotools: https://osgeo-org.atlassian.net/browse/GEOT-6340

[wmjolly]

Very cool contribution Alex!

MJ

On Wed, Jul 10, 2019 at 10:04 AM alexander-petkov notifications@github.com wrote:

Submitted a patch to Geotools: https://osgeo-org.atlassian.net/browse/GEOT-6340

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/11?email_source=notifications&email_token=AA4G3D47QTLEVWGVIA7WTUDP6YCCVA5CNFSM4H3VG5OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZT6KBI#issuecomment-510125317, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4G3D4A4ZBTUS4XQIH47IDP6YCCVANCNFSM4H3VG5OA .

[alexander-petkov]

I recompiled a new gt-coverage-api jar with this change and deployed a new geoserver cluster. NDFD Grib files with more than 32 bands are read and configurable now.

[wmjolly]

Outstanding! So cool that you were able to find and fix that problem.

MJ

On Fri, Jul 12, 2019 at 9:06 PM alexander-petkov notifications@github.com wrote:

I recompiled a new gt-coverage-api jar with this change and deployed a new geoserver cluster. NDFD Grib files with more than 32 bands are read and configurable now.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/11?email_source=notifications&email_token=AA4G3D6AZQZCEVO4P6ZC2ALP7FBDTA5CNFSM4H3VG5OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ3IAMQ#issuecomment-511082546, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4G3D4IGAORWHK5DYXC3Q3P7FBDTANCNFSM4H3VG5OA .

[alexander-petkov]

After downloading a new file (with the same name), a coverage can be refreshed simply by executing a PUT request via REST:

curl -v -u user:pass -XPUT -H "Content-type: text/xml" -d @/path/to/coverage/file/coverage.xml "http://192.168.59.56:8081/geoserver/rest/workspaces/weather/coveragestores/ndfd_rhm_grb/coverages/Relative_humidity_height_above_ground"

This forces the coverage to be reindexed (the time dimension, for example). No need to delete, then configure again. Nice!!

[wmjolly]

That is very cool.

On Mon, Jul 15, 2019 at 12:59 PM alexander-petkov notifications@github.com wrote:

After downloading a new file (with the same name), a coverage can be refreshed simply by executing a PUT request via REST:

curl -v -u user:pass -XPUT -H "Content-type: text/xml" -d @/path/to/coverage/file/coverage.xml "http://192.168.59.56:8081/geoserver/rest/workspaces/weather/coveragestores/ndfd_rhm_grb/coverages/Relative_humidity_height_above_ground"

This forces the coverage to be reindexed (the time dimension, for example). No need to delete, then configure again. Nice!!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/11?email_source=notifications&email_token=AA4G3D3JEUHNYVLBBIFANSTP7TCKLA5CNFSM4H3VG5OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ6UWUI#issuecomment-511527761, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4G3D24GJXBHX64PIN2SA3P7TCKLANCNFSM4H3VG5OA .

[alexander-petkov]

87 NDFD data sets!! Headspin!! https://www.weather.gov/media/mdl/ndfd/NDFDelem_fullres_201906.xls

The closest corresponding dataset I can find in that spreadsheet for "cloud cover" is "sky cover".

[wmjolly]

That's the one.... Sky Cover....

On Tue, Jul 16, 2019, 08:47 alexander-petkov notifications@github.com wrote:

87 NDFD data sets!! Headspin!! https://www.weather.gov/media/mdl/ndfd/NDFDelem_fullres_201906.xls

The closest corresponding dataset I can find in that spreadsheet for "cloud cover" is "sky cover".

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/11?email_source=notifications&email_token=AA4G3D7EHLFICZHBT3NLISLP7XNOLA5CNFSM4H3VG5OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2BDBQY#issuecomment-511848643, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4G3DYOUN77X4M7AFKHKNLP7XNOLANCNFSM4H3VG5OA .

[alexander-petkov]

I managed to configure all NDFD variables as Imagemosaics today, except pop12....

Something is different about the pop12 Grib files, and Geoserver throws and error during configuration.

On the other hand, 6 NDFD datasets can now be viewed through Geoserver:

[wmjolly]

Can I see this from a machine on Internet2?

On Tue, Jul 16, 2019, 13:41 alexander-petkov notifications@github.com wrote:

I managed to configure all NDFD variables as Imagemosaics today, except pop12....

Something is different about the pop12 Grib files, and Geoserver throws and error during configuration.

On the other hand, 6 NDFD datasets can now be viewed through Geoserver:

[image: Screenshot 2019-07-16 at 1 40 07 PM] https://user-images.githubusercontent.com/39599557/61324277-48070800-a7cf-11e9-9eeb-aa100ad48950.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/11?email_source=notifications&email_token=AA4G3DYGBHCDRDHMIYWFT4TP7YP43A5CNFSM4H3VG5OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2B5R7A#issuecomment-511957244, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4G3D4Y2A7BEVJLXN2RVXTP7YP43ANCNFSM4H3VG5OA .

[alexander-petkov]

This is the link for the 2 slave instances: http://192.168.59.41:8081/geoserver/web/

[alexander-petkov]

The NDFD FTP archive gets updated approx 16:15 UTC. Important for cron scheduling.

[alexander-petkov]

I managed to configure all NDFD variables as Imagemosaics today, except pop12....

Something is different about the pop12 Grib files, and Geoserver throws and error during configuration.

I think I figured the problem--there is a table name length limitation in Postgres. This is the variable name ~in the Grib file~: Total_precipitation_surface_12_Minute_Accumulation_probability_above_0p254

Edit: This gets worse--that long name is actually stored in an external XML file (grib2/grib2VarMap.xml) which is packaged in the grib jar library.

A Grib file doesn't store a variable name itself, but a code, which is decoded in an external table, if available: https://code.mpimet.mpg.de/boards/1/topics/6565?r=6582#message-6582

[alexander-petkov]

Here are some Imagemosaic configurations hints: https://geoserver.geo-solutions.it/downloads/releases/2.9.x-netcdf/readme.txt

CONFIGURATION HINTS

===================

• In order to support custom NetCDF CRSs, you may update the netcdf.projections.properties file (it already contains a couple of examples) and place it in GEOSERVER_DATA_DIR/user_projections folder (GeoServer restart is required).

• We found that some GRIB files (such as the .cip ones) have no valid magic number in the first bytes, nor a recognized GRIB extension. You should consider adding a -DNETCDF_FORCE_OPEN_CHECK=true JVM argument system property.

• In case you are configuring an ImageMosaic on top of GRIB/NetCDF datasets having parameter/variable names longer than 63 chars, please adds the parameter to the Parameters section of the indexer.xml file if you are configuring a PostGIS based datastore. (PostgresSQL default installation has Identifiers length limit = 63)

• In case the native CRS doesn't match the expected custom EPSG code (this may happen due to the underlying UCAR Grib library using float precision math instead of double) you may consider increasing the COMPARISON_TOLERANCE (default is 1E-9) to 1E-7. Add a -DCOMPARISON_TOLERANCE=1E-7 JVM argument to your container and restart GeoServer.

• for HRRR datasets (and other GRIB files having multiple time dimensions changing across files) it is required to add a "uniqueTimeAttribute=true" parameter to the auxiliary.xml file and make sure that all "timeX" attributes specified in schemas (both on low level index and imageMosaic index) are simply "time".

TheWrapStore=true parameter should fix the long name issue for the pop12 dataset.

Edit: indeed, the ds.pop12 dataset is now configured.

[alexander-petkov]

This is the latest list of NDFD configured coverages.

The archive is updated once a day, at 16:30 UTC.

I consider this issue resolved for now, and will close it

[alexander-petkov]

The NDFD forecast data is limited by the precip data, which is for 72 hour period with 6 hour resolution. The metadata description says "6 minute accumulation", but I think it is 6 hours. If that is the case hourly data can be derived pretty easily.

[alexander-petkov]

Regarding QPF, documentation says this:

Precipitation Amount is the expected quantity of liquid precipitation accumulated over a six hourly period....

So, I will divide by 6 to derive hourly QPF data.

[wmjolly]

Yes, that makes sense to me. We have some other ways to model the total hours of rainfall from 24 hour QPF that we will implement later.

On Wed, Oct 30, 2019 at 2:26 AM alexander-petkov notifications@github.com wrote:

Regarding QPF, documentation https://digital.weather.gov/staticpages/definitions.php says this:

Precipitation Amount is the expected quantity of liquid precipitation accumulated over a six hourly period....

So, I will divide by 6 to derive hourly QPF data.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/11?email_source=notifications&email_token=AA4G3DZK3DME4BF2JCNBHQDQRFAJRA5CNFSM4H3VG5OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECTJNAQ#issuecomment-547788418, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4G3D7ZQBQIFGSVDZOOXATQRFAJRANCNFSM4H3VG5OA .

[alexander-petkov]

Consider aligning NODATA values across datasets, so NODATA does not show in styles.

Example is from NDFD Temperature: