search

alexander-petkov / wfas

A placeholder for the WFAS project.
5 stars 1 forks source link

Add NBM forecast data to Geoserver #28

Open alexander-petkov opened 4 years ago

alexander-petkov commented 4 years ago

Add NBM forecast data to Geoserver https://www.weather.gov/mdl/nbm_home

alexander-petkov commented 4 years ago
These are the bands to extract: Short Name Description Band # Level
APCP Total precipitation [kg/(m^2)] 28 0[-] SFC="Ground or water surface"
DSWRF Downward short-wave radiation flux [W/(m^2)] 6 0[-] SFC="Ground or water surface"
RH Relative humidity [%] 29 2[m]
TMP Temperature @2[m] 35 2[m]
TCDC Total cloud cover [%] 30 0[-] SFC="Ground or water surface"
WIND Wind speed (m/s) 44 10[m]
WDIR Wind Direction from which blowing 41 10[m]
alexander-petkov commented 4 years ago

Hourly resolution is up to 36 hours from forecast time.

3-hour resolution is for hours 36-192

I will start with the hourly data for now.

wmjolly commented 4 years ago

We should pipeline the solar radiation into this dataset too.

On Wed, Jul 1, 2020 at 1:45 PM alexander-petkov notifications@github.com wrote:

Hourly resolution is up to 36 hours from forecast time.

3-hour resolution is for hours 36-192

I will start with the hourly data for now.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/28#issuecomment-652612194, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4G3DYFBOB3N7LFPM6GGI3RZOG5PANCNFSM4NS2SF6A .

alexander-petkov commented 4 years ago

We should pipeline the solar radiation into this dataset too.

Will do!!

alexander-petkov commented 4 years ago

Get mosaic granules via REST:

http://localhost:8080/geoserver/rest/workspaces/nbm/coveragestores/TMP/coverages/Temperature/index/granules.json

alexander-petkov commented 4 years ago

cdo has problems reading grib files that are for every 6th hour of the NBM forecast. Hours 6, 12 18 and so on. Very strange...

/mnt/cephfs/miniconda3/bin/cdo showtime blend.t00z.master.f006.co.grib2 
Warning (gribapiScanTimestep2): Record 12 (name= id=204.3.10 lev1=0 lev2=0) timestep 2: Parameter not defined at timestep 1!

cdo showtime: Open failed on >blend.t00z.master.f006.co.grib2<
Unsupported file structure

That said, in this case I only need to extract the timestamp.

alexander-petkov commented 4 years ago

Use gdal to get seconds since epoch for the forecast valid time, then convert with date:

gdalinfo blend.t00z.master.f006.co.grib2 |grep '   GRIB_VALID_TIME=' -m 1|cut -d ' ' -f 7
date -d @1593756000 +'%Y%m%d%H%M'
alexander-petkov commented 4 years ago

I finisned writing the update script and configured the mosaics, but some of the downoaded files are problematic, causing breakage.

alexander-petkov commented 4 years ago

The Grib files for this archive differ in structure from one another--files for different time stamps have a different number of bands. 

for g in `ls blend.20200710/*.grib2`;do /mnt/cephfs/miniconda3/bin/gdalinfo $g |grep ^Band |wc -l;done|sort|uniq -c
      4 46
     20 47
      1 50
      5 51
      1 57
      1 59
      1 62
      1 89
      2 91

That means that I cannot use a set of indices to extract data for the 7 variables we need.

I will probably resort to cdo, which gives cryptic names. Attempting to decipher names shown by cdo: aptmp: Apparent temperature @2m height above ground dswrf: Solar Radiation wdir: Wind direction ws: Wind Speed

alexander-petkov commented 4 years ago

Use the grib utilities from eccodes, they seem to read these files OK so far Link to wiki: https://confluence.ecmwf.int/pages/viewpage.action?pageId=52462916

It is also already installed by anaconda.

Variables names reported by grib_ls: 10wdir 10si 2t 2r tcc dswrf tp

alexander-petkov commented 4 years ago

Total precipitation: how to copy only the 1-hour forecast data into a grib file:

for g in blend.20200710/*.grib2;do /mnt/cephfs/miniconda3/bin/grib_copy -w shortName=tp,lengthOfTimeRange:=1 $g $g.tp ;done

the filters -w shortName=tp,lengthOfTimeRange:=1 do the magic. This will copy the % probability (Band 1) , as well as the forecasted accumulation (Band 2).

Now if I can only figure a filter for Total precipitation [kg m2]

alexander-petkov commented 4 years ago

Examining the # of bands for different forecast days:

for g in `seq -w 001 1 36` ;do echo $g ' ' `/mnt/cephfs/miniconda3/bin/gdalinfo blend.20200710/*.f$g*.grib2 |grep Band |wc -l` ' ' `/mnt/cephfs/miniconda3/bin/gdalinfo blend.20200713/*.f$g*.grib2 |grep Band |wc -l`;done
forecast hour 07/10 07/13
001 47 47
002 47 47
003 51 51
004 47 47
005 47 47
006 57 57
007 47 47
008 47 47
009 51 51
010 47 47
011 47 47
012 62 62
013 47 47
014 47 47
015 51 51
016 47 47
017 47 47
018 59 59
019 47 47
020 47 47
021 51 51
022 47 47
023 47 47
024 91 91
025 47 47
026 47 47
027 51 51
028 47 47
029 47 47
030 89 89
031 46 46
032 46 46
033 50 50
034 46 46
035 46 46
036 91 91

The number of bands correspond for each hour, which is good.

alexander-petkov commented 4 years ago

grib_copy a subset for each variable, filtering by a shortName:

var # of bands filters to extract the needed band
10wdir 1
10si 2 shortName=10si,productDefinitionTemplateNumber=0
2t 1
2r 1
tcc 1
dswrf 1 (not found in 007, 031--probably b/c is night over US?)
tp 2,4 or 5 shortName=tp,lengthOfTimeRange:=1,productDefinitionTemplateNumber=8
alexander-petkov commented 4 years ago

Solar radiation (dswrf) is missing 1 or 2 timesteps, denending at what hour the forecast was run.

It seems to be missing very hour @ midnight West Coast time, which makes sense I guess. Also makes gaps in the data.

alexander-petkov commented 4 years ago

NBM data is now configured: Screenshot from 2020-07-13 15-23-41

alexander-petkov commented 4 years ago

Committed update script for NBM

wmjolly commented 4 years ago

Could we make a table to each dataset and which times are available? It would be good to see what data are available for different forecast time periods.

On Mon, Jul 13, 2020 at 3:25 PM alexander-petkov notifications@github.com wrote:

NBM data is now configured: [image: Screenshot from 2020-07-13 15-23-41] https://user-images.githubusercontent.com/39599557/87355101-ffd31080-c51c-11ea-92ca-99a38f700ca1.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/28#issuecomment-657801261, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4G3DYHGTESWQAHFSP5C3TR3N3TZANCNFSM4NS2SF6A .

alexander-petkov commented 4 years ago

A summary that reflects the current status of weather archives, and that is also updated after archives are synchronized?

wmjolly commented 4 years ago

Just thinking about table for the forecasts that summarize the forecast times to show where there is hourly, 3x, 6x per day etc.

On Tue, Jul 14, 2020 at 3:05 PM alexander-petkov notifications@github.com wrote:

A summary that reflects the current status of weather archives, and that is also updated after archives are synchronized?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/28#issuecomment-658413094, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4G3D2LT5ICYJF77GQTLTTR3TCBFANCNFSM4NS2SF6A .

alexander-petkov commented 4 years ago

I made a table summary in the general ticket: https://github.com/alexander-petkov/wfas/issues/7#issuecomment-658433718

alexander-petkov commented 3 years ago

Fixed broken archive download, due to remote FTP server changes in this commit

Problems with dswrf remain, in fact there are a few other granules broken. Will monitor for changes.