search

ropensci / rnoaa

R interface to many NOAA data APIs
https://docs.ropensci.org/rnoaa
Other
330 stars 84 forks source link

Wavewatch #141

Closed sckott closed 4 years ago

sckott commented 8 years ago

data: http://polar.ncep.noaa.gov/waves/nopp-phase1/

planetos has this data http://data.planetos.com/datasets/noaa_ww3_global_1.25x1d:noaa-wave-watch-iii-nww3-ocean-wave-model - but rate limits are pretty low and users have to sign up there

jebyrnes commented 8 years ago

http://polar.ncep.noaa.gov/waves/products.shtml ftp://polar.ncep.noaa.gov/pub/history/waves/ or ftp://polar.ncep.noaa.gov/pub/waves/

Global hindcasts back to 1997.

jebyrnes commented 8 years ago

But grib2 formats - not always fun. Although wgrib2 can convert them to netcdf files.

sckott commented 6 years ago

think we can go with raster for reading grib files

sckott commented 6 years ago

http://polar.ncep.noaa.gov/waves/download.shtml?

jebyrnes commented 6 years ago

Yes! ftp://polar.ncep.noaa.gov/pub/history/waves in particular has the hindcasts, which is often what folk want!

On Dec 1, 2017, at 11:04 AM, Scott Chamberlain notifications@github.com<mailto:notifications@github.com> wrote:

http://polar.ncep.noaa.gov/waves/download.shtml?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/rnoaa/issues/141#issuecomment-348533891, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAuOosZ5eKBqUG4fYYssH2AOHk-CeQDiks5s8COHgaJpZM4HxLYu.

sckott commented 6 years ago

okay, thanks!

sckott commented 5 years ago

looked at this again, but we still don't have a robust way to read grb/grb2 files, so not going to do this yet

jebyrnes commented 5 years ago

Hrm. rNomads has a function - https://github.com/cran/rNOMADS/blob/master/R/ReadGrib.R - and I believe that stars handles these filetypes, no?

sckott commented 5 years ago

ah, i see the rnomads one, but didn't want to depend on that since it's calling cli tools that look very unwieldy (wgrib/wgrib2).

didn't think about stars though, looks like that works. is the output from stars::read_stars(), a stars object, what users would expect/want?

jebyrnes commented 5 years ago

Dunno - have actually not used it yet! In the future, definitely! But - can it be converted to a raster (or give the user the option either way, like rnaturalearth does?)

sckott commented 5 years ago

not sure about raster, just used stars for the first time now. @mdsumner any idea if we can read_stars, then convert to a raster?

mdsumner commented 5 years ago

raster and stars can both read grib, because GDAL - it will usually require some additional care for the metadata though.

Got any direct links, none of those above work for me (??)

I thought there was a new package for Grib in R, (not just gribr) but I can't find it

sckott commented 5 years ago

thanks @mdsumner ftp://polar.ncep.noaa.gov/pub/history/waves/nww3 and ftp://polar.ncep.noaa.gov/pub/history/waves/multi_1/

mdsumner commented 5 years ago

Some of this will be OS and version dependent, but I get ok results via the url and for the file: 

  u <- "ftp://polar.ncep.noaa.gov/pub/history/waves/nww3/akw.dp.200409.grb"
library(stars)
#> Loading required package: abind
#> Loading required package: sf
#> Linking to GEOS 3.7.0, GDAL 2.4.0, PROJ 5.2.0
  library(raster)
#> Loading required package: sp
raster::brick(u)
#> class       : RasterBrick 
#> dimensions  : 123, 155, 19065, 241  (nrow, ncol, ncell, nlayers)
#> resolution  : 0.5, 0.25  (x, y)
#> extent      : 159.25, 236.75, 44.625, 75.375  (xmin, xmax, ymin, ymax)
#> coord. ref. : +proj=longlat +a=6371200 +b=6371200 +no_defs 
#> data source : ftp://polar.ncep.noaa.gov/pub/history/waves/nww3/akw.dp.200409.grb 
#> names       : akw.dp.200409.1, akw.dp.200409.2, akw.dp.200409.3, akw.dp.200409.4, akw.dp.200409.5, akw.dp.200409.6, akw.dp.200409.7, akw.dp.200409.8, akw.dp.200409.9, akw.dp.200409.10, akw.dp.200409.11, akw.dp.200409.12, akw.dp.200409.13, akw.dp.200409.14, akw.dp.200409.15, ...
# class       : RasterBrick 
# dimensions  : 123, 155, 19065, 241  (nrow, ncol, ncell, nlayers)
# resolution  : 0.5, 0.25  (x, y)
# extent      : 159.25, 236.75, 44.625, 75.375  (xmin, xmax, ymin, ymax)
# coord. ref. : +proj=longlat +a=6371200 +b=6371200 +no_defs 
# data source : ftp://polar.ncep.noaa.gov/pub/history/waves/nww3/akw.dp.200409.grb 
# names       : akw.dp.200409.1, akw.dp.200409.2, akw.dp.200409.3, akw.dp.200409.4, akw.dp.200409.5, akw.dp.200409.6, akw.dp.200409.7, akw.dp.200409.8, akw.dp.200409.9, akw.dp.200409.10, akw.dp.200409.11, akw.dp.200409.12, akw.dp.200409.13, akw.dp.200409.14, akw.dp.200409.15, ... 

##stars::read_stars(u, proxy = TRUE)  ## needs fixing

f <- basename(u)
if (!file.exists(f)) curl::curl_download(u, f)
(a <- stars::read_stars(f, proxy = TRUE))
#> stars_proxy object with 1 attribute in file:
#> $akw.dp.200409.grb
#> [1] "akw.dp.200409.grb"
#> 
#> dimension(s):
#>      from  to offset delta                       refsys point values    
#> x       1 155 159.25   0.5 +proj=longlat +a=6371200 ...    NA   NULL [x]
#> y       1 123 75.375 -0.25 +proj=longlat +a=6371200 ...    NA   NULL [y]
#> band    1 241     NA    NA                           NA    NA   NULL
# stars_proxy object with 1 attribute in file:
#   $akw.dp.200409.grb
# [1] "akw.dp.200409.grb"
# 
# dimension(s):
#   from  to offset delta                       refsys point values    
# x       1 155 159.25   0.5 +proj=longlat +a=6371200 ...    NA   NULL [x]
# y       1 123 75.375 -0.25 +proj=longlat +a=6371200 ...    NA   NULL [y]
# band    1 241     NA    NA                           NA    NA   NULL 

plot(a[,,,1], reset = F)
maps::map("world2", add = T)


devtools::session_info()
#> ─ Session info ──────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.0 (2019-04-26)
#>  os       Ubuntu 18.04.2 LTS          
#>  system   x86_64, linux-gnu           
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_AU.UTF-8                 
#>  ctype    en_AU.UTF-8                 
#>  tz       Etc/UTC                     
#>  date     2019-05-02                  
#> 
#> ─ Packages ──────────────────────────────────────────────────────────────
#>  package     * version    date       lib source                        
#>  abind       * 1.4-5      2016-07-21 [2] CRAN (R 3.6.0)                
#>  assertthat    0.2.1      2019-03-21 [2] CRAN (R 3.6.0)                
#>  backports     1.1.4      2019-04-10 [2] CRAN (R 3.6.0)                
#>  callr         3.2.0      2019-03-15 [2] CRAN (R 3.6.0)                
#>  class         7.3-15     2019-01-01 [4] CRAN (R 3.6.0)                
#>  classInt      0.3-3      2019-04-26 [2] CRAN (R 3.6.0)                
#>  cli           1.1.0      2019-03-19 [2] CRAN (R 3.6.0)                
#>  codetools     0.2-16     2018-12-24 [4] CRAN (R 3.6.0)                
#>  crayon        1.3.4      2017-09-16 [2] CRAN (R 3.6.0)                
#>  curl          3.3        2019-01-10 [2] CRAN (R 3.6.0)                
#>  DBI           1.0.0      2018-05-02 [2] CRAN (R 3.6.0)                
#>  desc          1.2.0      2018-05-01 [2] CRAN (R 3.6.0)                
#>  devtools      2.0.2      2019-04-08 [2] CRAN (R 3.6.0)                
#>  digest        0.6.18     2018-10-10 [2] CRAN (R 3.6.0)                
#>  e1071         1.7-1      2019-03-19 [2] CRAN (R 3.6.0)                
#>  evaluate      0.13       2019-02-12 [2] CRAN (R 3.6.0)                
#>  fs            1.2.7      2019-03-19 [2] CRAN (R 3.6.0)                
#>  glue          1.3.1      2019-03-12 [2] CRAN (R 3.6.0)                
#>  highr         0.8        2019-03-20 [2] CRAN (R 3.6.0)                
#>  htmltools     0.3.6      2017-04-28 [2] CRAN (R 3.6.0)                
#>  KernSmooth    2.23-15    2015-06-29 [4] CRAN (R 3.6.0)                
#>  knitr         1.22       2019-03-08 [2] CRAN (R 3.6.0)                
#>  lattice       0.20-38    2018-11-04 [4] CRAN (R 3.6.0)                
#>  magrittr      1.5        2014-11-22 [2] CRAN (R 3.6.0)                
#>  maps          3.3.0      2018-04-03 [2] CRAN (R 3.6.0)                
#>  memoise       1.1.0      2017-04-21 [2] CRAN (R 3.6.0)                
#>  pkgbuild      1.0.3      2019-03-20 [2] CRAN (R 3.6.0)                
#>  pkgload       1.0.2      2018-10-29 [2] CRAN (R 3.6.0)                
#>  prettyunits   1.0.2      2015-07-13 [2] CRAN (R 3.6.0)                
#>  processx      3.3.0      2019-03-10 [2] CRAN (R 3.6.0)                
#>  ps            1.3.0      2018-12-21 [2] CRAN (R 3.6.0)                
#>  R6            2.4.0      2019-02-14 [2] CRAN (R 3.6.0)                
#>  raster      * 2.8-19     2019-01-30 [2] CRAN (R 3.6.0)                
#>  Rcpp          1.0.1      2019-03-17 [2] CRAN (R 3.6.0)                
#>  remotes       2.0.4      2019-04-10 [2] CRAN (R 3.6.0)                
#>  rgdal         1.4-3      2019-03-14 [2] CRAN (R 3.6.0)                
#>  rlang         0.3.4.9003 2019-05-02 [2] Github (r-lib/rlang@6a232c0)  
#>  rmarkdown     1.12       2019-03-14 [2] CRAN (R 3.6.0)                
#>  rprojroot     1.3-2      2018-01-03 [2] CRAN (R 3.6.0)                
#>  sessioninfo   1.1.1      2018-11-05 [2] CRAN (R 3.6.0)                
#>  sf          * 0.7-4      2019-04-25 [2] CRAN (R 3.6.0)                
#>  sp          * 1.3-1      2018-06-05 [2] CRAN (R 3.6.0)                
#>  stars       * 0.3-1      2019-05-01 [1] local                         
#>  stringi       1.4.3      2019-03-12 [2] CRAN (R 3.6.0)                
#>  stringr       1.4.0      2019-02-10 [2] CRAN (R 3.6.0)                
#>  testthat      2.1.1      2019-04-23 [2] CRAN (R 3.6.0)                
#>  units         0.6-2      2018-12-05 [2] CRAN (R 3.6.0)                
#>  usethis       1.5.0.9000 2019-05-02 [2] Github (r-lib/usethis@5482403)
#>  withr         2.1.2      2018-03-15 [2] CRAN (R 3.6.0)                
#>  xfun          0.6        2019-04-02 [2] CRAN (R 3.6.0)                
#>  yaml          2.2.0      2018-07-25 [2] CRAN (R 3.6.0)                
#>

Created on 2019-05-02 by the reprex package (v0.2.1)

(good example data, thanks!)

sckott commented 5 years ago

thanks much @mdsumner !!

@jebyrnes do you prefer to get back a raster brick or a stars object?

sckott commented 5 years ago

ugh, this README ftp://polar.ncep.noaa.gov/pub/history/waves/nww3/README explains the multi_* files at ftp://polar.ncep.noaa.gov/pub/history/waves/multi_1/ and this README ftp://polar.ncep.noaa.gov/pub/history/waves/multi1/00README also explains those `multi*` files - whereas no README explains the files in ftp://polar.ncep.noaa.gov/pub/history/waves/nww3

and YYYYMM is the year, month

sckott commented 5 years ago

any thoughts on my above comment about file names @jebyrnes ? or @rmendels ?

rmendels commented 5 years ago

@sckott @jebyrnes I see a README. I think this is the info:

.  Field output
    Field output refers to bulk spectral properties over the spatial domain. 

    The multi-grid WW3 model is run as a mosaic of grids that are two way
    nested. The spectral parameters are computed on each of these grids and
    are stored in grib 2 format. The naming convention is as follows

      multi_1.[GRIDID].[PARID].[YYYYMM].grb2

    where

       GRIDID are the different grids that make up the WW3 domain. These are
       (in order of resolution)    
          glo_30m - The 30 arc-minute global grid
          ao_30m  - The 30 arc minute arctic grid 
          ak_10m  - The 10 arc minute regional Alaska grid
                    (it is 15 arc minute along longitude and 10 arc minute
                     along lattitude)
          at_10m  - The 10 arc minute Northern Atlantic regional grid 
                    (this grid covers the Gulf of Mexico and out to the 
                     continental shelf on the US East Coast. It does not 
                     extend across the Atlantic)
          wc_10m  - The 10 arc minute regional grid for teh US West Coast
          ep_10m  - The 10 arc minute grid covering some of the islands in 
                    the Pacific (including the Hawaiian islands)
          ak_4m   - The 4 arc minute coastal grid for Alaska
                    (8 arc minute along longituide and 4 arc minute along
                     lattitude)
          at_4m   - The 4 arc minute coastal grid for the US East Coast
                    (includes the waters around the island of Peurto Rico)
          wc_4m   - The 4 arc minute coastal grid for the US West Coast
                    (includes the coastal waters of Hawaii)

       PARID refers to the spectral parameters. The archived parameters are
          wind    - The 10 m wind speeds and direction gtom GFS
          hs      - Significant wave height (in m)
          tp      - Peak period (in s)
          dp      - Average direction at the peak period

       YYYMM refers to the year month in a six digit format

I would also point out that at least some of this data is available on the UAF ERDDAP,
see here:

And this the perfect example of why there is the need for standardized services that provide metadata, virtual aggregation, hide the underlying format, and provide powerful subsetting. The real solution is to tell NCEP to add these datasets to the UAF effort.

At least some of the data are available through the NCEP THREDDS](https://data.nodc.noaa.gov/thredds/catalog/ncep/nww3/catalog.html)

which can be accessed using ncdf4 or RNetCDF.

sckott commented 5 years ago

yes, i have read that readme, but AFAICT it does not explain the file names, for example

akw.dp.200005.grb

what is the "akw"? the "dp" is explained and the date, but not the first 3 letters

and at the top they seem to be explaining only the files that start with multi because they say

The naming convention is as follows ... multi_1.[GRIDID].[PARID].[YYYYMM].grb2

right, i understand some of the data is avail. elsewhere, but it's a lot more work for me if I have to maintain two ways to get subsets of some data - I"m much prefer to maintain one way to do a thing.

rmendels commented 5 years ago

@sckott - This is NOAA, you expect clear explanations??!!! Hahahaha. Again why metadata standards are so important, and that they be included in both the files and the services, as in ERDDAP. But if I were a betting man, I would say "akw" for instance stands for:

ak - one of the alaska arcs (depending on the resolution) w - wind dp - Average direction at the peak period

If you want I can post this to a group I belong to to double-check.

sckott commented 5 years ago

great, thanks.

there's also these other options for the first set of letters, 3 or 4 characters i guess:

rmendels commented 5 years ago

@sckott I have posted this. Let's see if I get a response.

jebyrnes commented 5 years ago

I used this page as a shibboleth back when I was doing some work with these products - https://www.nco.ncep.noaa.gov/pmb/products/wave/

jebyrnes commented 5 years ago

N.B. it's issues like these that made me think that rnoaa would be a great place to access WW this with a nice clear vignette stating what each thing is. HA!

kjschaudt commented 5 years ago

Even if you get the data using Rnoaa, the WW3 modelers do not include the water depth at the gridpoint.   I've asked for the gridded bathymetry file so I can make sense out of the data near the coast and been informed that this information is not available.   Oh well.

On 5/14/19 6:14 PM, Jarrett Byrnes wrote:

N.B. it's issues like these that made me think that rnoaa would be a great place to access WW this with a nice clear vignette stating what each thing is. HA!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ropensci/rnoaa/issues/141?email_source=notifications&email_token=AFBHLFIVHHIKVLIVUKP6MI3PVNBVBA5CNFSM4B6EWYXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVNBYEY#issuecomment-492444691, or mute the thread https://github.com/notifications/unsubscribe-auth/AFBHLFINS3PEXLONVDEQA73PVNBVBANCNFSM4B6EWYXA.

rmendels commented 5 years ago

@kjschaudt @sckott do the WW3 files give the lat-lons of the grid points? And do you have to have their bathymetetry or just a good bathymetry. We have several bathymetry datasets on our ERDDAP server, including the latest high resolution one from Scripps, you gina use rerddap to do extracts (and please do an extract - if you want global, get the data from Scripps, the entire file is too large otherwise)

kjschaudt commented 5 years ago

The gridpoint lat/lon can be determined so I approximated the water depth from the best available bathymetry.   However, the actual model bathymetry is significant since the sites were in shallow water.    Without it, no clue as to if the bathymetry reasonably represents the situation at the site of interest.

On 5/14/19 9:00 PM, Roy Mendelssohn wrote:

@kjschaudt https://github.com/kjschaudt @sckott https://github.com/sckott do the WW3 files give the lat-lons of the grid points? And do you have to have their bathymetetry or just a good bathymetry. We have several bathymetry datasets on our ERDDAP server, including the latest high resolution one from Scripps, you gina use |rerddap| to do extracts (and please do an extract - if you want global, get the data from Scripps, the entire file is too large otherwise)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ropensci/rnoaa/issues/141?email_source=notifications&email_token=AFBHLFKGKQXCAGVLYY62SKLPVNVCVA5CNFSM4B6EWYXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVNJDLY#issuecomment-492474799, or mute the thread https://github.com/notifications/unsubscribe-auth/AFBHLFMORN5GCHCPCKT6DP3PVNVCVANCNFSM4B6EWYXA.

sckott commented 5 years ago

@jebyrnes had to look up "shibboleth" - that's at least a $20 word - thanks for the link - rnoaa will be a good place to access these data once I figure it out :)

sckott commented 5 years ago

the WW3 modelers do not include the water depth at the gridpoint.

@kjschaudt so does that make this data useless to you? to everyone?

kjschaudt commented 5 years ago

Scott,

The 3 letters are explained (sort of) at the following site which links to the grib files

https://www.nco.ncep.noaa.gov/pmb/products/wave/

AKW is alaska regional waters, ENP is eastern N Pacific, etc

On 5/14/19 4:53 PM, Scott Chamberlain wrote:

yes, i have read that readme, but AFAICT it does not explain the file names, for example

akw.dp.200005.grb

what is the "akw"? the "dp" is explained and the date, but not the first 3 letters

and at the top they seem to be explaining only the files that start with |multi| because they say

The naming convention is as follows ...
multi_1.[GRIDID].[PARID].[YYYYMM].grb2

right, i understand some of the data is avail. elsewhere, but it's a lot more work for me if I have to maintain two ways to get subsets of some data - I"m much prefer to maintain one way to do a thing.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ropensci/rnoaa/issues/141?email_source=notifications&email_token=AFBHLFLVQEHI3PA4XQVMOP3PVMYEHA5CNFSM4B6EWYXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVM4WKQ#issuecomment-492423978, or mute the thread https://github.com/notifications/unsubscribe-auth/AFBHLFIDRLYPRVEHI2X2TP3PVMYEHANCNFSM4B6EWYXA.

kjschaudt commented 5 years ago

They also do not publish a bathymetry file which would be used.

Less useful certainly since it limits confidence on the shelf. Can't tell if the shelf was even modelled.  In the old days, one simply used deepwater and ignored the shelf.

In court, I'd tear apart opposing experts who used it on the shelf.

On 5/15/19 1:47 PM, Scott Chamberlain wrote:

the WW3 modelers do not include the water depth at the gridpoint.

@kjschaudt https://github.com/kjschaudt so does that make this data useless to you? to everyone?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ropensci/rnoaa/issues/141?email_source=notifications&email_token=AFBHLFJZO42QX5PDRNI4PGTPVRLFLA5CNFSM4B6EWYXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVPTAPY#issuecomment-492777535, or mute the thread https://github.com/notifications/unsubscribe-auth/AFBHLFMQKSTXF43QA3WTIMDPVRLFLANCNFSM4B6EWYXA.

sckott commented 5 years ago

@kjschaudt thanks for the link !

hmm, well if there's anything we can do to help users get that bathymetry data, let me know where it can be retrieved

sckott commented 5 years ago

on branch waves

sckott commented 4 years ago

closing - if someone else wants to tackle this go ahead and we can re-open