search

ropensci / wateRinfo

R package to download time series data from waterinfo.be
https://docs.ropensci.org/wateRinfo
MIT License
14 stars 3 forks source link

Tidal data from MOW-HIC stations? #11

Open LennertSchepers opened 7 years ago

LennertSchepers commented 7 years ago

On the waterinfo.be website, also the data from the navigable waterways is available, for example the tidal data of the Scheldt river (data from MOW-HIC). These stations are not accessible with the wateRinfo package. Is this a matter of adding some more functions, or are these data not (yet?) available through the API?

WillemMaetens commented 7 years ago

Downloading MOW-HIC data will require expansion of the extdata/lookup_timeseriesgroup.txt table with the timeseriesgroup_id's for the MOW-HIC data. Maybe these can be requested from HIC via waterinfo.be, e-mail contact button in the upper right corner > select "HIC" as your recepient.

Furthermore, the resolve_datasource function will need to be expanded to accommodate the stations in these timeseriesgroups. However, the whole resolve_datasource procedure can be avoided by incorporating the datasource in the timeseriesgroup_id (01 or 02 preceding the timeseriesgroup_id) and using datasource=0 by default in the call URL.

stijnvanhoey commented 6 years ago

This is indeed not supported by the timeseriesgroup_id at the moment, as the timeseriesgroup_id do not provide information on the MOW-HIC data. However, for many of the MOW-HIC stations, you can still request the available variables using the station name as provided on waterinfo.be (e.g. zes42a-1066), see this package tutorial.

Still, I also received the question at INBO to specifically target the tidal data as shown on the waterinfo.be Tide tab. As a temporay solution, I compiled a list of ts_id identifiers for a this subset of stations: tij_all_identifiers.txt. By filtering list of identifiers to your needs, you can still use the get_timeseries_tsid function.

With these identifiers, and using datasource = 2 you can actually download tidal data using the package functionalities.

Load the tidal stations ts_id information:

ts_id_getij <- read.csv("tij_all_identifiers.txt")

For example, imagine you're interested in the data of Dendermonde, i.e. zes21a-1066 (the code is very comparable to the batch examples):

library(wateRinfo)
dmonde <- ts_id_getij %>%
    filter(station_no == "zes21a-1066") %>%  # select all variables for Dendermonde
    group_by(ts_id) %>%
    do(get_timeseries_tsid(.$ts_id, period = "P1D",
                           to = "2017-01-02", datasource = 2)) %>% # 1 day of data
    ungroup() %>%
    left_join(ts_id_getij, by = "ts_id")

Plotting the downloaded data:

library(ggplot2)
ggplot(dmonde, aes(x = Timestamp, y = Value)) +
    geom_point() +
    facet_wrap(c("ts_name")) +
    scale_x_datetime(date_labels = "%H:%M",
                     date_breaks = "6 hours")

image

So, for the station, time series with different time resolution and some high/low water summary time series are available for download.

Another example: you're interested in the 10min based data of Liefkenshoek:

hwlw_campaign <- ts_id_getij %>%
    filter(grepl(".*10min.base", ts_name)) %>%
    filter(station_name == "Liefkenshoek tij/Zeeschelde") %>%
    group_by(ts_id) %>%
    do(get_timeseries_tsid(.$ts_id,
                           from = "2017-10-01 10:00:00",
                           to = "2017-10-02 11:00:00",
                           datasource = 2)) %>%
    ungroup() %>%
    left_join(ts_id_getij, by = "ts_id")
ggplot(hwlw_campaign, aes(x = Timestamp, y = Value)) +
    geom_line() +
    scale_x_datetime(date_labels = "%Y-%m-%d\n%H:%M",
                     date_breaks = "12 hours")

image

As @WillemMaetens suggests, the availability of tidal data as a timeseriesgroup_id would be easier to query the data and I hope we can improve the package when this information would become available. Hopefully, this can support the tidal data request for the moment.

stijnvanhoey commented 6 years ago

@LennertSchepers, was the explanation above sufficient to solve your case?

LennertSchepers commented 6 years ago

Hi Stijn. Thanks for quickly solving this! I think this is really useful for the people in our research group, who are doing quite some field work along the Scheldt River (also INBO's estuaries team). I will spread the news :).

Another useful -related- feature for fieldwork would be to have access to the short-term tidal water level prediction (https://www.waterinfo.be/default.aspx?path=NL/Thema/Getij_KT), e.g. to know when a field site at a certain elevation is accessible (no longer inundated). The prediction graphs on the website are sometimes hard to read accurately. However it seems that these data are not available for download from the waterinfo website. Maybe we can ask if they can make these data available?

stijnvanhoey commented 6 years ago

This is indeed something the INBO estuaries team uses as well. I'll try to convert the explanation above on the tidal data download as a new tutorial/vignette of the package, incorporating the information on the tij_all_identifiers.txt. Keeping this issue open in the meanwhile as reference for potential users.

For your data question of the new feature, I'll make a new issue and take contact with HIC.

stijnvanhoey commented 5 years ago

Notice, as waterinfo.be changed its time series identifiers, the file tij_all_identifiers.txt is no longer accurate. We need to find a more sustainable solution, but for the moment, this code snippet could help to collect the tidal stations:

library(httr)
library(jsonlite)
library(dplyr)
library(stringr)
library(purrr)
library(wateRinfo)

request_url <- "https://www.waterinfo.be/tsmpub/KiWIS/KiWIS?service=kisters&type=queryServices&request=getTimeseriesValueLayer&datasource=0&format=esrijson&timeseriesgroup_id=04156183,04156182&metadata=true&custattr_returnfields=dataprovider,dataowner,significant,Portal_Bekken&crs=webmercator&orderDir=asc&orderBy=value&md_returnfields=custom_attributes,station_id,station_no,station_name,ts_id,ts_name,stationparameter_name,ts_unitsymbol,parametertype_name&userId=PortalPublic"
response <- GET(request_url)
parsed <- fromJSON(content(response, "text"))$features$attributes

# add ts_identifier of the 10 minute time series
tidal_stations <- parsed %>% 
  filter(str_detect(station_name, "tij"), str_detect(station_no, "1066")) %>%
  mutate(station_no_short = str_sub(station_no, 3)) %>%
  mutate(station_identifier = map_chr(.$station_no_short, 
    function(x) {
      get_variables(x) %>% 
        filter(str_detect(ts_name, "Pv.10")) %>% 
        pull(ts_id)})
    )

returning the following table:

ts_value_id ts_id datetime req_timestamp ts_value station_latitude station_longitude station_id station_no station_name ts_name stationparameter_name ts_unitsymbol parametertype_name dataprovider dataowner significant Portal_Bekken station_no_short station_identifier
1 0454522010 2018-11-29T15:07:00.000+01:00 NA -0.29 51.14328 4.330275 0430078 04zes28a-1066 Hemiksem tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes28a-1066 54493010
2 0454018010 2018-11-29T15:07:00.000+01:00 NA -0.17 51.22747 4.399914 0419484 04zes21a-1066 Antwerpen tij/Zeeschelde Pv W m W MOW-HIC HIC 1 Beneden-Scheldebekken zes21a-1066 53989010
4 0455522010 2018-11-29T15:05:00.000+01:00 NA 0.01 51.12281 4.218673 0430081 04zes36a-1066 Temse tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes36a-1066 55493010
5 04102372010 2018-11-29T15:06:00.000+01:00 NA 0.16 51.09256 4.170995 0430084 04zes39a-1066 Driegoten tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes39a-1066 102376010
6 0454635010 2018-11-29T15:07:00.000+01:00 NA 0.16 51.26800 4.298524 0430071 04zes14a-1066 Kallosluis tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes14a-1066 54606010
7 0455594010 2018-11-29T15:05:00.000+01:00 NA 0.17 51.10964 4.173916 0430029 04dur01a-1066 Tielrode tij/Durme Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken dur01a-1066 55565010
8 0454965010 2018-11-29T15:05:00.000+01:00 NA 0.29 51.29639 4.285743 0430068 04zes10a-1066 Liefkenshoek tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes10a-1066 54936010
10 0456117010 2018-11-29T15:05:00.000+01:00 NA 0.41 51.34833 4.237932 0430062 04zes01a-1066 Prosperpolder tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes01a-1066 56088010
11 0455448010 2018-11-29T15:05:00.000+01:00 NA 0.47 51.05264 4.197217 0430087 04zes42a-1066 Sint-Amands tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes42a-1066 55419010
12 0455934010 2018-11-29T15:05:00.000+01:00 NA 0.52 51.07367 4.418806 0430053 04rup03a-1066 Walem tij/Rupel Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken rup03a-1066 55905010
19 0455009010 2018-11-29T15:05:00.000+01:00 NA 1.20 51.03378 4.467076 0430023 04dij06a-1066 Mechelen Benedensluis tij/Dijle Pv W m W MOW-HIC HIC 0 Dijlebekken dij06a-1066 54980010
20 0454215010 2018-11-29T15:05:00.000+01:00 NA 1.33 51.03456 4.103857 0419418 04zes47a-1066 Dendermonde tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes47a-1066 54186010
22 0454312010 2018-11-29T15:05:00.000+01:00 NA 1.44 51.10914 4.530167 0430164 04bnt03a-1066 Duffel Sluis tij/Nete Pv W m W MOW-HIC HIC 0 Netebekken bnt03a-1066 54283010
24 0455384010 2018-11-29T15:05:00.000+01:00 NA 1.99 51.00467 4.007578 0430091 04zes49a-1066 Schoonaarde tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes49a-1066 55355010
25 0494310010 2018-11-29T15:05:00.000+01:00 NA 2.00 51.02364 4.496889 0430233 04dij07a-1066 Mechelen Stuw Afwaarts tij/Dijle Pv W m W MOW-HIC HIC 0 Dijlebekken dij07a-1066 94280010
27 0454570010 2018-11-29T15:05:00.000+01:00 NA 2.16 51.01522 4.448936 0430056 04zen01a-1066 Hombeek tij/Zenne Pv W m W MOW-HIC HIC 0 Dijlebekken zen01a-1066 54580010
28 04102431010 2018-11-29T15:05:00.000+01:00 NA 2.27 51.01269 3.959686 0430096 04zes52a-1066 Uitbergen tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes52a-1066 102435010
29 0454852010 2018-11-29T15:05:00.000+01:00 NA 2.54 51.12636 4.571825 0430018 04bnt07a-1066 Lier Molbrug tij/Nete Pv W m W MOW-HIC HIC 1 Netebekken bnt07a-1066 54823010
31 04101662010 2018-11-29T15:05:00.000+01:00 NA 2.56 51.00709 3.876655 04227974 04zes55c-1066 Wetteren Brug tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes55c-1066 101631010
33 04102461010 2018-11-29T15:06:00.000+01:00 NA 2.73 51.10186 4.087806 0430032 04dur04a-1066 Waasmunster Brug tij/Durme Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken dur04a-1066 102465010
37 0455088010 2018-11-29T15:04:00.000+01:00 NA 2.80 51.00433 3.805347 0430242 04zes57a-1066 Melle tij/Zeeschelde Pv W m W MOW-HIC HIC 1 Beneden-Scheldebekken zes57a-1066 55059010
40 0456256010 2018-11-29T15:05:00.000+01:00 NA 3.24 50.99522 4.464076 0430059 04zen02a-1066 Zemst tij/Zenne Pv W m W MOW-HIC HIC 0 Dijlebekken zen02a-1066 56227010
41 0454356010 2018-11-29T15:05:00.000+01:00 NA 3.41 51.15803 4.608634 0430047 04knt01a-1066 Emblem tij/Kleine Nete Pv W m W MOW-HIC HIC 0 Netebekken knt01a-1066 54366010
42 0455844010 2018-11-29T15:05:00.000+01:00 NA 3.61 51.10794 4.067105 0430035 04dur05a-1066 Waasmunster Manta tij/Durme Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken dur05a-1066 55854010
43 0454683010 2018-11-29T15:05:00.000+01:00 NA 3.62 51.12722 4.648965 0430044 04gnt02a-1066 Kessel tij/Grote Nete Pv W m W MOW-HIC HIC 0 Netebekken gnt02a-1066 54693010
46 0494348010 2018-11-29T15:05:00.000+01:00 NA 4.72 51.02339 4.497031 0430236 04dij08a-1066 Mechelen Stuw Opwaarts tij/Dijle Pv W m W MOW-HIC HIC 0 Dijlebekken dij08a-1066 94318010
48 0454401010 2018-11-29T15:05:00.000+01:00 NA 4.99 51.04706 3.747216 0430105 04zes58a-1066 Gentbrugge tij/Zeeschelde Pv W m W MOW-HIC HIC 0 Beneden-Scheldebekken zes58a-1066 54411010

make sure to use the station_identifier column to request the 10 minute tidal data!

stijnvanhoey commented 5 years ago

A more complete overview of all available variables for these stations (so not only the 10 min data) can be retrieved by:

library(httr)
library(jsonlite)
library(dplyr)
library(stringr)

request_url <- "https://www.waterinfo.be/tsmpub/KiWIS/KiWIS?service=kisters&type=queryServices&request=getTimeseriesValueLayer&datasource=0&format=esrijson&timeseriesgroup_id=04156183,04156182&metadata=true&custattr_returnfields=dataprovider,dataowner,significant,Portal_Bekken&crs=webmercator&orderDir=asc&orderBy=value&md_returnfields=custom_attributes,station_id,station_no,station_name,ts_id,ts_name,stationparameter_name,ts_unitsymbol,parametertype_name&userId=PortalPublic"
response <- GET(request_url)
parsed <- fromJSON(content(response, "text"))$features$attributes

tidal_stations <-
  parsed %>% 
  filter(str_detect(station_name, "tij"), str_detect(station_no, "1066")) %>%
  select(-ts_id, -parametertype_name, 
         -stationparameter_name, -ts_name, 
         -ts_unitsymbol, -ts_value) %>%
  mutate(station_no_short = str_sub(station_no, 3)) %>%
  group_by(station_no_short) %>%
  mutate(ts_id = map(station_no_short, get_variables) %>% 
           map(as.tibble)) %>%
  unnest() %>%
  select(-station_name1, -station_no1)