add real-time hydrometric observations

[tomkralidis]

In GitLab by @tomkralidis on Dec 22, 2016, 08:17

Dataset Overview:

• base: http://dd.weather.gc.ca/hydrometric
• daily: last 30 days + current day incomplete
• hourly: last 2 days + current day incomplete
• contact Lingling Li

Setup

Configuration

• update sr_subscribe etc/amqp.conf.in
• setup layers
  • /Canadian Hydrology/Hydrometric Observations
  • /Canadian Hydrology/Hydrometric Monitoring Stations
• create .mcf (@alexandreleroux)
• needs YAML update
• single layer to be offered with two modes (WMS/WFS)

WMS

• display latest obs
• styles: 2 CLASSes
  • water level
  • discharge
• annotations at certain zoom levels?
• consult with Lingling

WFS

• all data
• WaterML2 MapServer template

Orchestration

• station list: create/setup
• aggregate all results into single Shapefile
  • convert local times to UTC?

[tomkralidis]

In GitLab by @tomkralidis on Jan 27, 2017, 10:17

mentioned in commit geomet@0a9bad6554a3043a6e85f4d5764477bd6bce3080

[tomkralidis]

In GitLab by @tomkralidis on Jan 8, 2018, 12:31

changed title from add hydrometric observations to add {+real-time +}hydrometric observations

[tomkralidis]

In GitLab by @alexandreleroux on Sep 17, 2019, 14:02

removed milestone

[tomkralidis]

In GitLab by @tomkralidis on Sep 18, 2019, 09:48

moved from geomet#135

[tomkralidis]

In GitLab by @tomkralidis on Sep 18, 2019, 09:48

assigned to @ahurka and unassigned @tomkralidis

[tomkralidis]

In GitLab by @tomkralidis on Sep 18, 2019, 09:48

FYI I've moved this issue into msc-pygeoapi. Assigning to @ahurka to update on status.

[tomkralidis]

In GitLab by @ahurka on Sep 26, 2019, 11:51

Realtime hydrometrics are being tested in an ElasticSearch index hydrometric_realtime on geo-dev-01. A realtime hydrometric loader, triggered by a Sarracenia plugin, has been filling the index for a few weeks. The loader/plugin parses all incoming files into GeoJSON, adds some bits of extra metadata, and upserts (update if already present, else insert) them straight into ElasticSearch.

The two types of data files we use (hourly and daily) share the same format and the loader/plugin doesn't distinguish between them. Hourlies are updated every 30-60 minutes, and provincial-level files are used so that only a few large files are handled. Dailies are updated around noon every day, but station-level files are used so that large numbers of small files are received.

To keep the index small an "overflow tool" runs every night at 1:00 am UTC as a cron job, which deletes anything in the index older than 30 days.

For more information see https://wiki.cmc.ec.gc.ca/wiki/GeoMet_2/Data/Hydrometric.

[tomkralidis]

In GitLab by @tomkralidis on Sep 26, 2019, 12:19

I met with Water Survey earlier today. Feedback:

1. change the following property names:
   • FLOW -> DISCHARGE
   • FLOW_SYMBOL_EN -> DISCHARGE_SYMBOL_EN
   • FLOW_SYMBOL_FR -> DISCHARGE_SYMBOL_FR

@ahurka can you make the same changes in our HYDAT loader as well? The next HYDAT release is mid-October 2019 at which point we can re-index with the updates.

2. change ES ID and GeoJSON properties.IDENTIFIER format to station-number.ISO8601 format with timezone. Example: 04GB004.2019-08-27T00:01:00Z

[tomkralidis]

In GitLab by @ahurka on Sep 26, 2019, 12:31

For 1) the fields in hydrometric_realtime are already named DISCHARGE and DISCHARGE_SYMBOL(EN|FR), were they talking about the old hydrometric indices only?

For 2) We can do that, it will take a day for the index to fill itself back up again.

[tomkralidis]

In GitLab by @tomkralidis on Sep 26, 2019, 13:42

@ahurka ah, right ok. Just the HYDAT loader then.

[tomkralidis]

In GitLab by @tomkralidis on Nov 22, 2019, 12:30

FYI @ahurka's running prototype is in https://gccode.ssc-spc.gc.ca/ahurka/msc-pygeoapi/tree/realtime-dev