Closed tomkralidis closed 4 years ago
In GitLab by @tomkralidis on Jan 27, 2017, 10:17
mentioned in commit geomet@0a9bad6554a3043a6e85f4d5764477bd6bce3080
In GitLab by @tomkralidis on Jan 8, 2018, 12:31
changed title from add hydrometric observations to add {+real-time +}hydrometric observations
In GitLab by @alexandreleroux on Sep 17, 2019, 14:02
removed milestone
In GitLab by @tomkralidis on Sep 18, 2019, 09:48
moved from geomet#135
In GitLab by @tomkralidis on Sep 18, 2019, 09:48
assigned to @ahurka and unassigned @tomkralidis
In GitLab by @tomkralidis on Sep 18, 2019, 09:48
FYI I've moved this issue into msc-pygeoapi. Assigning to @ahurka to update on status.
In GitLab by @ahurka on Sep 26, 2019, 11:51
Realtime hydrometrics are being tested in an ElasticSearch index hydrometric_realtime
on geo-dev-01
. A realtime hydrometric loader, triggered by a Sarracenia plugin, has been filling the index for a few weeks. The loader/plugin parses all incoming files into GeoJSON, adds some bits of extra metadata, and upserts (update if already present, else insert) them straight into ElasticSearch.
The two types of data files we use (hourly and daily) share the same format and the loader/plugin doesn't distinguish between them. Hourlies are updated every 30-60 minutes, and provincial-level files are used so that only a few large files are handled. Dailies are updated around noon every day, but station-level files are used so that large numbers of small files are received.
To keep the index small an "overflow tool" runs every night at 1:00 am UTC as a cron job, which deletes anything in the index older than 30 days.
For more information see https://wiki.cmc.ec.gc.ca/wiki/GeoMet_2/Data/Hydrometric
.
In GitLab by @tomkralidis on Sep 26, 2019, 12:19
I met with Water Survey earlier today. Feedback:
FLOW
-> DISCHARGE
FLOW_SYMBOL_EN
-> DISCHARGE_SYMBOL_EN
FLOW_SYMBOL_FR
-> DISCHARGE_SYMBOL_FR
@ahurka can you make the same changes in our HYDAT loader as well? The next HYDAT release is mid-October 2019 at which point we can re-index with the updates.
properties.IDENTIFIER
format to station-number.ISO8601
format with timezone. Example: 04GB004.2019-08-27T00:01:00Z
In GitLab by @ahurka on Sep 26, 2019, 12:31
For 1) the fields in hydrometric_realtime
are already named DISCHARGE and DISCHARGE_SYMBOL(EN|FR), were they talking about the old hydrometric indices only?
For 2) We can do that, it will take a day for the index to fill itself back up again.
In GitLab by @tomkralidis on Sep 26, 2019, 13:42
@ahurka ah, right ok. Just the HYDAT loader then.
In GitLab by @tomkralidis on Nov 22, 2019, 12:30
FYI @ahurka's running prototype is in https://gccode.ssc-spc.gc.ca/ahurka/msc-pygeoapi/tree/realtime-dev
In GitLab by @tomkralidis on Dec 22, 2016, 08:17
Dataset Overview:
Setup
Configuration
etc/amqp.conf.in
WMS
WFS
Orchestration