search

earthobservations / luftdatenpumpe

Acquire and process live and historical air quality data without efforts. Filter by station-id, sensor-id and sensor-type, apply reverse geocoding, store into time-series and RDBMS databases, publish to MQTT, output as JSON, or visualize in Grafana. Data sources: Sensor.Community (luftdaten.info), IRCELINE, and OpenAQ.
https://luftdatenpumpe.readthedocs.io/
GNU Affero General Public License v3.0
34 stars 3 forks source link

[EEA] Ingest data from European Environment Agency's Air Quality Index #12

Open amotl opened 4 years ago

amotl commented 4 years ago

@wetterfrosch recently requested adding support for the European Environment Agency's European Air Quality Index [1].

[1] https://www.eea.europa.eu/themes/air/air-quality-index

/cc @gutzbenj

amotl commented 4 years ago

Some progress has been made with 4684fc4fcd4 and 9e73156e.

Now, when invoking a command like:

luftdatenpumpe stations --network=eea --country=CZ --reverse-geocode --progress

It will ingest the list of monitoring stations from EEA and display the gathered data in JSON format. Two details work well already:

  1. Filtering by country.
  2. Enriching location metadata by reverse geocoding through Nominatim.

[1] https://ptrace.hiveeyes.org/2019-10-01_eea-cz-stations.json

amotl commented 2 years ago

Problem

It looks like the URL https://ereporting.blob.core.windows.net/downloadservice/metadata.csv has been decommissioned.

$ luftdatenpumpe stations --network=eea --country=CZ --reverse-geocode --progress
2022-08-03 17:22:25,342 [luftdatenpumpe.source               ] INFO   : Applying filter: Munch({'country': ['CZ']})
2022-08-03 17:22:25,344 [luftdatenpumpe.commands             ] INFO   : Acquiring list of stations from network "eea" with source "api"
2022-08-03 17:22:25,344 [luftdatenpumpe.source.eea           ] INFO   : Requesting EEA at https://ereporting.blob.core.windows.net/downloadservice/metadata.csv
2022-08-03 17:22:25,515 [luftdatenpumpe.source.eea           ] ERROR  : Request failed: unknown
Traceback (most recent call last):
[...]
  File "/Users/amo/dev/earthobservations/sources/luftdatenpumpe/.venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: The specified blob does not exist. for url: https://ereporting.blob.core.windows.net/downloadservice/metadata.csv

Thoughts

The resources should be updated, and EEA ingest should be completed. From a quick research at [1-4], replacement resources might be [5] and [6].

[1] https://aqportal.discomap.eea.europa.eu/index.php/users-corner/ [2] https://discomap.eea.europa.eu/map/fme/ [3] https://discomap.eea.europa.eu/map/fme/AirQualityExport.htm [4] https://discomap.eea.europa.eu/map/fme/doc/UTDAirQualityDownloadGuide.pdf [5] https://discomap.eea.europa.eu/map/fme/metadata/PanEuropean_metadata.csv [6] https://discomap.eea.europa.eu/map/fme/latest/CZ_PM10.csv

amotl commented 1 year ago

It looks like the URL ... has been decommissioned.

Retrieving the station list from EEA has been fixed with #40, and is available per luftdatenpumpe 0.21.1.