search

earthobservations / luftdatenpumpe

Acquire and process live and historical air quality data without efforts. Filter by station-id, sensor-id and sensor-type, apply reverse geocoding, store into time-series and RDBMS databases, publish to MQTT, output as JSON, or visualize in Grafana. Data sources: Sensor.Community (luftdaten.info), IRCELINE, and OpenAQ.
https://luftdatenpumpe.readthedocs.io/
GNU Affero General Public License v3.0
35 stars 3 forks source link

Program croaks with KeyError: 'road_and_name_and_id' #5

Closed amotl closed 4 years ago

amotl commented 4 years ago

Some users running

luftdatenpumpe stations --network=ldi --source=postgresql://luftdatenpumpe@localhost/weatherbase  --target=json.flex+stream://sys.stdout --target-fieldmap='key=station_id,name=road_and_name_and_id'
2019-09-05 16:32:52,520 [luftdatenpumpe.commands             ] INFO   : Acquiring list of stations from network "ldi" with source "postgresql://luftdatenpumpe@localhost/weatherbase"
2019-09-05 16:32:52,700 [luftdatenpumpe.target.rdbms         ] INFO   : PostGIS version: POSTGIS="2.5.2 r17328" [EXTENSION] PGSQL="110" GEOS="3.7.1-CAPI-1.11.1 27a5e771" PROJ="Rel. 4.9.3, 15 August 2016" GDAL="GDAL 1.11.4, released 2016/01/25" LIBXML="2.9.1" LIBJSON="0.11" RASTER
2019-09-05 16:32:52,988 [luftdatenpumpe.commands             ] INFO   : Acquired #2195 stations
2019-09-05 16:32:52,989 [luftdatenpumpe.commands             ] INFO   : Will publish data to ['json.flex+stream://sys.stdout']
2019-09-05 16:32:52,989 [luftdatenpumpe.engine               ] INFO   : Configuring data sink "json.flex+stream://sys.stdout" with domain "stations"
2019-09-05 16:32:52,989 [luftdatenpumpe.engine               ] INFO   : Emitting to target data sinks, this might take some time

reported this error:

Traceback (most recent call last):
  File "/opt/luftdatenpumpe/bin/luftdatenpumpe", line 10, in <module>
    sys.exit(run())
  File "/opt/luftdatenpumpe/lib/python3.6/site-packages/luftdatenpumpe/commands.py", line 161, in run
    run_engine(options)
  File "/opt/luftdatenpumpe/lib/python3.6/site-packages/luftdatenpumpe/commands.py", line 302, in run_engine
    engine.process(data)
  File "/opt/luftdatenpumpe/lib/python3.6/site-packages/luftdatenpumpe/engine.py", line 79, in process
    target.flush(final=True)
  File "/opt/luftdatenpumpe/lib/python3.6/site-packages/luftdatenpumpe/target/stream.py", line 19, in flush
    self.handle.write(self.formatter(self.buffer))
  File "/opt/luftdatenpumpe/lib/python3.6/site-packages/luftdatenpumpe/target/json.py", line 44, in formatter
    value = station[str(key_right)]
KeyError: 'road_and_name_and_id'
amotl commented 4 years ago

The database field road_and_name_and_id is provided through the database view called ldi_network [1].

So, there might this very command be missing:

luftdatenpumpe database --network=ldi --target=postgresql://luftdatenpumpe@localhost/weatherbase --create-view --grant-user=grafana

which is described on the documentation page at [2].

[1] https://github.com/hiveeyes/luftdatenpumpe/blob/0.16.0/luftdatenpumpe/target/rdbms.py#L317 [2] https://github.com/hiveeyes/luftdatenpumpe/blob/master/doc/setup/ldview-databases.rst

amotl commented 4 years ago

We have been signaled this issue has been resolved.