`KeyError: 'station_id\\'` when exporting stations to file

[amotl]

Hi. Over at https://community.panodata.org/t/problems-to-create-get-the-stationsfile/258, @ohobby asked:

I need some advice on what else I might be doing wrong here.

luftdatenpumpe stations --source=postgresql://luftdatenpumpe@localhost/weatherbase --target=json.flex+stream://sys.stdout --target-fieldmap='key=station_id\|str,name=road_and_name_and_id' > $stationsfile

2022-12-12 21:14:54,690 [luftdatenpumpe.source.common        ] INFO   : Downloading without caching
2022-12-12 21:14:54,691 [luftdatenpumpe.commands             ] INFO   : Acquiring list of stations from network "ldi" with source "postgresql://luftdatenpumpe@localhost/weatherbase"
2022-12-12 21:14:54,716 [luftdatenpumpe.target.rdbms         ] INFO   : PostGIS version: POSTGIS="3.1.1 aaf4c79" [EXTENSION] PGSQL="130" GEOS="3.9.0-CAPI-1.16.2" PROJ="7.2.1" LIBXML="2.9.10" LIBJSON="0.15" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)"
2022-12-12 21:14:55,679 [luftdatenpumpe.commands             ] INFO   : Acquired #9499 stations
2022-12-12 21:14:55,679 [luftdatenpumpe.commands             ] INFO   : Will publish data to ['json.flex+stream://sys.stdout']
2022-12-12 21:14:55,679 [luftdatenpumpe.engine               ] INFO   : Configuring data sink "json.flex+stream://sys.stdout" with domain "stations"
2022-12-12 21:14:55,679 [luftdatenpumpe.engine               ] INFO   : Emitting to target data sinks, this might take some time

Traceback (most recent call last):
  File "/home/ldp/venv/bin/luftdatenpumpe", line 8, in <module>
    sys.exit(run())
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/commands.py", line 186, in run
    run_engine(options)
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/commands.py", line 327, in run_engine
    engine.process(data)
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/engine.py", line 86, in process
    target.flush(final=True)
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/target/stream.py", line 21, in flush
    self.handle.write(self.formatter(self.buffer))
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/target/json.py", line 43, in formatter
    value = station[str(key_right)]
KeyError: 'station_id\\'

[amotl]

@ohobby: We may want to post the outcome of our discussion here, so that others from the community can benefit from it.

[amotl]

I've just exercised the corresponding steps within the Luftdaten-Viewer Grafana walkthrough, and I am observing the same error. It looks like it works better when omitting the backslash.

export LDP_NETWORK=ldi
luftdatenpumpe stations \
  --source=postgresql://luftdatenpumpe@localhost/weatherbase \
  --target=json.flex+stream://sys.stdout \
  --target-fieldmap='key=station_id|str,name=road_and_name_and_id'

[amotl]

GH-49 has the corresponding fix. Thank you, @ohobby.

[amotl]

I am not sure if the backslash was needed within the cron file.

https://github.com/earthobservations/luftdatenpumpe/blob/0219120d98d37887dda848510afd92ca9da25612/etc/luftdaten-viewer.cron#L36

Let's keep this issue open, until this detail has been clarified. Will you let us know, @ohobby?