search

earthobservations / luftdatenpumpe

Acquire and process live and historical air quality data without efforts. Filter by station-id, sensor-id and sensor-type, apply reverse geocoding, store into time-series and RDBMS databases, publish to MQTT, output as JSON, or visualize in Grafana. Data sources: Sensor.Community (luftdaten.info), IRCELINE, and OpenAQ.
https://luftdatenpumpe.readthedocs.io/
GNU Affero General Public License v3.0
35 stars 3 forks source link

Redis server frequently fails #7

Closed amotl closed 4 years ago

amotl commented 4 years ago

We and others noticed that while running Luftdatenpumpe in production, the Redis server frequently fails. On a subsequent invocation, luftdatenpumpe will fail likewise.

amotl commented 4 years ago

We added the following bits to our systemd configuration in order to mitigate this issue.

cat /etc/systemd/system/redis.service.d/restart.conf:

[Service]
# https://jonarcher.info/2015/08/ensure-systemd-services-restart-on-failure/
# Please run "systemctl daemon-reload" after making changes to this file.
Restart=always
RestartSec=3

Please run systemctl daemon-reload after adding this file or making changes to it.

While we recognize this is just a lousy workaround, it helped us tremendously for just keeping the whole system up and running. Saying this, we should keep in mind that we someday should investigate the root cause why Luftdatenpumpe is regularly crashing Redis.

amotl commented 4 years ago

We also updated the documentation and added the relevant information at [1].

[1] https://github.com/panodata/luftdatenpumpe/blob/master/doc/setup/redis.rst#running-in-production