Set up alerts from syslog

[harpermann]

• Errors & Warnings or just Errors
• List of alerts we want to page on?
• Possible tools
• https://github.com/prometheus-community/rsyslog_exporter
• https://grafana.com/docs/loki/latest/alert/

[harpermann]

Alerts from syslog are working with mtail Alerts are captured with simple Regex

@syslog { /adsb_actions/ { /WARNING (.)/ { mtail_adsb_actions_warning_total[$1]++ del mtail_adsb_actions_warning_total[$1] after 5m } /ERROR (.)/ { mtail_adsb_actions_error_total[$1]++ del mtail_adsb_actions_error_total[$1] after 5m }

ANY Warning or Error will be posted to Prometheus and Alertmanager as metrics. We have the option of graphing the metrics with Grafana. The metric will be deleted (from mtail, not Prom) after 5 minutes. If it is stall active, syslog is still posting warning or errors, the alert will got active again and post another alert. This may well be too often so we can adjust this after we get some production run experience or if we want to start slower.

• [ ] still need to go through all the syslog messages in adsb_actions and verify priorities. We may want to alert only on ERROR log entries but we also don't want to miss anything important.

Mtail configuration is in https://github.com/harpermann/brc_airport_prometheus/blob/main/adsb_actions.mtail

[harpermann]

This is checked in on harper-dev branch so we can talk about what might get merged. I've messed with the module paths so this branch may be broken for a moment. The modules work for me after failing to get the relative path ones to work.

A new module called adsb_logger.py has the global syslog config. If you change the level in that file, the config propagates across all modules. If you set the log level in the particular modules it will override the global syslog and output at the local level that specific module. You can set all logging to say global INFO but override one or more specific modules to DEBUG. You can mix and match levels in the hierarchy. I also changed everything to use logger. instead of logging. Let me know if there was a something about using logging instead of logger. I can't tell the difference.