search

kumina / postfix_exporter

A Prometheus exporter for Postfix.
Apache License 2.0
241 stars 140 forks source link

Don't bail out from systemd log reader when reaching the end #90

Closed Ma27 closed 1 year ago

Ma27 commented 2 years ago

The log search starts at the current time, i.e. the end of the journal (SeekTail() / SeekRealtimeUsec) and thus runs into an EOF.

Not a big deal because the code takes care of that, but the log-reading loop is exited then and further messages are ignored.

Potentially fixes #55.

elsgaard commented 2 years ago

The log search starts at the current time, i.e. the end of the journal (SeekTail() / SeekRealtimeUsec) and thus runs into an EOF.

Not a big deal because the code takes care of that, but the log-reading loop is exited then and further messages are ignored.

Potentially fixes #55.

I made a build with the PR, and deployed the exporter to a couple of our production servers (RHEL 8), but they went to 100% CPU until i stopped the exporters again 15 min later. Has anybody else seen this ?

ktosiek commented 2 years ago

I don't think this is the right fix: log sources should wait for new data, and handle basic problems inside Read (at least that's what FileLogSource is doing by using a tailer).

sd_journal_next doesn't wait for new data, the fix is to .Wait() if sd_journal_next returns 0. See the example in sd_journal_wait man page: https://manpages.debian.org/testing/libsystemd-dev/sd_journal_wait.3.en.html#EXAMPLES

Rudd-O commented 1 year ago

We need this fix too, and sadly the CPU spinning problem is a blocker for us.

Ma27 commented 1 year ago

Yeah I only tested it on a mailserver which doesn't have too much traffic (and relatively few logs).

However I think that it's perhaps better to just use loki for that purpose (and define alerts in Grafana), at least that's what I'm planning to do these days. Since I'm not planning to find a better solution, I'll close this now, sorry.