Open TheMeier opened 2 months ago
Hello @TheMeier,
I think the exit of the exporter is related to the termination of your rsyslog process rsyslog.service: Main process exited, code=killed, status=7/BUS
.
The exporter is started by rsyslog itself via omprog
(doc)
If the program terminates, it is re-started. If rsyslog terminates, the program’s stdin will see EOF. The program must then terminate.
The log input ended, exiting normally
is produced here:
https://github.com/prometheus-community/rsyslog_exporter/blob/1b156170a378081348920102ebc026c45c1462ad/exporter.go#L205-L218
And as said in the bufio's Scanner.Scan():
It returns false when there are no more tokens, either by reaching the end of the input or an error. After Scan returns false, the Scanner.Err method will return any error that occurred during scanning, except that if it was io.EOF, Scanner.Err will return nil
From this analysis, I think rsyslog terminates, send a EOF
to the exporter, then exit normally (else you would have the error reading input
log before the exit). So the exporter is not involved in the termination of rsyslog here.
I did some research on the Internet, and it seems your issue is the same as here which highlight this issue https://github.com/systemd/systemd/issues/24320 This issue appears in v250, fixed in v255. Bookworm is running v252, and bullseye v247.
Oooooh. Thank you so much @aleroyer. The issue links are very welcome. I will try to validate this ASAP. Sadly bookworm-backports only ships systemd v254 currently.
Versions: rsyslog_exporter: 1.1.0 rsyslog: 8.2302.0
On Debian bookworm we see infrequent restarts of rsyslog triggered by the exporter exiting normally without any warning or error. The journal entries when this happens look like this:
The
Forwarding to syslog missed xxx messages
is already addressed, I think it is unrelated. What is suspicious is that every time theses terminations happen 3-4 times the reload is logged by rsyslog-imjournal-silent
is not setWe run the same setup also on Debian bullseye systems (rsyslog 8.2102.0) and do not see these terminations there.