search

owntracks / docker-recorder

Docker image for OwnTracks Recorder
151 stars 68 forks source link

When using systemd, HEALTHCHECK causes tons of syslog spam #35

Closed marcopaganini closed 4 years ago

marcopaganini commented 4 years ago

It appears systemd monitors the mounts on the system and reports on any mountpoint activity. For some reason, healthcheck in docker causes a temporary filesystem to be mounted and dismounted every time the recorder-health.sh script is called. This causes an large amount of syslog spam, making it difficult to see useful information.

Sample log message, repeated ad nauseam:

Mar 25 19:57:10 foo systemd[1]: run-docker-runtime\x2drunc-moby-d83a458ad894bcbdb570b0042d046a3ca2887635cf6d0dfc830c96de8e41ac70-runc.H9q28H.mount: Succeeded.

While using healthcheck is possibly a good idea, the vast majority of docker configurations I see do not use it. Also, it's arguable if monitoring of a container should be under docker or delegated to another system.

It would be a good idea to balance the usefulness of healthcheck vs the immense amount of syslog spam the feature generates, maybe turning the feature off until the problem is better understood.

jpmens commented 4 years ago

How about turning off the systemd spamming of syslog? Isn't that the better way to go about this?

Also, can you say whether this occurs on more than one Linux distribution (e.g. RHEL, CentOS, Debian, ..)?

marcopaganini commented 4 years ago

Turning it off in systemd would be ideal (in fact, why it logs "Success" on mounts evades me). Issue #6432 (systemd) requests a filter mechanism for logs in journald. That issue was opened in 2017 and is not going anywhere, unfortunatelly. Poettering was initially unconvinced we should do filtering at logging time .

In docker issue #679, user @gertjanklein suggests rsyslog filtering as a workaround. I haven't tested this (yet) but I'm most certain it will work.

Maybe a PR to add a note to the documentation? The amount of spam in the system logs is substantial.

jpmens commented 4 years ago

I very much appreciate your involvment and your investiagion. I've thought about this a bit, and agree with you.

Let's disable healthcheck.

If you would please provide a PR, I will apply it directly.

marcopaganini commented 4 years ago

My pleasure. I just sent you a PR for review. I put some comments in the Dockerfile for those who prefer to have healthcheck and/or are not using systemd. Thanks for maintaining this repo!

jpmens commented 4 years ago

owntracks/recorder:0.8.6-11 pushed a moment ago.