containers / conmon

An OCI container runtime monitor.
Apache License 2.0
420 stars 128 forks source link

podman with healthcheck fills journal with error #454

Open HerrFrutti opened 1 year ago

HerrFrutti commented 1 year ago

problem

My containers are healthy but my journal gets filled with error messages. The error seems to be related with healthchecks.

workaround

For now I disabled all healthchecks and the error disappeared.

logs

example line from my journal:

conmon[11328]: conmon 62de48d99d5858ab629f <error>: Unable to send container stderr message to parent Broken pipe

266649948-aee6269b-69e2-4ec6-846c-6f215ba270eb

additional info

conmon version 2.1.7
podman version 4.6.1
podman-compose version 1.0.6
OS: Fedora release 37 (Thirty Seven)

I'm using podman-compose to deploy my containers.

mufeedali commented 11 months ago

Any known workarounds for this other than just disabling the health checks?

hdonnay commented 11 months ago

This is still a problem on Fedora 39

mufeedali commented 11 months ago

I have a couple of health checks and memory usage for journald is way beyond normal due to this. Until this bug gets fixed, I might have to disable my health checks, which is kind of sad :(

benblasco commented 11 months ago

@mufeedali I am experiencing this issue too. How have you been disabling the health checks?

mufeedali commented 11 months ago

@benblasco by simply removing them from my podman-compose files

shyzus commented 11 months ago

@benblasco If you are dealing with images like the gitlab image which provides health checks by default. You can use this option to disable it --health-cmd=none

stryan commented 10 months ago

I've been noticing this problem on my OpenSUSE Leap 15.5 box. Containers are run through quadlet with podman 4.7.1.

symysak commented 10 months ago

I am also facing this problem with Ubuntu 22.04 LTS 64-bit. I am not using podman-compose.

raspberrypi@raspberrypi:~$ podman -v podman version 4.6.2

benblasco commented 10 months ago

@benblasco If you are dealing with images like the gitlab image which provides health checks by default. You can use this option to disable it --health-cmd=none

Thanks for sharing this info. I will have to figure out how to do this with quadlet!

benblasco commented 10 months ago

Going through the documentation it appears as though the healthcmd directive can only be used for container units, and not pod units. Is there a way to override the healthcmd for a pod unit? Am I just reading the documentation incorrectly?

rhatdan commented 10 months ago

I don't think healthcmds effect Pods, they are a container concept. Pod's only have an infra container, which does not use healthcmds. Individual containers within the pod can have healthcmds.

hdonnay commented 10 months ago

I ended up adding the following to my .container file:

[Service]
# The following is for a conmon bug where it emits warnings for expected behavior.
LogFilterPatterns=~Broken pipe

See systemd.exec(1) for more information. YMMV if your container expects to have Broken pipe in its logging output during normal operation.

daniarla commented 8 months ago

I'm having the same issue on debian unstable. Hoping for a fix soon!