OOM killed using haproxy image

[andreibreban]

Running haproxy using haproxy:2.2.14 docker image.

On high load/memory usage the master haproxy process gets oom-killed. The expected behaviour is that the haproxy container is restarted if the master process gets killed.

System docker version Client: Version: 19.03.6-ce API version: 1.40 Go version: go1.13.4 Git commit: 369ce74 Built: Fri May 29 04:01:26 2020 OS/Arch: linux/amd64 Experimental: false

Server: Engine: Version: 19.03.6-ce API version: 1.40 (minimum version 1.12) Go version: go1.13.4 Git commit: 369ce74 Built: Fri May 29 04:01:57 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.3.2 GitCommit: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6 runc: Version: 1.0.0-rc10 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd docker-init: Version: 0.18.0 GitCommit: fec3683

OS ID_LIKE="centos rhel fedora" PRETTY_NAME="Amazon Linux 2"

[TimWolla]

Running haproxy using haproxy:2.2.14 docker image.

You should use a newer version. 2.2.14 is 1.5 years old and affected by 290 fixed bugs: https://www.haproxy.org/bugs/bugs-2.2.14.html

The expected behaviour is that the haproxy container is restarted if the master process gets killed.

You should then configure the Docker container to do so.

[andreibreban]

Hi,

Thanks for your reply. Will update the version.

As it spawns multiple processes on config reload it is hard to track figure out the current main process as it keeps changing.

On the provided docker-entrypoint.sh tried something like:

...
exec "$@" & 
while [ -z "$(echo 'show info' | socat stdio /var/run/haproxy.sock 2>&1 | grep 'Connection refused' )" ]; do 
    sleep 15
    echo "checking proxy"
done
echo "haproxy stopping"

Unfortunately this causes other issues. I was wondering if someone else encountered same issue and if there are any solutions to it. Thanks,

[Bubbelb]

I'm running the latest alpine version (2.7.3) and I have the same issue. Host OS: Arch (linux-hardened kernel) Older docker (and containerd) versions run ok, but newer have issues. Working versions:

• docker: <= 20.10.23
• containerd: <= 1.6.16 (See edit)

Latest failed test with versions:

• docker: 23.0.1
• containerd: 1.6.19 (See edit)

Also seeing an issue with a samba container. Not checked what is happening there, but downgrading fixes both issues.

EDIT: Issue seems to lie in the docker package. Containerd updates to 1.70 without issues.

[Bubbelb]

Got time to get into the matter and found the problem. I found out that maximum TCP connections must be set at an appropriate number. So in the global section of your haproxy.conf, add the following: maxconn 60000

That did it for me.

[splatch]

@Bubbelb Thank you for note you left. It saved me from wasting yet-another hour of tinkering with this issue. I've noticed that my config which I initially used with haproxy 2.0.5 (I know it is old) stopped to work, as I purged docker images and re-fetched new image with same tag. I was trying to switch versions and all of them failed leaving very little information why it takes insane amount of resources. To make matter worse - standalone haproxy, without container surrounding, didn't suffer from this issue.

Once again, thank you very much!