Description

A memory leak happens on dockerd in Debian 12. dockerd by taking 120 MB then grows to take several GB of RAM over a few days. The leak is not caused by a leaking container, in my case they are extremely stable in RAM and I use deploy.resources.limit.memory on all containers to be extra safe.

Downgrading to 2.26 “fixes” the issue.

Impacted versions

I could not pinpoint exactly which combination of versions started trigerring the issue. But it happens with these versions:

# apt list --installed | grep -E "(docker|containerd)"

containerd.io/bookworm,now 1.7.22-1 amd64 [installed]
docker-buildx-plugin/bookworm,now 0.17.1-1~debian.12~bookworm amd64 [installed]
docker-ce-cli/bookworm,now 5:27.3.1-1~debian.12~bookworm amd64 [installed]
docker-ce-rootless-extras/bookworm,now 5:27.3.1-1~debian.12~bookworm amd64 [installed,automatic]
docker-ce/bookworm,now 5:27.3.1-1~debian.12~bookworm amd64 [installed]
docker-compose-plugin/bookworm,now 2.29.7-1~debian.12~bookworm amd64 [installed]

Steps To Reproduce

In Debian 12 bookworm (experienced on a server with no GUI)
with Docker Engine and Docker Compose installed using the apt repo 2 weeks ago
On start, dockerd takes 100 to 120 MB of RAM
Run docker compose up -d on a project with 10 containers with deploy limits to contain the total RAM usage to 800 MB max
After a day, in addition to the 600-800 MB of total RAM used by containers, dockerd takes around 2 GB of RAM
After 2 days, dockerd takes around 4 GB of RAM, exhausting my server memory (I know, small server, but the issue would still happen on any amount of RAM)

Workaround

Downgrading to Docker Compose 2.26 fixes the issue (no noticeable memory leak in the past 5 days):

apt install docker-compose-plugin=2.26.1-1~debian.12~bookworm
apt-mark hold docker-compose-plugin
systemctl restart docker

Compose Version

Docker Compose version v2.29.7

Docker Environment

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 12
  Running: 10
  Paused: 0
  Stopped: 2
 Images: 13
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-25-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 4.013GiB
 Name: noripyt-services
 ID: a61e68dc-c783-4943-b844-b84826566ce6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Anything else?

Previous report

This was reported as https://github.com/moby/moby/issues/48144 but the issue seems more relevant on Docker Compose, since downgrading Docker Compose works around the memory leak. Hence the current issue.

docker / compose

[BUG] Memory leak in 2.29.7 with Debian 12 #12181