"docker context ls" leaves behind zombie processes

lukas-lang commented 7 months ago

Description

Running docker context ls --format "{{json .}}" (as is done e.g. by the VS Code Dev containers extension) leaves behind a zombie process that is never cleaned up. This leads to a loss of available RAM & causes the system process to use a lot of CPU (due to all the page table entries).

Some observations:

The FindZombieHandles tool sees the process handles, but can't assign them to anything:

242093 total zombie processes.
8 total zombie threads.
  232 zombies held by Taskmgr.exe(1006460)
      133 zombies of WindowsTerminal.exe - process handle count: 133 - thread handle count: 0
      80 zombies of com.docker.cli.exe - process handle count: 80 - thread handle count: 0
      7 zombies of msedge.exe - process handle count: 7 - thread handle count: 0
      6 zombies of dllhost.exe - process handle count: 6 - thread handle count: 0
      1 zombie of SearchProtocolHost.exe - process handle count: 1 - thread handle count: 0
      1 zombie of PsExec.exe - process handle count: 1 - thread handle count: 0
      1 zombie of ctfmon.exe - process handle count: 1 - thread handle count: 0
      1 zombie of consent.exe - process handle count: 1 - thread handle count: 0
      1 zombie of conhost.exe - process handle count: 1 - thread handle count: 0
      1 zombie of clinfo.exe - process handle count: 1 - thread handle count: 0
  101 zombies held by svchost.exe(12312)
      34 zombies of python.exe - process handle count: 34 - thread handle count: 0
      29 zombies of Code.exe - process handle count: 29 - thread handle count: 0
      10 zombies of python.exe - process handle count: 10 - thread handle count: 0
      10 zombies of docker.exe - process handle count: 10 - thread handle count: 0
      9 zombies of python.exe - process handle count: 9 - thread handle count: 0
      8 zombies of msedge.exe - process handle count: 8 - thread handle count: 0
      1 zombie of msiexec.exe - process handle count: 1 - thread handle count: 0
  7 zombies held by msedge.exe(19676)
      7 zombies of msedge.exe - process handle count: 7 - thread handle count: 0
  6 zombies held by WindowsTerminal.exe(263716)
      3 zombies of ssh.exe - process handle count: 3 - thread handle count: 3
      3 zombies of OpenConsole.exe - process handle count: 3 - thread handle count: 0
  4 zombies held by Code.exe(992060)
      2 zombies of conhost.exe - process handle count: 2 - thread handle count: 0
      2 zombies of cmd.exe - process handle count: 0 - thread handle count: 2
  1 zombie held by LockApp.exe(645916)
      1 zombie of explorer.exe - process handle count: 1 - thread handle count: 0
  1 zombie held by OUTLOOK.EXE(12412)
      1 zombie of SearchProtocolHost.exe - process handle count: 1 - thread handle count: 0
  1 zombie held by PowerToys.exe(12388)
      1 zombie of PowerToys.ColorPickerUI.exe - process handle count: 1 - thread handle count: 0
  1 zombie held by svchost.exe(6536)
      1 zombie of explorer.exe - process handle count: 1 - thread handle count: 1
  1 zombie held by svchost.exe(2672)
      1 zombie of userinit.exe - process handle count: 1 - thread handle count: 0
Pass -verbose to get full zombie names.
Press any key to continue

RamMap shows tens of GB of memory used by "Page table", "System PTE", and "Unused":
RamMap shows that the culprit is indeed docker (the list is hundreds of thousands of entries long, with 1/3 "conhost.exe", "docker.exe" and "com.docker.cli" each - note also the PIDs that are in the millions by now):
When doing absolutely nothing, the System process eats up half a core of CPU:
Windows Performance Recorder confirms that this is due to stuff related to the page table:

Reproduce

Run docker context ls --format "{{json .}}" a bunch of times (docker engine is not running in my case)
Observe how the number of zombie handles goes up by approximately 3x-5x (depending on how it's run) number of calls to docker

Expected behavior

The processes should be cleaned up

docker version

Client:
 Cloud integration: v1.0.35+desktop.11
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:02 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.28.0 (139021)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.22
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.5.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.27GiB
 Name: docker-desktop
 ID: 3861acb3-ccc4-48dc-a7b1-718eaa00ea5d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

096291B3-D04B-4013-8FAF-A02527A5E4CB/20240421192036

Additional Info

No response

lukas-lang commented 7 months ago

Update: After some more debugging, I am now more confused than ever: It looks like the problem might after all not be specific to Docker, but rather a general issue on my system, i.e. starting any application will leave behind a zombie process. I will need to investigate further once I find some more time. In the meantime, any insights would of course be greatly appreciated

totoguy commented 6 months ago

@lukas-lang I've been having the exact same experience - thousands and thousands of zombie processes, ram getting eaten up, system process constantly around 5-15%, etc.

Pretty much ran all the tests you have, on RamMap + WPR, and reached the same results as in your screenshots. Only difference is the symbols I noticed in my WPR report were somewhat different, mostly variations of "CcUnpinRepinnedBcb".

I ended up finding out that the "docker context ls" command is called from both the mentioned VSCode extension, and Docker Desktop itself - I'm assuming for polling the container status, cpu usage, etc. Removing the extension only fixed part of the problem though, since most of the polling is still being done by Docker Desktop.

After reading your last comment I got curious and ended up finding your related post on the Framework forums. After trying the test described there (not on a clean install) I now suspect that this is indeed unrelated to docker, just a side effect due to docker desktop calling that process in loop.

I'm not even using the same laptop as you though (Asus X13 2022, AMD Ryzen 9 model), so not sure where to go from here.

totoguy commented 6 months ago

@lukas-lang This seems promising, will test out the driver downgrade in the next day or two. https://community.amd.com/t5/drivers-software/memory-leak-on-zen4/td-p/662281

lukas-lang commented 6 months ago

@totoguy Wow, nice find! Downgrading to version 23.10.2 of the AMD drivers does indeed seem to fix the issue

joelsodias commented 1 month ago

Hello there...

That's the same situation here, and I'm using an Intel(R) Core(TM) i5-1035G1 CPU

Does anybody have a clue how to avoid this situation?

docker_desktop

docker / for-win