Open knuclechan opened 2 months ago
i have the same issue since 2 months… im just using nginx proxy manager in docker. Win 11 Pro
Please inform me, if u find a solution :)
we fixed a memory leak with 4.27.2 - can you confirm if the leak is present after restarting Docker Desktop?
we fixed a memory leak with 4.27.2 - can you confirm if the leak is present after restarting Docker Desktop?
my currently version is 4.27.2 :/
I've updated to v4.28 The leak continues to happen. But it seems that after I closed the containers and docker desktop, the leak cleans up itself. The page table's size reduced from 4G to 400mb I am not 100% sure yet. Will need a few days to observe.
Previously, the leak remains there after close the docker desktop. I need to restart the computer everyday
v4.28.0 have the same problem
Huge amount of com.docker.cli and docker.exe page table entries with PC uptime of a single day and half a day of containers running, stopping engine doesn't help the issue.
Unsure if this is caused by the same issue or not, but I've had this issue with docker for some time, but it's so intermittent that by the time you try and catch it, the system crashes due to lack of memory. Normally I'm not interacting with, or using Docker at the time. I wasn't interacting with Docker this morning.
Docker Desktop (windows) v4.28.0, Ryzen 5 3600, 32gb RAM. WSL2.
First sign I noticed something was wrong, was Visual Studio complained about not having enough RAM to debug - quickly opened Task Manager and noticed (before it too, locked up) that I had 483 background processes. My baseline is 129. There was around 98% RAM usage.
I managed to terminate Chrome and Visual Studio, which gave me enough memory back that Task Manager would load properly (TIL it has a "low memory" mode!). I then managed to quit Docker Desktop by using the tray icon - and after what seemed like an age - it quit, and memory usage dropped to around 30%.
Apologies for the rubbish images, the screenshot utility wasn't working and the computer was on the verge of crashing! I believe I have a copy of the logs directory if you can tell me what you need.
Edit: To add, I'm pretty sure I had no containers running at the time.
Hi folks, this is Cesar Talledo, I am developer at Docker. Thanks for finding the issue and apologies for the inconvenience it's caused and for the belated response.
From the problem description above, it's clear there's a bug somewhere that is causing Docker Desktop (or some other program) to unnecessarily spawn lots of com.docker.cli
processes, which over time are eating up host resources (e.g., memory).
However, I tried reproducing on a Windows host with Docker Desktop 4.27.2 and 4.28 (WSL), but haven't been able to do so. So there must be something different between your environment and mine that's triggering the bug.
@sparxooo: if possible, could you capture and upload a Docker Desktop diagnostics bundle so I can get more info about your environment? It would be best if you can capture the bundle as you start seeing the number of com.docker.cli
processes growing without reason, but before your machine runs out of resources.
@knuclechan: thanks for capturing the diagnostics bundle for Docker Desktop. Unfortunately because of my belated response to this, the bundle is no longer available :(
Also, any other info you can provide regarding Docker Desktop config would be helpful; e.g.,:
1) Does the problem reproduce consistently?
2) Do you have any Docker Desktop extensions installed?
3) Is Docker Desktop's resource saver on when the problem occurs?
4) Is Docker Desktop integrated with your WSL distro (Settings->Resources->WSL integration)?
5) Do you have VS-Code + Docker extension installed? If so, what happens if you close VS-Code?
Thanks!
In case the problem is related to the interaction of docker stats --no-stream
with Docker Desktop's resource saver mode, here's an unofficial Docker Desktop 4.30 pre-release build that has a fix:
https://desktop-stage.docker.com/win/main/amd64/146335/Docker%20Desktop%20Installer.exe
If possible, please check if the problem reproduces with this build or not.
Thanks!
Stumbled upon this prerelease by chance and it fixes most the memory issues I had the past few weeks. 👍🏻
Stumbled upon this prerelease by chance and it fixes most the memory issues I had the past few weeks. 👍🏻
Thanks @luastoned for trying it out. Do you know if the memory issues you had were related to lots of com.docker.cli
processes eating up host resources? (see comments above).
@ctalledo RAMMap had my page table at ~5GB before the fix, now it's closer to ~1GB.
com.docker.cli
, conhost.exe
and docker.exe
still take up 90% of the list in the Processes tab though.
I did notice that vmmem
steadily consumes up to max memory allowed via .wslconfig
, freeing is only possible via empty working sets in RAMMap..
Thanks @luastoned.
com.docker.cli, conhost.exe and docker.exe still take up 90% of the list in the Processes tab though.
Mmm ... so you are still seeing lots of com.docker.cli
processes with the DD 4.30 pre-release build I posted above?
I did notice that vmmem steadily consumes up to max memory allowed via .wslconfig, freeing is only possible via empty working sets in RAMMap..
That could be because the Linux kernel inside the WSL VM steadily fills up it's kernel buffer cache, without releasing the associated memory back to the host (because it does not know it's running in a VM). Have you tried enabling the autoMemoryReclaim feature in WSL?
autoMemoryReclaim string disabled Automatically releases cached memory after detecting idle CPU usage. Set to gradual for slow release, and dropcache for instant release of cached memory.
If not, try setting it to gradual
and see if that helps.
My current DD version is 4.30.0 (146335)
After starting my machine this morning I dumped the process list in RAMMap (~5 minutes uptime):
45k process list entries, 42k are docker.exe
, com.docker.cli
and conhost.exe
I do have autoMemoryReclaim
set to dropcache
but I'll try gradual
now.
Thanks @luastoned .
45k process list entries, 42k are docker.exe, com.docker.cli
Mmm ... let me see if I can repro with RAMMap.
I do have autoMemoryReclaim set to dropcache but I'll try gradual now.
I don't think that will make a difference unfortunately.
Hi @luastoned,
45k process list entries, 42k are docker.exe, com.docker.cli
I tried reproducing by starting DD 4.30.0 (146335) on my Windows hosts and leaving it idle for 5 minutes, then checked RAMMap. However I am not able to repro. I only see a handful of Docker Desktop and com.docker.* processes.
So there's something different in your environment that must be spawning all those com.docker.cli processes.
Could you upload a Docker Desktop diagnostics bundle please?
Also, what Docker Desktop extensions (if any) do you have installed? And do you have any VS-Code extensions that could be invoking Docker commands?
Thanks again!
Hi @ctalledo
I did more digging into that issue and it seems a recent update modified Windows power plans, which turned fast boot on again (ie. not clearing the page table, etc). After solving this there are no more docker related entries after a reboot, so I will monitor the growth/usage today and update this message later on.
Well, after a day of work the page table process count is ~52k and docker related things add up to 45k :(
Hi @ctalledo, apologies for not responding earlier. I will try and get a diagnostics bundle sorted if it does it again - but it's very intermittent.
Do you wish me to update to the v4.30 version or remain on the version i'm on, just in case I can repro it?
Cheers,
Thanks @sparxooo for the response.
Do you wish me to update to the v4.30 version or remain on the version i'm on, just in case I can repro it?
If you get a chance to reproduce the "lots of com.docker.cli processes" and capture the Docker Desktop diagnostics bundle before the machine runs out of memory, that would be amazing. And if that's the case, try the v4.30 version I added and check if that resolves the issue please.
Thanks for all the help!
Hello @ctalledo ,
Facing the similar issue with Docker Desktop 4.29.0 (145265).
As you pointed to some extension, I disabled VS Code extension of Docker. The call to docker cli got reduced, nearly half. Then I closed VS Code before 2 hours. The call to docker cli every few seconds has stopped. Zero call to docker and conhost too.
I will try opening VS Code and monitor again. And if I can create capture as suggested in previous post, I will do and post here.
@ctalledo,
Local diagnostic run says "[FAIL] DD0004: is the Docker engine running? unable to create docker client: protocol not available
But 3 containers are already running.
@ctalledo Here are the diagnostic ids
1] 7BE15776-926F-4950-8DA6-9CF88046EC6C/20240419085152
created before 5 hours appx. [RAM used was around 8GB]
2] 7BE15776-926F-4950-8DA6-9CF88046EC6C/20240419135801
created just now [RAM used is 14.3GB]
Here is the RamMap saved for your review.
Hi @jd4u,
Thank you very much for the additional info.
As you pointed to some extension, I disabled VS Code extension of Docker. The call to docker cli got reduced, nearly half. Then I closed VS Code before 2 hours. The call to docker cli every few seconds has stopped. Zero call to docker and conhost too.
OK so that means the problem is likely caused by an interaction between VSCode and Docker Desktop.
Would you mind trying with the 4.30 prelease build I posted above?
Local diagnostic run says "[FAIL] DD0004: is the Docker engine running? unable to create docker client: protocol not available
Thanks for capturing the diagnostics, and please ignore that error (it's a bug in the diagnostics gathering that will be fixed in the next Docker Desktop release).
@ctalledo ,
There is no difference even with 4.30.0. The machine is restarted and initial ram usage was 3.6GB. Now 10+GB visible in screenshot. VS Code, Docker Desktop, Chrome and explorer are only programs running.
In 40 minutes there is a huge list of com.docker.cli, conhost, cmd, docker and relatively smaller list of wsl in RamMap
Hi @jd4u,
There is no difference even with 4.30.0. The machine is restarted and initial ram usage was 3.6GB. Now 10+GB visible in screenshot. VS Code, Docker Desktop, Chrome and explorer are only programs running.
In 40 minutes there is a huge list of com.docker.cli, conhost, cmd, docker and relatively smaller list of wsl in RamMap
Thanks for trying that experiment, I appreciate it.
So this means the problem is triggered by the VS-Code Docker Desktop Extension, and likely not related to Docker Desktop's resource saver mode. Let me try harder to reproduce locally so I can further debug.
Thanks again for all the help!
While using 4.30.0 with VS Code, the VS Code Docker Extension is disabled. The Docker extension is from Microsoft with version 1.29.0 .
(To review further, today stopped keeping open the terminal login session that started all containers.)
Sorry for the delay getting back to you.
Today at approx 1630-1635 (GMT+1), Docker was in "Resource Saver" mode, and the engine was paused. I spotted the influx of "docker" --stats processes with the increase in memory and sluggishness of the computer and also Task Manager. Generating a diagnostic bundle appeared to kill all of the "docker" --stats processes... and resumed the engine.
VS Code was not running at the time, only Chrome and Word.
As said previously, I'm still on v4.28 so I will now update to v4.30 and see if the problem reoccurs.
Diag id: 4E2EBDAD-12B0-4893-8641-E1727F59DB93/20240424153541
Even with "Resource Saver" mode is disabled, the issue persists...!!! (Docker Desktop v. 4.30.0, VS Code, VS Code Docker Extension Disabled, Terminal WSL2 Linux Session On)
Thanks @sparxooo and @jd4u for the additional info, very helpful. I am still debugging it but I agree it's not related to Docker Desktop's resource saver feature.
@jd4u : can you capture and upload a Docker Desktop diagnostics bundle please? That will really help us root-cause since I can't reproduce locally yet.
Also, when the problem occurs:
1) What happens if you run docker run --all --no-trunc --no-stream
from a terminal? Does it work correctly?
2) What happens if you stop all containers? e.g., docker stop -t0 $(docker ps -aq)
followed by docker rm $(docker ps -aq)
Thanks again!
7BE15776-926F-4950-8DA6-9CF88046EC6C/20240426143507
Following are result of
- What happens if you stop all containers? e.g., docker stop -t0 $(docker ps -aq) followed by docker rm $(docker ps -aq)
- What happens if you run docker run --all --no-trunc --no-stream from a terminal? Does it work correctly?
--no-trunc
, --all
, --no-stream
: all these are unknown flags
--no-trunc
, --all
, --no-stream
: all these are unknown flags
docker compose up
is working as expected. (without those flags)
jd@ulara:~/lirt$ docker version
Client:a:~/lirt$
Cloud integration: v1.0.35+desktop.13
Version: 26.0.1
API version: 1.45
Go version: go1.21.9
Git commit: d260a54
Built: Thu Apr 11 10:52:27 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Desktop
Engine:
Version: 26.0.1
API version: 1.45 (minimum version 1.24)
Go version: go1.21.9
Git commit: 60b9add
Built: Thu Apr 11 10:53:39 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.31
GitCommit: e377cd56a71523140ca6ae87e30244719194a521
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
jd@ulara:~/lirt$
Thanks @jd4u!
--no-trunc, --all, --no-stream: all these are unknown flags
Sorry meant to say docker stats --no-trunc --all --no-stream
.
Output of docker stats --no-trunc --all --no-stream
jd@ulara:~/lirt$ docker stats --no-trunc --all --no-stream
CONTAINER ID
NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
4dc57642e373e37d4edee053796d62b9f46081a39c371c7fab9e2c0b928efdd8 lirt-laravel.test-1 0.05% 73.3MiB / 1.926GiB 3.72% 3.5kB / 936B 795kB / 4.1kB 3
89af0dcb633f6ea61f4f2525dfb55f250e5b4256b18c603825522b175425592c lirt-phpmyadmin-1 0.00% 13.98MiB / 1.926GiB 0.71% 3.76kB / 936B 274kB / 24.6kB 6
59581d2b459e1c64f31968472a2ae67def6bfd5fe429bba43cc0b41943637ac1 lirt-mysql-1 0.51% 377.9MiB / 1.926GiB 19.16% 3.96kB / 936B 2.6MB / 23.2MB 38
jd@ulara:~/lirt$
NOTE: wslconfig is set to use only 2GB memory max.
# Settings apply across all Linux distros running on WSL 2
[wsl2]
memory=2GB
processors=4
swap=2GB
kernelCommandLine="systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all"
[experimental]
autoMemoryReclaim=gradual
Thanks again @jd4u.
Question: did you execute the docker stats
command after the problem occurs?
I other words, I am interested in understanding the output of docker stats --no-trunc --all --no-stream
after the problem occurs (i.e., after you start seeing lots of leaked com.docker.cli
processes).
Problem: Windows 11 RAM gets consumed gradually within 2-3 days. (32GB DDR5 - 30.7 GB Allocated)
docker stats
is run today on 24 hours usage with 14.9GB (right now 16.8GB) consumed RAM.
stats
command may not fetch any RAM overuse!!!docker stats
is run within WSL2 Ubuntu. So all stats will remain within 2GB usageRight now, memory usage is 28.1GB/30.7GB. Facing issue of slow movements and screen displays...
Anything need be tested in this state, let me know...
MEMORY: CPU:
Diagnostics ID 7BE15776-926F-4950-8DA6-9CF88046EC6C/20240429155503
jd@ulara:~/lirt$ docker stats --no-trunc --all --no-stream
CONTAINER ID
NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
ee642fb5b5a3187078afb303fdc15b6ffa2b07d2bddc54990db962edb7972b0e lirt-laravel.test-1 0.05% 84.06MiB / 1.926GiB 4.26% 3.57MB / 31.9MB 658MB / 136MB 3
05d547099f148ca765b09c1eff9cb13fc045fea8d931066d1e49fff98b6dd868 lirt-phpmyadmin-1 0.00% 39.45MiB / 1.926GiB 2.00% 1.32MB / 1.1MB 172MB / 74.1MB 10
e1f36e90c106d708857a3f64d89d4c5be7843cb526d633055f596f302e2a5820 lirt-mysql-1 0.36% 102.4MiB / 1.926GiB 5.19% 2.75MB / 3.32MB 577MB / 1.07GB 40
jd@ulara:~/lirt$
The below is result of the program FindZombieHandles (Referred from: why is my memory at 65 usage when im not running any programs)
Note: Docker Desktop is quit and not found in Task Manager details tab.
svchost.exe (10380) is "Program Compatibility Assistant Service"
Below is the rerun of FindZombieHandles, after staring Docker Desktop and working for few minutes...
Hi @jd4u, @sparxooo,
Thanks for uploading the diagnostics bundles, very helpful.
Hey I seems you both have the Kubescape and Telepresence extensions installed; if so, would you mind removing (i.e., uninstalling) all extensions and restarting Docker Desktop to see if the problem goes away?
Thanks!
@ctalledo , Docker Desktop Extensions are not enabled.. Docker Kubernetes is not enabled in settings. Also Resource Saver is not enabled.
This is since last reinstall of v.4.30.0
@ctalledo , Found another VS Code Extension "Dev Containers" which was making call to docker. Disabled it and memory consumption remained static, reduced while closing Chorme instances!!!
Found similar issue while searching google where the "Dev Container" behaviour is noted. Same found on my machine, so disabled and worked for 2 hours. Monitored using procmon before and after disabling. Current memory usage is 28.3GB/30.7GB. (Restart is needed now. after 24 hours working, I will report back here)
Current view is, it seems, the call to docker.exe by anyone causing the issue of memory leaks.
Hi @jd4u, thanks for the update, much appreciated.
Found another VS Code Extension "Dev Containers" which was making call to docker. Disabled it and memory consumption remained static, reduced while closing Chorme instances!!!
That's interesting; let me try to repro with the VSCode + Dev Container extension.
Let me try to repro with the VSCode + Dev Container extension.
I tried with that VSCode extension and while I do see a few more com.docker.cli
processes, the number of such processes remains stable and is not growing. So I can't repro still :/
Hi @jd4u, @sparxooo,
Thanks for uploading the diagnostics bundles, very helpful.
Hey I seems you both have the Kubescape and Telepresence extensions installed; if so, would you mind removing (i.e., uninstalling) all extensions and restarting Docker Desktop to see if the problem goes away?
Thanks!
I don't believe I have any extensions enabled - if I do - they're hidden. The "Enable Kubernetes" setting is disabled too.
I am wondering if I have a different issue to jd4u - my problem seems to develop very quickly and can result in a full system lock up and crash, whereas jd4u's is more total memory usage?
Hello, I'm reporting the reproduction. I encountered the same issue as the original post and have confirmed that disabling the two VSCode extensions, "Dev Containers" and "Docker" stops the abnormal occurrences of docker-related processes seen in RamMap.
@shocker-0x15 , try disabling the Dev Containers. VS Code extensions make calls to docker very frequently, this way the issue is noticeable within days. After stopping two VS Code extensions, the memory usage is relatively 80% slower in consuming memory.....
I'm yet to get off docker usage totally, so unable to say it's docker or some other app causing the memory leaks.
Hi @shocker-0x15,
Hello, I'm reporting the reproduction. I encountered the same issue as the original post and have confirmed that disabling the two VSCode extensions, "Dev Containers" and "Docker" stops the abnormal occurrences of docker-related processes seen in RamMap.
VSCode extensions:
- docker: v0.362.0
- Dev Containers: v0.362.0
- Docker Desktop: 4.29.0 (145265)
- Windows 11 Pro 23H2 (22631.3527)
Thanks for reporting.
I have both extensions installed but unable to repro on my end (i.e., I see only a few com.docker.cli
processes and they are not growing over time); I have:
Have you tried with just one of the extensions (i.e., either the Docker extension or the Dev Container extension), to see which one is the culprit?
Also, what's the version of the Docker extension you have? You said v0.362.0 but that appears incorrect since latest version is 1..29.1.
Thanks again!
It doesn't seem to be linked here yet but https://github.com/docker/for-win/issues/14027 appears to be the same issue. Over there they seem to have identified specific AMD drivers as being the cause of this issue.
@BobVul , Thank you for pointing to possible cause. Machine is AMD 8600G and software installed was 23.10.*.
Now installed the latest version released on 25 Apr 2024, I will review and let you know the result here by tomorrow. (Though I feel your suggested will solve this issue)
Description
Docker related processes eventually used up the computer's memory after running for a day or 2. I found out that it is related to the Page Table in memory by using RamMap. There's a huge amount (over a thousand) of
docker
conhost
com.docker.cli
cmd
remains in the page table. Each of them eats up 36k memory.I try close all the containers, docker desktop, wsl and all other applications, but the memory is not released at all.
I suspect there's a memory leak somewhere related to docker desktop
I am using Windows 10 Pro
Below is the memory usage:
All My memory is used up, but in process tab, I didn't use more than 2GB memory.
Unreasonable amount of memory is used by Page Table
This is the first page of processes sorted by pid. It shows there's a lot of
docker
conhost
com.docker.cli
cmd
There's way way way way more when I sorted them by name. Each of them eats up 36K Page Table memory. I don't list other process one by one.
Reproduce
Just run docker desktop for some days
Expected behavior
No response
docker version
docker info
Diagnostics ID
50A47C85-7750-446A-9865-A69828201F7D/20240223100956
Additional Info
No response