Open ramki88 opened 1 year ago
@ctalledo I gave Docker Desktop for Mac version 4.24.2 a go with Resource Saver Mode enabled and it is still not working for me (that is, Docker CLI commands do not cause Resource Saver Mode to exit).
Diagnostics ID: EF9239CE-8F64-4009-B01E-47A89F3F68A5/20231013032546 (uploaded)
@ctalledo I gave Docker Desktop for Mac version 4.24.2 a go with Resource Saver Mode enabled and it is still not working for me (that is, Docker CLI commands do not cause Resource Saver Mode to exit).
Diagnostics ID: EF9239CE-8F64-4009-B01E-47A89F3F68A5/20231013032546 (uploaded)
Hi Chip (@lgw4), thanks again for reporting.
I took a look and it's so strange, on exit from resource saver Docker Desktop restarts the Linux VM, but it seems to get stuck (it never actually starts). I don't see anything obvious in the diagnostic bundle as to why that occurs, but I will keep investigating.
I did notice that you enabled Rosetta on DD; would you mind disabling it and retrying? I want to see if there's a connection between Rosetta being enabled and the DD VM not restarting properly.
Thanks!
Also still happening (freezing, not crashing) on my Intel Mac with Ventura 13.5.2 and 4.24.2: Diagnostics- ID: 95E98686-D4BB-4DCF-B0CA-0F6B96B45D98/20231013171419
@ctalledo I gave Docker Desktop for Mac version 4.24.2 a go with Resource Saver Mode enabled and it is still not working for me (that is, Docker CLI commands do not cause Resource Saver Mode to exit).
Diagnostics ID: EF9239CE-8F64-4009-B01E-47A89F3F68A5/20231013032546 (uploaded)
Hi Chip (@lgw4), thanks again for reporting.
I took a look and it's so strange, on exit from resource saver Docker Desktop restarts the Linux VM, but it seems to get stuck (it never actually starts). I don't see anything obvious in the diagnostic bundle as to why that occurs, but I will keep investigating.
I did notice that you enabled Rosetta on DD; would you mind disabling it and retrying? I want to see if there's a connection between Rosetta being enabled and the DD VM not restarting properly.
Thanks!
@ctalledo I disabled Rosetta, but the problem persists.
Diagnostics ID: EF9239CE-8F64-4009-B01E-47A89F3F68A5/20231015014341 (uploaded)
I have the same problem
After a long time, the bug is creeping back again. This time it's a bit weird. (ps, images ps -a, info stats) commands work. But pull, push, rmi, start does not work. 😮
Diag ID: 073CCF80-81D3-426A-A5F1-1B13A4481DE2/20231017122146
I think I also face this problem. Details here: https://forums.docker.com/t/docker-desktop-on-mac-unable-to-calculate-image-disk-size/138090
Diagnostics ID: 2E6BD753-3354-4BFC-B793-8F222F4BC38A/20231017165138 (uploaded)
I did disable the Resource Mode later on but the app keeps on crashing.
Hi @ansiwen,
Also still happening (freezing, not crashing) on my Intel Mac with Ventura 13.5.2 and 4.24.2: Diagnostics- ID: 95E98686-D4BB-4DCF-B0CA-0F6B96B45D98/20231013171419
Thank you very much for reporting; in this case we were able to find the bug and fix it (it was an internal deadlock in Docker Desktop), the fix will be present in the next DD release (DD 4.25).
As a temporary workaround, you can disable resource saver via the dashboard's settings->resources menu.
Please let us know if you hit any other problems.
Thanks again!
Hi @ctalledo,
I had this issue couple time and it happen again this afternoon so I decided to report it so you can take a look.
Docker desktop is leaving Resource Saver for any reasons. I ran some containers earlier today. I stopped them and after the 30sec delay, resource saver kick in...
Then I didn't restarted any containers and DD goes out of Resource saver and doesn't go back. I always notice it after relogging after beeing away of my laptop. My mac doesn't go into sleep since I disabled it on Power saving option.
I'm on DD 4.24.2 running on Mac Intel - Sonoma 14.0
Diagnostics ID: 7A64C259-EBAF-4648-98F1-BACB1721299C/20231017200847
Thanks
Having the same issues as @oli-laban, Docker was stuck in saver mode and after disabling it now crashes.
Docker Desktop 4.24.2 (124339) macOS Ventura 13.6, M2 Ultra
Diagnostics ID: A1C5F5AD-06B2-4FD5-AAA5-50D249A779A7/20231018072016
Hi @ansiwen,
Also still happening (freezing, not crashing) on my Intel Mac with Ventura 13.5.2 and 4.24.2: Diagnostics- ID: 95E98686-D4BB-4DCF-B0CA-0F6B96B45D98/20231013171419
Thank you very much for reporting; in this case we were able to find the bug and fix it (it was an internal deadlock in Docker Desktop), the fix will be present in the next DD release (DD 4.25).
As a temporary workaround, you can disable resource saver via the dashboard's settings->resources menu.
Please let us know if you hit any other problems.
Thanks again!
Hi @ctalledo Even though the Resource Saver is disabled; it still appeared enabled, and the app crashed soon after it was run.
On the other hand, once I deleted the pre-existing containers and reinstalled Docker, the Resource Saver turned up disabled and the app stopped crashing:
$ ls -alh ~/Library/Containers/com.docker.docker/Data/vms/0/data
total 85848416
drwxr-xr-x@ 3 user group 96B Oct 14 21:33 .
drwxr-xr-x 6 user group 192B Oct 14 21:33 ..
-rw-r--r-- 1 user group 60G Oct 14 21:19 Docker.raw
$ rm -fr ~/Library/Containers/com.docker.docker/
rm: ~/Library/Containers/com.docker.docker//.com.apple.containermanagerd.metadata.plist: Operation not permitted
rm: ~/Library/Containers/com.docker.docker/: Operation not permitted
On the other hand, once I deleted the pre-existing containers and reinstalled Docker, the Resource Saver turned up disabled and the app stopped crashing:
This seems to work for me too @umayrh :)
Hi @lgw4, @ramki88:
When you get a chance, could you please check if the problem reproduces with the following (unofficial) DD 4.24.3 build?
We've not been able to root-cause yet, but the builds above contain a change which may help.
Thanks!
Hi @ctalledo
Did install 4.24.3 (125250) till now unable to reproduce the issue. The last time it happened was when devcontainer was started from vscode while the docker was in resource saver mode. But today tried it multiple times and DD recovered as expected. Will update if the issue reoccurs.
Thanks!
@ctalledo Unfortunately, 4.24.3 still exhibits the failure to wake from Resource Savor mode for me.
Diagnostics ID: EF9239CE-8F64-4009-B01E-47A89F3F68A5/20231021001236 (uploaded)
Same issue, I had to disable resource saver
Same issue. I'll disable the resource saver... running on Windows 11 with Desktop 4.24.2 (124339)
Hi Chip (@lgw4), thanks again for trying with the custom build I provided and reporting the failure.
I am still trying to figure out why Docker Desktop sometimes fails to power up the Linux VM on your host, and to be honest I don't yet have a strong clue. On my Mac host (Mac Air with Apple M2 chip + Ventura), I can have DD enter/exit resource saver mode hundreds of times without problem. So it's something subtle.
One thing I noticed, is that in your host, Docker Desktop is filesharing the /Volumes
directory (see settings->resources->filesharing) and there happens to be a network mount underneath that directory.
I am wondering if disabling that makes a difference; if possible could you try removing the /Volumes
dir from the list of fileshared directories in Docker Desktop's settings, and see if you hit the resource saver problem again?
Thanks again for all the help, we appreciate it.
I also had to disable resource saver. Its pretty amazing that this functionality made it to a public release. It basically bricks my Docker desktop instance on MacOS (ARM)
I also had to disable resource saver.
Thanks @cosmotek for reporting; if possible please collect and upload a diagnostics bundle so we can take a look.
Its pretty amazing that this functionality made it to a public release. It basically bricks my Docker desktop instance on MacOS (ARM)
Quality is very important to us, and this feature was tested thoroughly before release. In fact, out of almost 1M users using the resource-saver feature, less than 1% are currently hitting errors.
Nonetheless we understand your frustration (as well as that of other users experiencing problems), and therefore continue to work diligently to debug all issues. But it's not easy, especially since we can't always reproduce the problem locally due to the large variety of machine setups across our users.
We appreciate you reporting the issue though, and ask for patience as we debug all the problems with the feature. Thanks!
Hi @umayrh,
I think I also face this problem
Thanks for reporting; in your case I see the following error when Docker Desktop exits resource saver mode (i.e., when it tries to power up the Linux VM to run the Docker Engine inside):
[2023-10-14T16:33:26.967882000Z][com.docker.virtualization][I] VM has stopped: Internal Virtualization error. The virtual machine stopped unexpectedly.
That's an error coming from the MacOS hypervisor; unfortunately there's no information on what caused that VM crash, but having low memory (RAM) on the host could be one reason. Not sure if that's the case in your scenario, because the diagnostics bundle was captured after the fact.
If you hit this again, please check the RAM utilization on your host at the time. Otherwise you may want to disable the resource saver feature to prevent this from re-ocurring.
Thanks!
Hi @frobinette,
Thanks for reporting.
Docker desktop is leaving Resource Saver for any reasons. I ran some containers earlier today. I stopped them and after the 30sec delay, resource saver kick in...
Then I didn't restarted any containers and DD goes out of Resource saver and doesn't go back.
Did you mean DD goes into resource saver mode and doesn't come out? Or that it never went into resource saver mode when it should have?
I looked at the diagnostics bundle you collected (thanks for that), but did not spot any issues with resource saver. I can see DD went into resource saver mode 23 times, and in all of them it came out fine.
Let me know if I am missing something; thanks!
Hi @frobinette,
Thanks for reporting.
Docker desktop is leaving Resource Saver for any reasons. I ran some containers earlier today. I stopped them and after the 30sec delay, resource saver kick in... Then I didn't restarted any containers and DD goes out of Resource saver and doesn't go back.
Did you mean DD goes into resource saver mode and doesn't come out? Or that it never went into resource saver mode when it should have?
I looked at the diagnostics bundle you collected (thanks for that), but did not spot any issues with resource saver. I can see DD went into resource saver mode 23 times, and in all of them it came out fine.
Let me know if I am missing something; thanks!
Hi @ctalledo! Thanks for looking at this.
What I mean is DD is going out of resource saver mode when I think it should not. I'm not starting any container nor running docker commands. And like you said it stays out of resource saver mode without re-entering it after 30 secs...
Is there any watchdog/checks that can cause DD to go out resource saver mode automatically and preventing it to go back in resource saver mode again after 30 secs?
Again, thanks for looking at this!
Hi @frobinette,
What I mean is DD is going out of resource saver mode when I think it should not. I'm not starting any container nor running docker commands. And like you said it stays out of resource saver mode without re-entering it after 30 secs...
Is there any watchdog/checks that can cause DD to go out resource saver mode automatically and preventing it to go back in resource saver mode again after 30 secs?
Oh got it, thanks for clarifying. Yes, there are some internal watchdog checks that can cause DD to go out of resource saver mode periodically, though it should always go back into resource saver mode. We are working to improve that, so that once it enters resource saver mode it stays there until user-driven activity requires DD to start the VM.
Try also with the latest DD (v4.25) because there have been improvements in this area.
Thanks again!
Hi @aevenstad,
Having the same issues as @oli-laban, Docker was stuck in saver mode and after disabling it now crashes.
Thanks for reporting and uploading the diagnostics; that helped us catch a bug during resource saver exit which we've just fixed. The bug you hit is different than most other users in these issue have hit. The fix will be present in v4.26.
Thanks again!
This is happening on my Mac Pro, here's the diagnostic id: D4A82B8F-DBC5-4E65-A8C9-7830F4AAF5D8/20231027141308
It's also happening on my Mac Studio Apple M2 Max 32 GB - Sonoma 14.0
Also still happening (freezing, not crashing) on my Intel Mac with Ventura 13.5.2 and 4.24.2: Diagnostics- ID: 95E98686-D4BB-4DCF-B0CA-0F6B96B45D98/20231013171419
Thank you very much for reporting; in this case we were able to find the bug and fix it (it was an internal deadlock in Docker Desktop), the fix will be present in the next DD release (DD 4.25).
Hi @ctalledo , I can confirm my issue is gone with version 4.25. It enters and leaves Resource Saver mode without problem.
Thanks!
@jwcunhagit can you please put your long output in <details>
tags, so it doesn't blow up the comments here?
<details>
<summary>Report (click here)</summary>
(your report output here)
...
...
</details>
will look like this:
@jwcunhagit can you please put your long output in
<details>
tags, so it doesn't blow up the comments here?<details> <summary>Report (click here)</summary> (your report output here) ... ... </details>
will look like this:
Report (click here)
@ansiwen done, thank you.
Updated to Docker Desktop 4.25.0 (126437) and it appears as the issue has been fixed. DD returns from resource saver to normal operation instead of freezing.
2023 MBP M2 Pro, Sonoma 14.1
Getting this error and couldnt fix it with the help of this thread. @ctalledo Diagnostics ID: 1CAE625A-49B1-47DB-8DBB-3D6D03A1B828/20231107143437
Hi @spointecker, thanks for filing the report.
In your case it seems the issue is not a resource-saver bug per-se, but rather the Docker Engine failing to boot (probably while exiting resource saver mode) due to lack of space inside the Docker Desktop VM:
[2023-11-07T14:16:13.091024006Z][dockerd][I] time="2023-11-07T14:16:13.090652057Z" level=error msg="failed to mount overlay: no space left on device" storage-driver=overlay2
...
[2023-11-07T14:16:13.093757792Z][dockerd][I] failed to start daemon
Please try increasing the VM's disk size in the Docker Desktop settings -> Resources window and restart Docker Desktop.
Additionally, once Docker Desktop is up and running, consider removing old/stale images that may be taking up too much space.
Nonetheless, I acknowledge we need to improve Docker Desktop's error handling in this case so as to provide a clear error when such problems occur. I'll take that as an action item.
Hi @spointecker, thanks for filing the report.
In your case it seems the issue is not a resource-saver bug per-se, but rather the Docker Engine failing to boot (probably while exiting resource saver mode) due to lack of space inside the Docker Desktop VM:
[2023-11-07T14:16:13.091024006Z][dockerd][I] time="2023-11-07T14:16:13.090652057Z" level=error msg="failed to mount overlay: no space left on device" storage-driver=overlay2 ... [2023-11-07T14:16:13.093757792Z][dockerd][I] failed to start daemon
Please try increasing the VM's disk size in the Docker Desktop settings -> Resources window and restart Docker Desktop.
Additionally, once Docker Desktop is up and running, consider removing old/stale images that may be taking up too much space.
Nonetheless, I acknowledge we need to improve Docker Desktop's error handling in this case so as to provide a clear error when such problems occur. I'll take that as an action item.
Hi @ctalledo, thank you for your reply! Unfortunately I am not able to start the docker desktop app anymore and therefore can not edit the settings. It hangs in this state:
EDIT: I tried the "Clean/Purge Data" from the troubleshooting section but did not try the "Reset to factory settings" as I wanted to prevent that in anyway.
Hi @spointecker,
Unfortunately I am not able to start the docker desktop app anymore and therefore can not edit the settings.
I see; please try updating the settings directly via the Docker Desktop settings file then:
Mac: ~/Library/Group\ Containers/group.com.docker/settings.json
Windows: C:\Users\<username>\AppData\Roaming\Docker\settings.json
The setting you are looking for is "diskSizeMiB"
; please set it to a larger value and restart DD.
Hope that helps!
Hi @spointecker,
Unfortunately I am not able to start the docker desktop app anymore and therefore can not edit the settings.
I see; please try updating the settings directly via the Docker Desktop settings file then:
Mac:
~/Library/Group\ Containers/group.com.docker/settings.json
Windows:C:\Users\<username>\AppData\Roaming\Docker\settings.json
The setting you are looking for is
"diskSizeMiB"
; please set it to a larger value and restart DD.Hope that helps!
Thank you very much, that worked like a charm!
Docker v4.25.2 still same issue in macos sonoma. Once resource saver is active both docker desktop and docker commands become unusable. Only way to quit is to use activity monitor and force quit all docker executables and then restart. (docker-compose up commands in cli just hang/wait without output. selecting quit, or restart in the docker desktop from menu or dashboard is not responding).
The docker desktop gui also does not allow disabling resource saver anymore.
@w-A-L-L-e can you please share a diagnostics id?
Fyi, you can disable Resource Saver temporarily via Settings > Features in development > Experimental features.
This setting is not available at the location specified in Docker Desktop v4.26.1. Followed Aug 7 advice from @ctalledo and the issue was resolved. Oddly the configuration file disappeared after restarting the docker daemon from the Whalemenu.
Hi @vhscom,
This setting is not available at the location specified in Docker Desktop v4.26.1
That's correct; the feature was moved from experimental -> official, and can now be configured from the Docker Desktop Settings -> Resources tab as described in the Docker Desktop docs here.
Thanks!
I am having the same issue since updating to v4.30.0
I am having the same issue since updating to v4.30.0
Hi @esetnik, sorry to hear that. If possible, please upload a Docker Desktop diagnostics bundle so we can take a close look.
Thanks!
I am having the same issue since updating to v4.30.0
Hi @esetnik, sorry to hear that. If possible, please upload a Docker Desktop diagnostics bundle so we can take a close look.
Thanks!
5A85B16E-FBD9-49B2-90A9-6273F287E37E/20240528142240
Hi @esetnik,
Thanks for uploading the diagnostics bundle.
In your particular case, I see that the Docker Desktop Virtual Machine (VM) crashed for some unknown reason:
[2024-05-20T13:54:22.412383000Z][com.docker.virtualization] VM has stopped: Internal Virtualization error. The virtual machine stopped unexpectedly.
...
[2024-05-21T16:44:20.353045000Z][com.docker.virtualization] VM has stopped: Internal Virtualization error. The virtual machine stopped unexpectedly.
...
[2024-05-21T19:41:23.851525000Z][com.docker.virtualization] VM has stopped: Internal Virtualization error. The virtual machine stopped unexpectedly.
That appears to be a bug in the Mac's virtualization software, but not sure what triggered.
Is there a specific workload you are running to trigger this?
Is the "Use Rosetta x86 Emulation on Apple Silicon" option set in the Docker Desktop settings?
Thanks again for reporting and uploading the diagnostics bundle, much appreciated.
Is there a specific workload you are running to trigger this?
Yes, it always happens after a long time with my proprietary company workload of a bunch of different containers. i'm not sure whether the mac has to sleep in order to reproduce it. I haven't found consistent steps to make it happen. I did notice that by disabling resource saver mode the issue goes away entirely.
Is the "Use Rosetta x86 Emulation on Apple Silicon" option set in the Docker Desktop settings?
Yes
Is there a specific workload you are running to trigger this?
Yes, it always happens after a long time with my proprietary company workload of a bunch of different containers. i'm not sure whether the mac has to sleep in order to reproduce it. I haven't found consistent steps to make it happen. I did notice that by disabling resource saver mode the issue goes away entirely.
Oh that's interesting. Resource saver mode works by automatically shutting off the VM when no containers are running for some time (e.g., 5 minutes). I wonder if the VM is failing to shutdown properly when resource saver kicks in, or the VM fails to start when resource saver detects a docker run
command.
FYI, we test resource saver quite a bit internally, so it's strange that it's failing this way in your case.
Is the "Use Rosetta x86 Emulation on Apple Silicon" option set in the Docker Desktop settings?
Yes
Thanks; if it's not an issue, try disabling it to see if that is somehow related to the problem.
Yeah. Me too. If it goes into Resource Saver mode, the only way to get docker working again is to force quit and reopen. I'm going to see if I can disable it. I love the idea though. @Docker gurus, let me know if I can do anything to help.
Docker Desktop 4.30.0 Mac OS: Sanoma 14.5
I've had no issues once I disabled resource saver mode. I cannot disable Use Rosetta x86 Emulation on Apple Silicon because I require the x86 emulation for my workload.
Yeah. Me too. If it goes into Resource Saver mode, the only way to get docker working again is to force quit and reopen. I'm going to see if I can disable it. I love the idea though. https://github.com/docker gurus, let me know if I can do anything to help.
Thanks @iamfrisbee; if possible, please upload a Docker Desktop diagnostics bundle so we can check what's going on.
Thanks again!
I've had no issues once I disabled resource saver mode. I cannot disable Use Rosetta x86 Emulation on Apple Silicon because I require the x86 emulation for my workload.
Thanks @esetnik, that's good feedback ... we are still investigating on our side.
Description
After the recent update, docker does not seem to recover from resource saver mode. In the dashboard CPU and Mem shows zero and all docker commands do not respond or wake the engine. Only after a restart of the engine it starts to respond
Reproduce
docker info
Expected behavior
"
docker info
should provide the info about dockerdocker version
docker info
Diagnostics ID
073CCF80-81D3-426A-A5F1-1B13A4481DE2/20230804091953
Additional Info
No response