Docker does not recover from resource saver mode

[ramki88]

Description

After the recent update, docker does not seem to recover from resource saver mode. In the dashboard CPU and Mem shows zero and all docker commands do not respond or wake the engine. Only after a restart of the engine it starts to respond

Reproduce

docker info

Expected behavior

"docker info should provide the info about docker

docker version

Client:
 Cloud integration: v1.0.35-desktop+001
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:32:30 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.22.0 (117440)
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:38 2023
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.5
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.1
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2-desktop.1
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.20.0
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 7.765GiB
 Name: docker-desktop
 ID: efcb88bd-4e43-4b04-be3f-89edcd9a82d7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: true

Diagnostics ID

073CCF80-81D3-426A-A5F1-1B13A4481DE2/20230804091953

Additional Info

No response

[christophermclellan]

Thanks @ramki88 we'll look in to this asap. Fyi, you can disable Resource Saver temporarily via Settings > Features in development > Experimental features.

[ctalledo]

Hi @ramki88, sorry you hit a problem and thanks for filing the issue.

Looking at the diagnostics bundle, I see Docker Desktop hit an internal error while entering resource saver mode:

[2023-08-03T17:41:10.371280000Z][com.docker.backend.idle][I] busy -> idle
[2023-08-03T17:41:10.372035000Z][com.docker.backend.idle][I] starting timer (30s)
[2023-08-03T17:41:40.373905000Z][com.docker.backend.idle][I] timer expired (30s)
[2023-08-03T17:41:40.375186000Z][com.docker.backend.idle][I] system idle for 30.002479833s -> reducing
[2023-08-03T17:41:40.380947000Z][com.docker.backend.idle][I] populating cache
ESC[91m[2023-08-03T17:42:40.381430000Z][com.docker.backend.idle][E] failed to populate cache: calling HEAD http://localhost/_ping: calling http://localhost/_ping 500ESC[39m
[2023-08-03T17:55:40.426983000Z][com.docker.backend.idle][I] idle -> busy (was idle for 14m29.996941291s; reason: [in-flight APIs: map[/_ping:1]])

We will work internally to fix it.

Out of curiosity, how often does this reproduce for you?

Thanks!

[ramki88]

Hi @ctalledo, Thanks for you response. I have noticed these behaviour about 4 times today. Till now I have not observed any pattern. Will observe and report if I find anything.

[kangaechu]

I have same problem. I disabled "Access experimental features", but resource saver mode is still on.

I tried Factory Reset, but it doesn't changed.

[mikelax]

I am having similar issues with the 4.22.0 version that I just upgraded to this morning. Once I noticed that the app went into Resource Saver mode, I am now unable to do anything within the app. I can't exit or force quit using the Mac menu.

I have restarted my laptop, once I then start Docker Desktop it starts back in Saver mode but I am unable to do anything. I can't run docker build, restarting and clean/purge data don't respond with anything and Quit Docker Desktop from the system menu also does nothing.

[hrabe]

Have the same issue with version 4.22.0 (117440) After reaching the resource saver state:

• cli won't work
• restart docker won't work
• force quit the Docker Desktop application seems to close UI and removes icon but can't be started afterwards
• restart of Mac will make it work for some time (but starts in resource save mode)

Please provide a work around, a config option to deactivate this or make sure, it wakes up correctly when used.

[bconfrancisco]

Hi - I installed 4.22.0 last week and it was working on Friday 8/4. Today i have the same issues as the others. Thank you in advance for the fix.

[ctalledo]

Hi folks, Cesar Talledo from Docker here.

Sorry to hear you are having trouble with the resource-saver feature, and thank you very much for reporting the problem.

First, quick context:

The resource saver feature is meant to significantly reduce Docker Desktop's memory and CPU usage on the host when no containers are running, to improve overall user experience. It will reduce mem utilization by ~2GBs (or more) when DD goes idle. In DD v4.21, the feature would simply pause processes in the DD Linux VM, but in v4.22 we improved it to shutdown the DD Linux VM. We did a lot of testing prior to releasing the feature and for most users it's working well, but a few are reporting problems. Our apologies, we will try to fix them ASAP.

For those of you experiencing problems:

• If possible, please upload a diagnostics bundle so we can debug it.

• To disable the feature, edit the features-overrides.json file as follows:

{
  "IdleShutdown": {
    "name": "",
    "label": "Shutdown VM when idle",
    "description": "When this is set, the system will shutdown the VM when idle",
    "enabled": false,
    "type": 2
  }
}

The file can be found here:

Mac: ~/Library/Group Containers/group.com.docker/features-overrides.json Windows: %APPDATA%\Roaming\Docker\features-overrides.json Linux: $HOME/.docker/desktop/settings.json

After you disable the feature, make sure to quit Docker Desktop and start it again via the Whalemenu on the taskbar (don't just simply click "restart" on the Whalemenu).

If you have any issues with this, please let us know.

Thanks again and we hope to fix all issues very soon.

[bsousaa]

We are investigating the issue internally.

To help us reproducing the issue could you upload a diagnostics bundle and share the id? @bconfrancisco @hrabe @mikelax @kangaechu . Thanks in advance!

[safayildirim]

We are investigating the issue internally.

To help us reproducing the issue could you upload a diagnostics bundle and share the id? @bconfrancisco @hrabe @mikelax @kangaechu . Thanks in advance!

I have the same problem. My diagnastic id is: 17381DA9-8BEE-4201-8FF2-0FB58278F01C/20230808113242

[mikelax]

We are investigating the issue internally.

To help us reproducing the issue could you upload a diagnostics bundle and share the id? @bconfrancisco @hrabe @mikelax @kangaechu . Thanks in advance!

@bsousaa Here is an ID: D037C0D1-560C-409B-A11C-0629819A44CE/20230808114543

One item of note, I first started Docker this morning and immediately ran the Diagnostics and uploaded. During this time Docker appeared to be in regular mode. As soon as the Diagnostics ID uploaded completed then Docker switched into "Resource saver mode".

[benjaminreji]

Diagnostic ID: 8C632EC5-F963-42AB-B88E-4342E129FF2F/20230808115322

I keep getting this error no such service: build

Docker is in resource saver mode

[bconfrancisco]

Diagnostic ID: 8E671B87-5DFC-45A8-8460-34DBFE462181/20230808152154

MacOS Monterey: Version 12.5.1

[saillingaway]

I've also been having this issue with version 4.22.0 for the last 2 days.

I can't quit/start/restart docker via the Whalemenu on the taskbar. Quitting Docker Desktop via the app makes the app/UI disappear but the taskbar Whalemenu doesn't change and shows it's still running. I'm new to Docker though so this could be the expected behavior.

Diagnostic ID: 480CE2DB-48A1-4BB8-8B7B-78C895F45DAD/20230808163825 MacOS Ventura 13.4

@ctalledo I don't seem to have ~/Library/Group Containers/group.com.docker/features-overrides.json, but I do have a settings.json. Not sure if I should create a features-overrides.json file there and add the IdleShutdown snippet, or if I should add it to settings.json.

[ctalledo]

Hi @safayildirim, apologies you hit this issue (we are on it) and thanks for uploading the diagnostics bundle, very helpful.

Not sure if I should create a features-overrides.json file there and add the IdleShutdown snippet

Yes, that's what you should do: create a new features-overrides.json file in that same directory and then quit & start Docker Desktop.

[ctalledo]

Thank you @bconfrancisco @benjaminreji @mikelax @safayildirim @safayildirim for uploading the diagnostics bundle, this helps us root cause the problem.

[kangaechu]

Diagnostic ID: 60C25128-558D-4678-9961-8E6CA9B9DE27/20230809012117

[ctalledo]

Diagnostic ID: 60C25128-558D-4678-9961-8E6CA9B9DE27/20230809012117

Thanks @kangaechu; I looked at the diagnostic bundle but did not see any failure with the resource saver feature.

Did it fail for you somehow? Or are you just looking to disable it?

Thanks.

[kangaechu]

@ctalledo I downgraded Docker Desktop for Mac to 4.21.1 two days ago. Today I updated it to 4.22.0 in order to send a diagnostic bundle. As a result, the failure log for the Resource Saver feature may not be included.

[niktolis]

Same issue for me in 4.22 on macOS Ventura 13.4.1. Containers were stopped the Mac was put on sleep. When waking up the docker is in Resource Saver mode and unresponsive. Diagnostic ID: 0408424A-469D-48B6-8B3F-B92D0868B079/20230810045901

[chucheng92]

+1 same issue for me in v4.22.0 on macOS Ventura 13.4.1. when i disable experimental feature, it can't restore.

[doringeman]

Hi @chucheng92, You can leave Experimental Features on and disable this feature only by following the instructions that Cesar provided in https://github.com/docker/for-mac/issues/6933#issuecomment-1668306581.

[johnthagen]

As a side note, I don't recall ever opting in to experimental features. Was I automatically opted in? If so, I don't think that should be forced on users.

[3UR]

Issue still happening for me it's very annoying

[ctalledo]

Hi @3UR,

Issue still happening for me it's very annoying

Sorry to hear; did you try disabling resource saver as described above?

Thanks!

[djs55]

Hi,

After reading some of the diagnostics I think there is an intermittent problem connecting the VM to the host. I've fixed a couple of bugs in this area and added more diagnostics. If you'd like to try a new build:

• Apple Silicon
• Intel

If it still hangs, could you run this diagnostics command while in the broken state:

/Applications/Docker.app/Contents/MacOS/com.docker.diagnose gather -upload 

and quote the ID? It's safe to downgrade to a previous version after trying this or to stick on it if it works. Perhaps rename the previous version in /Applications to /Applications/Docker-last.app so you can switch back easily.

Thanks!

[bconfrancisco]

@djs55 - I downloaded and installed the new build (intel) and enabled the resource saver feature in the overrides file. I waited until Docker went into Resource Saver mode and then try to do a build with docker. Docker exited Resource Saver mode, built the docker image and went back into Resource Saver mode. Thank you for your help.

[jefejohnson]

I ran into this issue today with v4.22. I can confirm that v4.23 of Docker Desktop for macOS on an Intel machine has resolved my issue. I waited for docker to go into Resource Saver mode then started up several containers via docker compose. It took a wee bit longer to start up, but all my containers started up as expected.

Thanks for your help, everyone!

[ctalledo]

I ran into this issue today with v4.22. I can confirm that v4.23 of Docker Desktop for macOS on an Intel machine has resolved my issue.

Thanks @jefejohnson for confirming the fix works, we should have a new release with it very soon.

I waited for docker to go into Resource Saver mode then started up several containers via docker compose. It took a wee bit longer to start up, but all my containers started up as expected.

Yes it takes a few seconds for DD to exit resource saver mode as it's starting the Linux VM in the background. Once the VM is started, then all subsequent container runs should be almost instantaneous.

If you have any other feedback on how we can improve the resource saver feature, we are all ears. Thanks!

[niekbeckers]

Hi! Installed v4.23 on Apple silicon (M1), and unfortunately the fix does not work. Id: 3A32A7D6-AC16-4D13-8E06-099FDD84FC33/20230817061653

I'm now back on v4.21.1, works like a charm.

Thanks for looking into it!

[purwandi]

Confimed, rollback to v4.21.1 to

[fourn]

Hi! Installed v4.23 on Apple silicon (M2), and unfortunately the fix does not work.

[frobinette]

Hi!

I installed 4.23 (in comments above) and I didn't get the issue for 2 days.

But this morning, my containers didn't start.

I don't know if it can help in the investigation, but the issue is happening after I put my computer to sleep for the night.

I rollbacked to 4.21.1 until the next release.

On MacOS Ventura 13.5 Intel CPU.

[fourn]

这是来自QQ邮箱的自动回复邮件。 您好，收到此邮件代表我已经收到您的邮件，稍后会给您回复。

[trcoelho]

Also had do downgrade to v4.21.1 - this resource saver mode is so unstable, every time that is on resource saver, it failed to build images, check containers (docker ps), stats (docker stats) and it does not disable on "Settings" as said above.

Please, keep this "resource saver" optional and set as disabled by default.

This feature is so annoying :)

[fsantibanezleal]

Same issue here for 4.22 and Intel build for 4.23

Disabling experimental features is not stable at all.

Downgrading

[djs55]

@niekbeckers thanks very much for the diagnostics! I've identified two bugs: in the previous build I added a diagnostics service whose Unix socket path is slightly longer than the previous maximum. This triggered one bug and then unfortunately the error path had a second bug which caused a deadlock before the first bug could be logged properly (although there was still enough of a hint for me to spot the problem thankfully)

I've got a new experimental build with both bugs hopefully fixed. If you have time to try it, let me know what happens (and if it does break again -- hopefully not -- please upload a fresh diagnostic and I'll have another look):

• Intel Mac
• Apple Silicon Mac

Thanks for all your help so far!

[turbografx32]

I can note when docker turn to saver mode, testcontainers cant start and it cant find active docker instance.

[djs55]

@turbografx32 could you provide some repro steps so we can investigate? Thanks in advance!

[patkaehuaea]

@djs55 Your build above worked for me. I'm on an m2 and had been using v4.22.0 (version available via Homebrew as of this post). Previous version would go into resource saver mode and never exit. The new build enters, then exits resource saver mode as expected.

Thank you!

[kvasylchenko]

@djs55 Apple Silicon Mac also works fine for me. Finally! Thanks god :D Thank you very much for the build!

[robert914]

Can we just have an option in the Settings to just turn this off in the GUI? Why is this mode being thrust upon us with no clear way to disable it short of some custom config file modification? I have use cases where I don't want to even use this feature at all and would prefer to just have it turned off. Thanks!

I will also add, the custom config modification does disable it, but the menu icon still indicates it's enabled when it really is not.

[djs55]

@robert914: are you mainly concerned about bugs (e.g. failures to recover from resource saver mode) or would you rather run the VM continuously even when there are no containers running?

[robert914]

@robert914: are you mainly concerned about bugs (e.g. failures to recover from resource saver mode) or would you rather run the VM continuously even when there are no containers running?

The latter, I would rather have the option to run the VM continuously even when no containers are running.

[djs55]

@robert914 thanks, could you explain why? (e.g. is the restart time too long or something else?)

[robert914]

@robert914 thanks, could you explain why? (e.g. is the restart time too long or something else?)

The restart time is too long when I use batching to bring up and down multiple containers in batch jobs where there may be gaps in time between each container running so it goes into "resource saver" mode since there may be no containers running. This then inserts additional delay before the start of successive container starts and makes the batch run longer than necessary.

If we had a way to control the "resource saver" timeout (i.e. like a screen saver has), that would likely be more useful though.

[djs55]

@robert914 thanks for the explanation! If you'd like to experiment with timeouts, there is one in ~/Library/Group\ Containers/group.com.docker/settings.json:

  "autoPauseTimeoutSeconds": 30,

This can be bumped to any value you like.

[robert914]

@djs55 Thanks for the tip!

[robert914]

@djs55 It would be awesome if this existed in the actual GUI settings panel though at some point.

[ctalledo]

Hi folks, we just released a new Docker Desktop patch release (v4.22.1) that should fix several of the problems reported here (particularly for folks having trouble with Docker Desktop initial startup or resuming from resource-saver mode).

Please let us know if you continue to see problems, we apologize if that's the case and appreciate your feedback. It's a difficult feature to implement, and though we test it internally, nothing beats having thousands of users using it and reporting issues.

If you are still having problems and want to disable the feature, the DD v4.22.1 patch release also ensures that you can disable the feature by clearing the "Access Experimental Features" toggle in the Docker Desktop settings -> Features In Development -> Experimental Features tab. But note that this will disable all experimental features.

If you just want to disable the resource saver feature, then use the features-override.json file as described in a prior comment.

Finally, if you want to modify the resource saver timeout (i.e., the timeout after which if Docker Desktop is idle it enters resource saver mode; defaults to 30 seconds), do so as described above.

As the feature matures we will move it from experimental to a regular feature, at which point you'll be able to fully configure it via the Docker Desktop settings menu.