Closed rfay closed 1 year ago
I have reported the same issue. It also happened in the previous build.
@cconnoruk if you could please link your report/issue that would help everybody, thanks.
I believe this is another example: 760B65E8-A765-4928-ACF9-E6186565F8C4/20210418223447
@djs55 PTAL. Thanks.
Thanks for the report and diagnostics. Looking through the logs it seems that the connection to Docker is working, docker info
, docker ps
etc seem to be ok. The VM has network access still (although there are definitely bugs in the current network stack that we're working on).
Is it an individual docker run
that's hanging? Could the container be stuck downloading something from the network? I'm wondering whether it could be a cascade failure from a network glitch.
thanks @djs55 - Some of the times I've seen this (like the OP) it was just a launch of ddev, running docker-compose, with no known network activity except talking to the docker server. None of the times appeared related to network activity. One of the unusual things about this is it doesn't seem to happen in any predictable place, but is fixed by docker restart.
In the latest report, https://github.com/docker/for-mac/issues/5590#issuecomment-822073606, tests hung for 2.5 hours before being killed. So it's not a matter of waiting, and normal network activity would have timed out another way. The other experiences were manual (and clearer what was happening).
Here's another one: 760B65E8-A765-4928-ACF9-E6186565F8C4/20210422122719
At this point a docker run -it busybox:latest sh
just does absolutely nothing, no output, nothing.
An interesting factoid is that before 3.3, this test setup used to restart docker to try to improve reliability:
# On macOS, restart docker to avoid bugs where containers can't be deleted
#if [ "${OSTYPE%%[0-9]*}" = "darwin" ]; then
# killall Docker || true
# nohup /Applications/Docker.app/Contents/MacOS/Docker --unattended &
# sleep 10
#fi
But unfortunately that stopped working in 3.3 and had to be removed.
But this hang behavior doesn't seem to show up on amd64...
Interesting behavior when this happens - If you just docker run -it --rm busybox sh
in any window, it hangs, but hangs hard. You can't hit return, no input buffering, nothing. If you run docker run --rm busybox ls
it hangs, but you can at least hit return and see input buffering.
I've done factory reset a few times. Guess I'll do it again now. Open to any suggestions of what might stop this behavior.
When this happens, a docker run -it --rm busybox ls
will just hang, so it's not just the one docker client thing that's having an episode, it's docker being completely broken.
Current report is from experimental build 3.4.0 (64153)
@djs55 this happens often enough that I can give you access to the test runner when this happens if you want. We just have to figure out a way to notify you and not take the test runner offline forever.
This continues to be a significant and repeating issue.
@rfay sorry for the delay getting back to you.
In the latest diagnostics I see some interesting things:
docker run
attempt to start
the container, but the start
request never succeeds.docker ps
seem to be workingThe last log I see from the start
involves setting up file sharing (even though there are no volumes on this container).
The file sharing is definitely misbehaving. I see this scary looking warning:
time="2021-04-22T06:26:05-06:00" level=warning msg="unable to inject 50 events for 13h20m30.005067584s"
and I can see a thread stuck injecting an event.
I suspect that something happened earlier which broke event injection. It's possible this had a delayed effect and is now blocking docker run
. I'm going to make a test build which tries to prevent the hang by dropping inotify events in this case. It would be interesting to see if that fixes part of the problem. Separately I'd like to investigate why the inotify injection is blocking.
Thanks for taking a look @djs55 - I see this a few times a week, and I test to see that it's happening with docker run --rm -it busybox ls
, which just hangs when the problem is in effect, and yes my go-to docker ps
is always working when this happens. Happy to try any test builds. The test runner is currently running your experimental 3.4.0 build 64153
@rfay I've got an experimental M1 build here: https://desktop-stage.docker.com/mac/stable/arm64/64656/Docker.dmg (also available for amd64
by changing the arch in the URL). It would be interesting to see if this fixes the deadlock you were seeing.
Thanks, I have it installed on the test runner; we'll see what happens.
I haven't had the hang over many, many test runs on two M1 test runners with the 64656 build from https://github.com/docker/for-mac/issues/5590#issuecomment-844432815 - will keep monitoring. I'll be gone for the next week, but wanted to let you know the good news. I also haven't seen any new problems with that build over 64153. Thanks for the effort on this!
I have had no more problems with hangs using your experimental 3.4.0 from https://github.com/docker/for-mac/issues/5590#issuecomment-844432815
Thanks so much.
@rfay are you able to advise whether you had experimental features enabled and if you had the Use new virtualisation framework
on to resolve your issue?
No, no experimental features, no use of new virtualization framework. And all fixed with the experimental build above in https://github.com/docker/for-mac/issues/5590#issuecomment-844432815
Shoot, I just had this again @djs55 - E0801D1C-C895-4C7E-A3A1-1B030AACACEA/20210604140118
A random note to check @djs55 - the 3.4.0 test build does not auto-start on restart/login, so you'll want to verify that it's doing starting on system restart before you release.
A random note to check @djs55 - the 3.4.0 test build does not auto-start on restart/login, so you'll want to verify that it's doing starting on system restart before you release.
Is this Windows or Mac? On Windows we had regressed this internally but hopefully have fixed it before shipping 3.4.0
@mat007 the not-autostarting-on-reboot was about macOS, I think it's working correctly in 3.4.0 on both macOS and Windows.
Hi @djs55 - Although 3.4.0 improved this behavior massively, it's not solved. Here's a new diagnostic: E0801D1C-C895-4C7E-A3A1-1B030AACACEA/20210615082704
Another one: E0801D1C-C895-4C7E-A3A1-1B030AACACEA/20210617195741
This still happens periodically on 3.5.1 @djs55 , providing another diagnostic since I know they expire in 30 days, E0801D1C-C895-4C7E-A3A1-1B030AACACEA/20210705045751
Keeping this alive, latest diagnostic is E0801D1C-C895-4C7E-A3A1-1B030AACACEA/20210722015739 @djs55 -
For a mention of why this is important: There's no longer a way to restart docker on mac from the command line. I only have one M1 mac test runner for ddev. So when this happens, I discover many hours later that no testing has happened and docker is hung. It's familiar, but I wish it wouldn't happen.
And it would be lovely to be able to restart docker from the command line as well. That worked in the past.
I'm not seeing this in 3.6.0, so something must be better. Closing this for now.
Good news, thanks for the update @rfay.
Still getting this issue in 3.6.0 hrm
Using Version 3.6.0 (3.6.0.5487) and still getting this issue.
I have the same issue, it has been around for the last few months. It has changed a bit of the last couple of updates with the frequency reducing, and now mostly crashes during an overnight hibernate. Normally after a hibernate I will startup and find that all the containers have stopped. Last night the containers seemed to be still running (according to Docker Desktop), but I could not connect to them, restarted Docker Desktop and we are back in business for now.
Running 4.1.1 69879 of Desktop and 20.01.8 of Engine.
I have the same issue too, process in docker will hang up after 2.5 hours in MacOS 12.0.1 intel chip, not in Windows Version.
Same happening for me. Running Docker Desktop 4.2.0 (70708) on the new MacBook M1 Pro MacOS Monterey 12.0.1.
docker compose up starts the containers and some of them hangs during init. It is random, sometimes I have to stop and start them multiple times to get them all working. Running the same docker compose project on Intel laptops works just fine.
Seeing this on Docker Desktop 4.3.2 (72729) on a MacBook Pro (Apple M1 Max) running MacOS Monterey 12.1.
I'm seeing this while executing "docker build --platform x86_64", where the container is based on Maven for Eclipse Temurin Alpine and is building a Maven Java project.
Random hangs or sometimes crashes. Retrying enough times allows it to finish.
I get this too with Docker Desktop 4.3.2 (72729) on a MacBook Pro (Apple M1 Max) running MacOS Monterey 12.1
100% reproducible building
https://github.com/apache/fineract
Whenever I try to build this I get a hang (interestingly at different points in the build process, anecdotally seems related to networking activity).
I've never managed to get the build to complete successfully
Identical source built with same Docker on MacBook Pro Intel works perfectly every time (takes about 385s on an i9 oldish MacBook Pro)
I have the same problem on Docker Desktop 4.6.0 (75818) on a MacBook Pro (M1 Pro) running MacOS Monterey 12.3
Happened on both of my M1 Mac's (Macbook Pro M1 Pro and Macbook Air M1), reverted to previous version and running fine.
Brent Crammond MBA, BSc(Comp. Sci.) email: @. @.> mobile: +64 21 228 5366
“Live as if you were to die tomorrow. Learn as if you were to live forever” - Gandhi
On Mon, Mar 21, 2022 at 11:29 PM Volkan Welp @.***> wrote:
I have the same problem on Docker Desktop 4.6.0 (75818) on a MacBook Pro (M1 Pro) running MacOS Monterey 12.3
— Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/5590#issuecomment-1073730811, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPYWIQPTUFMTOSL4WSHW73VBBFPRANCNFSM43CELAFA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you commented.Message ID: @.***>
I have the same problem on Docker Desktop 4.6.1 (76265) on a MacBook Pro 2021 (M1 Pro - 32GB) running MacOS Monterey 12.3
I have the problem mainly when I run from inside the container phpstan or composer. I've used the same codebase on my Macbook Pro 2020 (M1 - 16GB) for almost 1 year without these problems.
Any further update on this issue? Also has it ever reliably worked on M1 Macs, if so what was the docker version?
I have raised a support call with Diagnostic Data.
Brent Crammond MBA, BSc(Comp. Sci.) email: @. @.> mobile: +64 21 228 5366
“Live as if you were to die tomorrow. Learn as if you were to live forever” - Gandhi
On Thu, Apr 7, 2022 at 6:53 AM Manny Aslam @.***> wrote:
Any further update on this issue? Also has it ever reliably worked on M1 Macs, if so what was the docker version?
— Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/5590#issuecomment-1090625982, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPYWIX6CSGZR7BRGDACWXTVDXMQ3ANCNFSM43CELAFA . You are receiving this because you commented.Message ID: @.***>
I have same issue, happening frequently.
Mac OS Monterey Version 12.3.1 (21E258)
Mac Studio (2022) Apple M1 Max 32 GB RAM
Containers suddenly freeze, not accessible locally, docker commands are not working via terminal and docker desktop suddenly shows no containers at all. Some times my containers are shown active in desktop app, but with all other symptoms happening.
When it freezes and containers still shown in Desktop app, if I try to stop a container, it gives me this error:
Cannot stop Docker Compose application. Reason: Error invoking remote method 'compose-action': Error: spawn /bin/sh ENOENT
Not ideal but since I last posted we have an interim solution:
Whilst it isn't a great solution and very specific to our container, we no longer get random hangs and it actually completes the build, prior to this change we had never been able to get a successful build on M1 Mac.
M1 Max 12.3.1
My base image is amazoncorretto:11 (OpenJDK 11 based) @mannynable Reducing no of cpus is helpful to me.
same here:
Model Name: iMac
Model Identifier: iMac21,1
Chip: Apple M1
Total Number of Cores: 8 (4 performance and 4 efficiency)
Memory: 16 GB
System Firmware Version: 7459.101.2
OS Loader Version: 7459.101.2
docker version
Client: Docker Engine - Community
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d28e
Built: Thu Jul 29 13:55:47 2021
OS/Arch: darwin/arm64
Context: default
Experimental: true
Server: Docker Desktop 4.5.0 (74594)
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:07 2021
OS/Arch: linux/arm64
Experimental: true
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
only one docker command works after docker restart, next one is 100% hanging
Same issue happens at 4.7.1 as well.
Hardware:
Hardware Overview:
Model Name: MacBook Pro
Model Identifier: MacBookPro18,1
Chip: Apple M1 Pro
Total Number of Cores: 10 (8 performance and 2 efficiency)
Memory: 32 GB
System Firmware Version: 7429.61.2
OS Loader Version: 7429.61.2
Serial Number (system): J3W2T6M4JC
Hardware UUID: D19F37A3-A494-5926-99DA-BCFFA138FE0E
Provisioning UDID: 00006000-001218D60A38801E
Activation Lock Status: Disabled
Client:
Cloud integration: v1.0.23
Version: 20.10.14
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 24 01:49:20 2022
OS/Arch: darwin/arm64
Context: default
Experimental: true
Server: Docker Desktop 4.7.1 (77678)
Engine:
Version: 20.10.14
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 87a90dc
Built: Thu Mar 24 01:45:44 2022
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Same exact issue - latest Docker for M1 version as well. I'm experiencing intermittent full container freezes that can only be fixed by a full reboot of Docker. Compose-down doesn't work, killing containers via CLI doesn't work, and killing containers in the Dashboard GUI doesn't work either. The GUI shows an error toast that says "tried to kill container, but did not receive an exit event" for every container deletion attempt until a reboot.
Working with Docker on an M1 Mac is extremely frustrating. It seems anecdotally like I get this error after one or two uses of docker exec
against a running container, but I haven't been able to create a consistent repro.
system_profiler SPSoftwareDataType SPHardwareDataType
output:
Software:
System Software Overview:
System Version: macOS 12.4 (21F79)
Kernel Version: Darwin 21.5.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled
Time since boot: 6 days 11:54
Hardware:
Hardware Overview:
Model Name: MacBook Pro
Model Identifier: MacBookPro18,4
Chip: Apple M1 Max
Total Number of Cores: 10 (8 performance and 2 efficiency)
Memory: 64 GB
System Firmware Version: 7459.121.3
OS Loader Version: 7459.121.3
docker version
output:
Client:
Cloud integration: v1.0.24
Version: 20.10.14
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 24 01:49:20 2022
OS/Arch: darwin/arm64
Context: default
Experimental: true
Server: Docker Desktop 4.8.2 (79419)
Engine:
Version: 20.10.14
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 87a90dc
Built: Thu Mar 24 01:45:44 2022
OS/Arch: linux/arm64
Experimental: true
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
I'm seeing this regularly with 4.8.1 on M1 mac. Randomly hangs during tests during builds which launch a handful of docker containers. Rarely does it happen in the same place. A docker container is outputting logs and showing startup messages, then it just stops and nothing else ever is output. Killing the containers and running my build again might allow the containers to all startup, or it will fail in another location. It fails perhaps 30% of the time I estimate.
Note this same build succeeds on Linux 100% of the time. I have been using Linux for years up to this point and recently attempted to switch to an M1 mac when this problem started. Switching back to my Linux machine for past week or two and I've never seen this on Linux (note no docker desktop on linux - just docker, CE?).
╰─ system_profiler SPSoftwareDataType SPHardwareDataType
Software:
System Software Overview:
System Version: macOS 12.3.1 (21E258)
Kernel Version: Darwin 21.4.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Computer Name: MacBook Pro 2022
User Name: My Nam (myusername)
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled
Time since boot: 7 days 1 minute
Hardware:
Hardware Overview:
Model Name: MacBook Pro
Model Identifier: MacBookPro18,1
Chip: Apple M1 Pro
Total Number of Cores: 10 (8 performance and 2 efficiency)
Memory: 16 GB
System Firmware Version: 7459.101.3
OS Loader Version: 7459.101.3
Serial Number (system): M2JH4WF0H0
...
Activation Lock Status: Disabled`
╰─ docker version
Client:
Cloud integration: v1.0.24
Version: 20.10.14
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 24 01:49:20 2022
OS/Arch: darwin/arm64
Context: default
Experimental: true
Server: Docker Desktop 4.8.1 (78998)
Engine:
Version: 20.10.14
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 87a90dc
Built: Thu Mar 24 01:45:44 2022
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Same as proth above:
system_profiler SPSoftwareDataType SPHardwareDataType Software:
System Software Overview:
System Version: macOS 12.4 (21F79)
Kernel Version: Darwin 21.5.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Computer Name: Steve Knight MacBook Pro 2022
User Name: Steve Knight (steveknight)
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled
Time since boot: 4:34
Hardware:
Hardware Overview:
Model Name: MacBook Pro
Model Identifier: MacBookPro18,1
Chip: Apple M1 Pro
Total Number of Cores: 10 (8 performance and 2 efficiency)
Memory: 16 GB
System Firmware Version: 7459.121.3
OS Loader Version: 7459.121.3
Serial Number (system): D77JXMJLJ6
Hardware UUID: 620C2CF2-9EF7-572E-8291-6FE4186A5C41
Provisioning UDID: 00006000-001459E00E62801E
Activation Lock Status: Disabled
docker version
Client: Cloud integration: v1.0.25 Version: 20.10.16 API version: 1.41 Go version: go1.17.10 Git commit: aa7e414 Built: Thu May 12 09:20:34 2022 OS/Arch: darwin/arm64 Context: default Experimental: true
Server: Docker Desktop 4.9.0 (80466) Engine: Version: 20.10.16 API version: 1.41 (minimum version 1.12) Go version: go1.17.10 Git commit: f756502 Built: Thu May 12 09:14:19 2022 OS/Arch: linux/arm64 Experimental: false containerd: Version: 1.6.4 GitCommit: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16 runc: Version: 1.1.1 GitCommit: v1.1.1-0-g52de29d docker-init: Version: 0.19.0 GitCommit: de40ad0
I'm having the same issue. Test container launches, all associated containers launch and run, then nothing ever happens in the test container. I can generally kill it, but full restarts of Docker desktop have been required.
system_profiler SPSoftwareDataType SPHardwareDataType
Software:
System Software Overview:
System Version: macOS 12.4 (21F79)
Kernel Version: Darwin 21.5.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Computer Name: <redacted>
User Name: <redacted> <redacted>
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled
Time since boot: 18 minutes
Hardware:
Hardware Overview:
Model Name: MacBook Pro
Model Identifier: MacBookPro18,2
Chip: Apple M1 Max
Total Number of Cores: 10 (8 performance and 2 efficiency)
Memory: 32 GB
System Firmware Version: 7459.121.3
OS Loader Version: 7459.121.3
Serial Number (system): <redacted>
Hardware UUID: <redacted>
Provisioning UDID: <redacted>
Activation Lock Status: Enabled
Client:
Cloud integration: v1.0.24
Version: 20.10.17
API version: 1.41
Go version: go1.17.11
Git commit: 100c701
Built: Mon Jun 6 23:04:45 2022
OS/Arch: darwin/arm64
Context: default
Experimental: true
Server: Docker Desktop 4.10.1 (82475)
Engine:
Version: 20.10.17
API version: 1.41 (minimum version 1.12)
Go version: go1.17.11
Git commit: a89b842
Built: Mon Jun 6 23:01:01 2022
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.6
GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc:
Version: 1.1.2
GitCommit: v1.1.2-0-ga916309
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Expected behavior
No hanging
Actual behavior
Hanging at unknown places. Some tests just hang in the middle and stop forever. I've also seen a manual hang.
Restarting Docker Desktop solves the problem temporarily.
The provided diagnostic is the first time I stopped to capture this, but it has happened to me at least 3 times since 3.3.1 came out
Information
Steps to reproduce the behavior