search

CircleCI-Public / circleci-cli

Use CircleCI from the command line
https://circleci-public.github.io/circleci-cli/
MIT License
411 stars 232 forks source link

Cannot run local job on Mac OSX Intel due to Docker error #672

Open AlexisTonneau opened 2 years ago

AlexisTonneau commented 2 years ago

Meta:

CircleCI CLI Version: 0.1.16947+263902a (homebrew)

Operating System: OSX Monterrey 12.2.1, Intel chip

Docker Desktop Version: 4.5.0 (74594)

Current behavior:

I cannot run any job in local, while those jobs in Cloud pass. This is probably linked to local Docker, but I have an up-to-date Docker Desktop on my laptop.

I tried to run the build job from here (and plenty of others)

Output:

Docker image digest: sha256:ffe408d1918ee63d412ab5c13e2adbfdbf97cb6117fa5487eff574a00842a599
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/base:2022.03
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image cache not found on this host, downloading cimg/base:2022.03
2022.03: Pulling from cimg/base
08c01a0ec47e: Pull complete 
1d1c3c495166: Pull complete 
1cd1b31a0b06: Pull complete 
e99d0fae9c42: Pull complete 
4ebcb4a2f6ca: Pull complete 
58dd3b8faeae: Pull complete 
Digest: sha256:ef15b3cd73e92cc53407da65863d7b1502e998e554050f6baeebe82137d901ed
Status: Downloaded newer image for cimg/base:2022.03
cimg/base:2022.03:
  using image cimg/base@sha256:ef15b3cd73e92cc53407da65863d7b1502e998e554050f6baeebe82137d901ed
  pull stats: download 333.3MiB in 3m10.408s (1.751MiB/s), extract 333.3MiB in 14.911s (22.35MiB/s)
  time to create container: 803ms
Creating Docker containers in parallel 
Time to upload agent and config: 2.6759242s
Error: 
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/fc0799a00885a53e63b75d735b16d8ddfc197209f7277880e9ff076179a40594/75198cc6fcc89554b5831fdbfeb2f9a2fe33bb5df1260f03fb38dc419f34718c" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/fc0799a00885a53e63b75d735b16d8ddfc197209f7277880e9ff076179a40594/75198cc6fcc89554b5831fdbfeb2f9a2fe33bb5df1260f03fb38dc419f34718c" with domain controllers -- it is in an invalid state: unknown

When did this begin / Was this previously working?:

I just installed circleci-cli on my Macbook, and it never worked. It also happened in a relative's machine.

Additional Information:

I don't have any issue with Docker in other use cases, I can properly run containers.

davidmparrott commented 2 years ago

I believe I have encountered the same issue. 2019 MBP with Intel silicon running the most up to date version of Docker 4.4.2 (73305), macOS Monterey, and a fresh install of the CLI via homebrew. 

Docker image digest: sha256:14f53f2b64b8d90a10f0b3612a5af20e3eb5fb6b067b84118554ed4fdaea498e
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/go:1.17
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/go:1.17, but refreshing...
cimg/go:1.17:
  using image cimg/go@sha256:1a164c32a25fb88acfbbda2634f0dd96975ee471a23721eda0f9f2ca16662db1
  pull stats: Image was already available so the image was not pulled
  time to create container: 52ms
Creating Docker containers in parallel
1.17: Pulling from cimg/go
Digest: sha256:1a164c32a25fb88acfbbda2634f0dd96975ee471a23721eda0f9f2ca16662db1
Status: Image is up to date for cimg/go:1.17
Time to upload agent and config: 2.7446627s
Error:
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/0116ee46ccd9259126ba3ca404601f19a8edc82b31723ff193e0ca97b025264a/a8573e73e6036f2065b80686a823d1ba62a1a2c855f5786e576b2601c84d174c" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/0116ee46ccd9259126ba3ca404601f19a8edc82b31723ff193e0ca97b025264a/a8573e73e6036f2065b80686a823d1ba62a1a2c855f5786e576b2601c84d174c" with domain controllers -- it is in an invalid state: unknown
davidmparrott commented 2 years ago

Reading this issue to see what the status is but seems to be a known issue, I believe https://github.com/CircleCI-Public/circleci-cli/issues/589

AlexisTonneau commented 2 years ago

To help to investigate with this issue, it was working on a colleague’s MacBook last week, but isn’t working anymore from now. Probably related to either a Docker Desktop breaking change, either OSX breaking change.

kienstra commented 2 years ago

Thanks for opening this, a user just reported this same when using the CLI on CircleCI-Public/circleci-demo-go.

KingMob commented 2 years ago

I'm seeing the same on my 2019 MBP running macos 12.2.1, using circleci version "0.1.16947+263902a (homebrew)"

❯ circleci local execute --job build
Docker image digest: sha256:0af60dca5199a3e5c0c27a96873ac342d6fbf72ba12b1f5f190dc7b7dc18797c
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container alpine:3.7
alpine:3.7:
  using image alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10
  pull stats: Image was already available so the image was not pulled
  time to create container: 40ms
Creating Docker containers in parallel 
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
Time to upload agent and config: 1.857438223s
Error: 
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/debe6c79a4e5800818589a5f74151e55590570b330a7619cb1aeb4632615ff43/0d24854fdecd9cf51aa99647fccdaf24cf4a624f00ecd6e42571fdb88b21d60e" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/debe6c79a4e5800818589a5f74151e55590570b330a7619cb1aeb4632615ff43/0d24854fdecd9cf51aa99647fccdaf24cf4a624f00ecd6e42571fdb88b21d60e" with domain controllers -- it is in an invalid state: unknown
imjustabill commented 2 years ago

I see the same error on Intel MacOS 12.1 with both Docker 4.4.2 & 4.5, and MacOS 12.2.1 with Docker 4.5

q42jaap commented 2 years ago

I have this too. circleci: 0.1.16947+263902a (homebrew) MacOSX docker desktop: 4.3.2 (72729) MacBook Pro (16-inch, 2019): macOS Montery 12.2.1, Intel chip

keenfury commented 2 years ago

I'm getting the same error CircleCi: 0.1.16947+263902a (homebrew) Docker Desktop: 4.5.0 (74594) MacBook Pro: Monterey 12.2.1; M1 Pro

andrebr commented 2 years ago

Same error and versions as the above https://github.com/CircleCI-Public/circleci-cli/issues/672#issuecomment-1065208184, but with Intel chip.

shalom06 commented 2 years ago

Same issue MB pro 2019 i9 with Monterey

seankibler commented 2 years ago

FWIW I'm not experiencing this issue on a 2019 MBP running Docker Desktop 4.2.0 but others on my team on M1 running Docker Desktop 4.5.0 are. Given other reports here on Intel-based Mac running Docker Desktop above 4.2.0 this seems to be a compatibility issue between CircleCI command line tool and Docker Desktop above 4.2.0.

I'm not experiencing the error CircleCI: 0.1.16947+263902a Docker Desktop: 4.2.0 (70708) MacBook Pro: Monterey 12.2.1 2019 i7

jcampbellco-wastebits commented 2 years ago

Hitting this issue on a M1 Mac with Docker Desktop 4.5.0.

CircleCI: 0.1.16947+263902a (homebrew) Docker Version: 20.10.12 Docker Desktop: 4.5.0 (74594)

evangallup commented 2 years ago

Can confirm, downgrading Docker Desktop to 4.2.0 solves this issue for now.

josep11 commented 2 years ago

Same Error with

omarluq commented 2 years ago

Same error with 2020 Mac running big sur, intel chip, docker 4.5.0 update: downgraded to docker 4.2.0 and now it works

ghost commented 2 years ago

Same error with 2020 Mac running big sur, intel chip, docker 4.5.0 update: downgraded to docker 4.2.0 and now it works

I am here to say I agree with it. Under 4.2.0 it works.

KingMob commented 2 years ago

Upgraded to the latest Docker 4.7.1, it's still failing. Going to try downgrading.

jschuur commented 2 years ago

Likewise, I had the same issue, until I downgraded to Docker Desktop 4.2.0, then I was able to run a circleci local execute.

Version numbers for when it wouldn't run:

circlecli: 0.1.17183+341e493 (homebrew)
Docker Desktop 4.7.1 (77678)
macOS: 12.3.1 (M1 MBP)

Edit: When I downgraded the circleci CLI version and changed the SHA256 in build_agent_settings.json to sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014 as described here it worked for me under the current Docker Desktop too.

fabioaraujopt commented 2 years ago

I have the same problem with macOS monterey 12.1, Docker Desktop 4.6.0 (75818), circleci 0.1.17554+2b4b295 (homebrew)

kienstra commented 2 years ago

Yeah, also still seeing this on an Intel Mac with Monterey 12.3.1, Docker Desktop 4.8.1 (78998), and CircleCI v0.1.17554.

naviat commented 2 years ago

Likewise, I had the same issue, until I downgraded to Docker Desktop 4.2.0, then I was able to run a circleci local execute.

Version numbers for when it wouldn't run:

circlecli: 0.1.17183+341e493 (homebrew)
Docker Desktop 4.7.1 (77678)
macOS: 12.3.1 (M1 MBP)

Edit: When I downgraded the circleci CLI version and changed the SHA256 in build_agent_settings.json to sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014 as described here it worked for me under the current Docker Desktop too.

you saved my day.

qinlime commented 2 years ago

same issue on Mac m1 with Monterey 12.3.1, docker desktop 4.8.1, circleci 0.1.18305+60f56cc (homebrew)

piermarcob commented 2 years ago

I confirm the issue and the resolution after downgrading to Docker 4.2.

Docker 4.2 can be downloaded here for various platforms.

lucasmellos commented 2 years ago

Any updates on when it's going to be solved?

floodico commented 2 years ago

Any updates on when it's going to be solved?

nope, 420 is the answer;) (docker desktop version)

amaidah commented 2 years ago

confirming downgrade from 4.3 to 420 is the answer

Gidgidonihah commented 2 years ago

Downgrading Docker Desktop (currently from 4.10) to 4.2 is not a solution. It's a workaround.

A fix should be implemented so we don't have to stay on an outdated version. Unfortunately it sounds like that fix may have to be implemented in Docker, and not circleci.

WillAbides commented 2 years ago

There is another workaround in the related Docker Desktop issue that doesn't require using an old Docker Desktop.

Edit "$HOME/Library/Group Containers/group.com.docker/settings.json" and update deprecatedCgroupv1 to true.

This requires Docker Desktop 4.6.0 or newer.

crimson-knight commented 2 years ago

@WillAbides thank you, that was extremely helpful! Easier than downgrading.

geraldstanje commented 2 years ago

how you fix that? failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount72869062/Dockerfile: no such file or directory

iskotar commented 2 years ago

There is a rather similar problem.

Error: 
Unexpected environment preparation error: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: cannot enter cgroupv2 "/sys/fs/cgroup/docker/8181ef1a86be3ff2e0a79caa79fd04c74c7cea84f15a6d23829b19bf168cc692/e60222c65cc74406b845b6986147f7192d09ed002d5092bae0b354aed4193497" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed

I preferred not to downgrade version, so I tried set deprecatedCgroupv1 to true, but it not helps for me.

My env:

iskotar commented 2 years ago

Found solution that works for me: https://github.com/CircleCI-Public/circleci-cli/issues/676#issuecomment-1107030440

tnarik commented 2 years ago

this workaround -> https://github.com/CircleCI-Public/circleci-cli/issues/672#issuecomment-1235768553 seems to work with 4.4.2 at least as well.

jamierpond commented 2 years ago

Can confirm, downgrading Docker Desktop to 4.2.0 solves this issue for now.

Can confirm this worked for me on macOS Monterey M1 Pro

Eder87rh commented 1 year ago

There is another workaround in the related Docker Desktop issue that doesn't require using an old Docker Desktop.

Edit "$HOME/Library/Group Containers/group.com.docker/settings.json" and update deprecatedCgroupv1 to true.

This requires Docker Desktop 4.6.0 or newer.

This works, the only missing step is that you need to restart docker

aks commented 1 year ago

Same issue on Docker v4.15.0, on MacOS: 12.6, 2.4 GHz 8-Core Intel Core i9 Worked around with https://github.com/CircleCI-Public/circleci-cli/issues/672#issuecomment-1235768553, which is:

  1. edit ~/Library/Group Containers/group.com.docker/settings.json
  2. update deprecatedCgroupv1 to true
  3. Save changes
  4. Restart Docker
randy commented 1 year ago

Adding on Docker v4.16.2, MacOS 12.6.2. I used the same solution as @aks.

rrozanka commented 1 year ago

@aks solution worked for me on Apple Silicon M1, Docker 4.17.0, and MacOS 13.0

olurepo commented 1 year ago

Also @aks suggestion works with the latest Docker Desktop version 4.19.0 on Apple M1. So, there's really no need to downgrade to version 4.2.0

Just vi ~/Library/Group\ Containers/group.com.docker/settings.json change the deprecatedCgroupv1 to true and you should be fine.