search

CircleCI-Public / circleci-cli

Use CircleCI from the command line
https://circleci-public.github.io/circleci-cli/
MIT License
411 stars 232 forks source link

[circleci local execute] error looking up cgroup: not implemented for cgroup v2 unified hierarchy #589

Closed ldorau closed 2 years ago

ldorau commented 3 years ago

Meta:

CircleCI CLI Version: 0.1.15195+595cc9e (release)

Operating System: Arch Linux (latest)

CircleCI CLI Diagnostic:

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /home/ldorau/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API... 
Ok.
Hello, Lukasz Dorau.

Current behavior:

$ circleci local execute
Docker image digest: sha256:fc6b14595a231dd8417c97fad26f5f5f294f64b6f58443d93dfba17186f9f738
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.9.14-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Expected behavior:

"circleci local execute" works

When did this begin?

Now

Was this previously working?:

No, never.

ucals commented 2 years ago

Exactly same error here. Running macOS Monterrey 12.2.1, installed CLI manually (v0.1.16947).

rm ~/.circleci/build_agent_settings.json
circleci local execute --job "node/test"

Trace:

Downloading latest CircleCI build agent...
Docker image digest: sha256:ea206e01751a9d4acc6dc21fb8184bdf67b0c2a524c156e526bdb991b1bd2e23
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/node:16.13.2
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image cache not found on this host, downloading cimg/node:16.13.2
16.13.2: Pulling from cimg/node
f3ef4ff62e0d: Pull complete
2c322a662952: Pull complete
fb8baef6d8e7: Pull complete
805b7ba0a256: Pull complete
63833a96b952: Pull complete
5d1ffe5f9ba5: Pull complete
6dbfea31d6de: Pull complete
44f09c08a855: Pull complete
Digest: sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
Status: Downloaded newer image for cimg/node:16.13.2
cimg/node:16.13.2:
  using image cimg/node@sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
  pull stats: download 358.9MiB in 44.413s (8.08MiB/s), extract 358.7MiB in 44.774s (8.012MiB/s)
  time to create container: 275ms
Creating Docker containers in parallel
Time to upload agent and config: 7.0508151s
Error:
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/f4e94be5ff47d02ebd9b580db37a9af1b9f222c692da90e0c266d6c03bdc3f6e/80d3b87d51ab7db04d27d38021b19568f05769e045c23541dc2fea62fa92838c" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/f4e94be5ff47d02ebd9b580db37a9af1b9f222c692da90e0c266d6c03bdc3f6e/80d3b87d51ab7db04d27d38021b19568f05769e045c23541dc2fea62fa92838c" with domain controllers -- it is in an invalid state: unknown
matschaffer commented 2 years ago

The older circleci/picard image seems to work, but the latest fails.

I opened https://github.com/CircleCI-Public/circleci-cli/issues/676 since it seems to be a recurrence of a similar issue on a new image.

deokseong-kim-toss commented 2 years ago

How to change circleci/picard to previous one? I'm facing this issue too.

====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/node:16.10
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/node:16.10, but refreshing...
cimg/node:16.10:
  using image cimg/node@sha256:4647574f341189fdb7d85cfb10ac70d464f6ccb5c7c38731c8e3e28c2bd80941
  pull stats: Image was already available so the image was not pulled
  time to create container: 34ms
Creating Docker containers in parallel 
16.10: Pulling from cimg/node
Digest: sha256:4647574f341189fdb7d85cfb10ac70d464f6ccb5c7c38731c8e3e28c2bd80941
Status: Image is up to date for cimg/node:16.10
Time to upload agent and config: 1.596606679s
Error: 
Unexpected environment preparation error: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/2aada119a145313e1719519731b3844c3efe245d94620972f3bce7857c69a90c/3e89c8ff0b0861b6285f5cd6d5ab5fa88c03d3352cd686dce8650d69227ea82e" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
deokseong-kim-toss commented 2 years ago

Okay. In my case, I changed LastestSha256 to sha256:6c640d1c0aed49afd1c6a426bfb8f5f40fbb7de439a154b73d53184977b9c1f6 and It works. there seems to be a problem with the latest circleci/picardimage

matschaffer commented 2 years ago

@deokseong-kim-toss nice :) Yeah I put my steps on https://github.com/CircleCI-Public/circleci-cli/issues/676 - maybe it'd be good for you to +1 that issue.

sebastian-lerner commented 2 years ago

Hi folks, thank you for the detailed reports of what you are seeing. My team is actively investigating and we hope to have a resolution as soon as possible. I'll update this issue when I have more information.

Kayden-lolasery commented 2 years ago

I have a very similar error:

Steps to replicate:

Starting container cimg/python:3.8 Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub. image is cached as cimg/python:3.8, but refreshing... cimg/python:3.8: using image cimg/python@sha256:db17fcbb1469f1f70bbbd76867c6d371d4bf9596cae84ece3b62de8fe9c20fb2 pull stats: Image was already available so the image was not pulled time to create container: 52ms Starting container cimg/postgres:12.7 3.8: Pulling from cimg/python Digest: sha256:db17fcbb1469f1f70bbbd76867c6d371d4bf9596cae84ece3b62de8fe9c20fb2 Status: Image is up to date for cimg/python:3.8 Creating Docker containers in parallel .... cimg/postgres:12.7: using image cimg/postgres@sha256:a488ce128a9e16e98f21a47af8981b0ea797b59cccee907d034d855e63b1a164 pull stats: Image was already available so the image was not pulled time to create container: 48ms Time to upload agent and config: 1.708347017s Error: Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/ad8c463b3491878d8cf8a44061b4f0ff3bf147998c4e2a980d30b89e2f57ad8b/dda3240f3a162cc38220cdfb64a84ce60f57a7478277cecdc500d7f5b981d314" with domain controllers -- it is in an invalid state: unknown

Step failed Task failed

matschaffer commented 2 years ago

@Kayden-lolasery see if the workaround in https://github.com/CircleCI-Public/circleci-cli/issues/676 works for you

rohan-kulkarni-phx commented 2 years ago

Hi, getting the following on Monterey 12.2, MacBook Pro 16-inch 2019 (intel)

$ uname -a
21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64 x86_64
$ rm ~/.circleci/build_agent_settings.json
$ circleci config process .circleci/config.yml > process.yml
$ circleci local execute -c process.yml --job lint-and-test

Output:

Docker image digest: sha256:dad8f737004adbb7722e416d99b53433f0c10fe5cabd6900e3c9d698470af054
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/python:3.9
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/python:3.9, but refreshing...
3.9: Pulling from cimg/python
Digest: sha256:c1b6220cb31a456613ab628c70831166388b7f37865c9dbc87b5ff80ca48e313
Status: Image is up to date for cimg/python:3.9
cimg/python:3.9:
  using image cimg/python@sha256:c1b6220cb31a456613ab628c70831166388b7f37865c9dbc87b5ff80ca48e313
  pull stats: Image was already available so the image was not pulled
  time to create container: 43ms
Creating Docker containers in parallel 
Time to upload agent and config: 1.463058971s
Error: 
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/888d43cae73f845ab9f699eeb5485e3080569fb65a68d424e3b6b67fd76f4a87/6b20aeb06ee03bb37a119d4008fb5c593cb4599816df51f7a659f378df8b98f1" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/888d43cae73f845ab9f699eeb5485e3080569fb65a68d424e3b6b67fd76f4a87/6b20aeb06ee03bb37a119d4008fb5c593cb4599816df51f7a659f378df8b98f1" with domain controllers -- it is in an invalid state: unknown
rohan-kulkarni-phx commented 2 years ago

Hi, getting the following on Monterey 12.2, MacBook Pro 16-inch 2019 (intel)

$ uname -a
21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64 x86_64
$ rm ~/.circleci/build_agent_settings.json
$ circleci config process .circleci/config.yml > process.yml
$ circleci local execute -c process.yml --job lint-and-test

Output:

Docker image digest: sha256:dad8f737004adbb7722e416d99b53433f0c10fe5cabd6900e3c9d698470af054
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/python:3.9
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/python:3.9, but refreshing...
3.9: Pulling from cimg/python
Digest: sha256:c1b6220cb31a456613ab628c70831166388b7f37865c9dbc87b5ff80ca48e313
Status: Image is up to date for cimg/python:3.9
cimg/python:3.9:
  using image cimg/python@sha256:c1b6220cb31a456613ab628c70831166388b7f37865c9dbc87b5ff80ca48e313
  pull stats: Image was already available so the image was not pulled
  time to create container: 43ms
Creating Docker containers in parallel 
Time to upload agent and config: 1.463058971s
Error: 
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/888d43cae73f845ab9f699eeb5485e3080569fb65a68d424e3b6b67fd76f4a87/6b20aeb06ee03bb37a119d4008fb5c593cb4599816df51f7a659f378df8b98f1" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/888d43cae73f845ab9f699eeb5485e3080569fb65a68d424e3b6b67fd76f4a87/6b20aeb06ee03bb37a119d4008fb5c593cb4599816df51f7a659f378df8b98f1" with domain controllers -- it is in an invalid state: unknown

FYI the workaround in #676 worked for me as well

steven-dicristofaro commented 2 years ago

Can also confirm the workaround from @deokseong-kim-toss re: https://github.com/CircleCI-Public/circleci-cli/issues/676 solves the issue for me as well. Thank you for sharing that!

mrickard commented 2 years ago

@sebastian-lerner I'm having the same problem.

MacOS: Monterey (12.3.1) circleci: 0.1.16947+263902a (homebrew) Docker Desktop: 4.6.1 (76265).

The problem occurs with cimg/14.19.0, as well as cimg/14.19.1 and cimg/14.18.2.

I've removed the build agent file, though the error I'm getting is consistently this:

Error: Unhandled prepare executor error: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/cab3a2678cf64754071971f9089730323c1f06468efe1002e50e5ec3115345e6/15512e227f7fde93a3b0b5b6300679978c5ad0382b34eadd79fa8131f8260393" with domain controllers -- it is in an invalid state: unknown

matschaffer commented 2 years ago

@mrickard the workaround on https://github.com/CircleCI-Public/circleci-cli/issues/676 may help

mrickard commented 2 years ago

@matschaffer Thank you! I have been able to get that to work.

sebastian-lerner commented 2 years ago

Hey folks, a quick update on this. We believe we've identified the root cause with the latest issue that has been reported and we think we've attributed it to a bug with Docker Desktop for Mac which we've filed here: https://github.com/docker/for-mac/issues/6288

We're in the process of working with Docker to try and get this fixed. I'll update when I have more

cc @Scipi @pete-woods

mzarismithnyc commented 2 years ago

Hey folks, (I'm from CircleCI btw). The workaround that we described before, still continue to use that. We have raised the issue with Docker but they will take awhile to resolve the cgroupsv2 issue. We will update folks here on the thread once they have resolved the issue and thank you for your patience.

mkotsollaris commented 2 years ago

FWIW, the issue still persists, even if ~/.circleci/build_agent_settings.json is no longer present in macOS:

cd ~/.circleci/
USER:.circleci USER$ ls
cli.yml                 update_check.yml

Would be nice to see if there's any workaround for this 🙏

mzarismithnyc commented 2 years ago

We're still waiting to see if there is a workaround but at the moment, we're still investigating with folx.