search

CircleCI-Public / circleci-cli

Use CircleCI from the command line
https://circleci-public.github.io/circleci-cli/
MIT License
411 stars 232 forks source link

[circleci local execute] error looking up cgroup: not implemented for cgroup v2 unified hierarchy #589

Closed ldorau closed 2 years ago

ldorau commented 3 years ago

Meta:

CircleCI CLI Version: 0.1.15195+595cc9e (release)

Operating System: Arch Linux (latest)

CircleCI CLI Diagnostic:

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /home/ldorau/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API... 
Ok.
Hello, Lukasz Dorau.

Current behavior:

$ circleci local execute
Docker image digest: sha256:fc6b14595a231dd8417c97fad26f5f5f294f64b6f58443d93dfba17186f9f738
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.9.14-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Expected behavior:

"circleci local execute" works

When did this begin?

Now

Was this previously working?:

No, never.

mshang816 commented 3 years ago

got the same issue. I have docker installed and working properly.

drownbes commented 3 years ago

Same for me

circleci version 
0.1.15224+605a617 (release)

Build-agent version  ()
System information:
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.12.9-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
Draiken commented 3 years ago

Same issue

➜  hub-core-rb 82c60b9c:luiz/ch85328/e2e-docker  circleci local execute
Downloading latest CircleCI build agent...
Docker image digest: sha256:e35e2f2d58aed45b54bdf31dd5dd933626e28b10152d92b6b41c01029c8e9b82
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.12.10-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

I'm fairly certain this will only work if the kernel parameter systemd.unified_cgroup_hierarchy=false is set. This is not a runtime parameter so it has to be set on kernel compilation or at the bootloader

dreamsmasher commented 3 years ago

Same here, guess everybody's on Arch.

> circleci version
0.1.15384+149d854 (release)

Build-agent version  ()
System information:
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.12.12-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
fanninpm commented 3 years ago

I get the same error, on Manjaro (an Arch derivative).

Docker image digest: sha256:9524aa2e013fc3b8fe0244034a59af8d71d9f32c77b33d41a008b3a533f2cc77
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.12.9-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy
mike-hearn commented 3 years ago

I updated my Arch packages, annnnnnd here we are.

====>> Spin up environment
Build-agent version  ()
Docker Engine Version: 20.10.7
Kernel Version: Linux b20d89cca69a 5.11.22-2-MANJARO #1 SMP PREEMPT Fri May 21 17:45:54 UTC 2021 x86_64 Linux
Error: 
Unexpected environment preparation error: error looking up cgroup: mountpoint for cpu not found

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: mountpoint for cpu not found
dgoffredo commented 3 years ago

Looks like this error from this library.

These docs say that with a recent version of the library and with a recent Linux kernel, the kernel parameter systemd.unified_cgroup_hierarchy=1 will cause cgroups v2 to be used by the library. However, I have this parameter on my system (as part of GRUB_CMDLINE_LINUX and GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub), but I still get the error mentioned in this issue.

It looks like circleci-cli needs to update its version of runc [edit: I did some digging around, and it seems more likely that the non-public circleci/picard Docker image is what needs to update runc.]. See, for example, a similar error message in another issue:

docker: Error response from daemon: OCI runtime create failed: this version of runc doesn't work on cgroups v2: unknown.

Edit: Update

I pulled circleci/picard and poked around inside:


$ docker run --rm --interactive --tty --entrypoint=/bin/sh circleci/picard

/ #

/ # cd /opt/circleci

/opt/circleci # grep 'runc@' ./linux/arm64/circleci-agent
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/user/lookup.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/user/user.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/configs/namespaces_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/configs/namespaces_syscall.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/cgroups/utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/cgroups/v1_utils.go

It looks like the latest circleci/picard is using runc@v1.0.0-rc92. In that release, cgroupv2 support was still considered experimental per the previous release (rc91), and wasn't considered "fully ready for production" until the following release (rc93).

DominikWolek commented 3 years ago

Same issue on Arch Linux:

Docker image digest: sha256:151ec092e49f9a333d76e111818289af750d1b31d3a0ee2f22e24214e2724d49
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.12.14-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy
fanninpm commented 3 years ago

I have more information about this problem that the CircleCI folks are conveniently ignoring. (Yes, I'm still using the AUR package, not the snap.)

Downloading latest CircleCI build agent...
Docker image digest: sha256:26c7e7103a6e2e20816ab4fb2529700910db617b2da3a642021d25cf0be36ba0
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.13.4-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
process panic
goroutine 1 [running]:
runtime/debug.Stack(0x1476a36, 0x8, 0x0)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/circleci/build-agent/cmd/internal/setup.PanicHandler()
        /mnt/ramdisk/circleci/project/cmd/internal/setup/handlers.go:23 +0xce
panic(0x12b2b20, 0x1ea31d0)
        /usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/circleci/build-agent/outer/docker.(*DockerExecutor).Cleanup(0xc000120000, 0x16dc7a8, 0xc000519560, 0x0, 0x0)
        /mnt/ramdisk/circleci/project/outer/docker/cleanup.go:35 +0x223
github.com/circleci/build-agent/outer.ConfigureAndRun.func4(0x16e0618, 0xc000120000)
        /mnt/ramdisk/circleci/project/outer/run.go:228 +0x14d
github.com/circleci/build-agent/outer.ConfigureAndRun(0x16dc7e0, 0xc00003c100, 0xc0000d6c78, 0x15, 0x0, 0x0, 0xc000040120, 0x16, 0x0, 0x0, ...)
        /mnt/ramdisk/circleci/project/outer/run.go:260 +0x22c0
github.com/circleci/build-agent/cmd/internal/outer/local.runExecuteJob(0xc00048c700, 0x5fec00, 0x169c2d0, 0x12, 0x0, 0x0)
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:123 +0x295
github.com/circleci/build-agent/cmd/internal/outer/local.NewExecuteJobCmd.func1(0xc000480280, 0xc0004edd20, 0x0, 0x2, 0x0, 0x0)
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:65 +0x45
github.com/spf13/cobra.(*Command).execute(0xc000480280, 0xc0004edd00, 0x2, 0x2, 0xc000480280, 0xc0004edd00)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc000480000, 0xc00003d190, 0x3, 0xc00009a070)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/circleci/build-agent/cmd.Execute()
        /mnt/ramdisk/circleci/project/cmd/root.go:64 +0x194
main.main()
        /mnt/ramdisk/circleci/project/main.go:14 +0xdb

I wonder if this will become a bigger problem when Debian Bullseye gets released as stable.

patrickclery commented 3 years ago

I can confirm that the fix mentioned by @dgoffredo works for Manjaro.

How to fix

sudo vim /etc/defaults/grub

Change this line:

GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"

Update Grub:

sudo update-grub

Reboot.

dgoffredo commented 3 years ago

I still get the following error when trying to locally run a CI job from a repository I work on:

$ circleci local execute
Docker image digest: sha256:5bc33117a2ea3be8ed7cc8aab995fc69906a1112249e814fa8e7476b570a03a0
====>> Spin up environment
Build-agent version  ()
System information:
Error determining system info
Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

I see that the most recent circleci/picard image has a newer runc version:

$ docker pull circleci/picard
Using default tag: latest
latest: Pulling from circleci/picard
29291e31a76a: Pull complete 
8afb3f12a298: Pull complete 
f150479d1511: Pull complete 
2e647f59c548: Pull complete 
da193f18bce6: Pull complete 
7c602d616b9c: Pull complete 
c402e98532e9: Pull complete 
Digest: sha256:5bc33117a2ea3be8ed7cc8aab995fc69906a1112249e814fa8e7476b570a03a0
Status: Downloaded newer image for circleci/picard:latest
docker.io/circleci/picard:latest

$ docker run --rm --interactive --tty --entrypoint=/bin/sh circleci/picard
/ # cd /opt/circleci/linux/amd64
/opt/circleci/linux/amd64 # ls
circleci-agent
/opt/circleci/linux/amd64 # grep 'runc@' circleci-agent 
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.1/libcontainer/user/user.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.1/libcontainer/devices/device_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.1/libcontainer/configs/namespaces_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.1/libcontainer/configs/namespaces_syscall.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.1/libcontainer/userns/userns_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.1/libcontainer/user/lookup_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.1/libcontainer/cgroups/utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.1/libcontainer/cgroups/v1_utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.1/libcontainer/cgroups/file.go
/opt/circleci/linux/amd64 # 

$

I tried running all of sudo circleci update, circleci update build-agent, and reinstalling circleci from source based off of the current master branch. None of those prevent the error message above on my system.

cgroup v2 is enabled already:

$ grep 'cgroup' /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=1"
# Enable cgroup v2.  See <https://kind.sigs.k8s.io/docs/user/rootless/>.
GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"

$
DavidS-cloud commented 3 years ago

FWIW, I think systemd.unified_cgroup_hierarchy needs to be set to 0 (zero) to enable cgroup v1 behaviour. At least it worked for me when setting it to 1 did not.

snowinferno commented 2 years ago

I'm experiencing this same issue on OSX Big Sur. Happens whether I use brew install circlecli or use the provided curl command for Mac/Linux

$ circleci diagnostic

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /Users/greg.wilburn/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API...
Ok.
Hello, Greg Wilburn.
$ circleci local execute

Docker image digest: sha256:130f2f80984be6c2336ae57c48cd2b666061a82660740e027ac63e96eed8c0ad
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Error:
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy
umihico commented 2 years ago

My macbook is facing same error. circleci/picard version can be specified by modifing ~/.circleci/build_agent_settings.json JFYI. I tried some but couldn't get fixed yet. Could be not related.

miketheman commented 2 years ago

Hitting this today on MacOS Monterey.

$ circleci local execute --job build
Docker image digest: sha256:e025cba3f5236a7e3ed9b97dfadce44c824ea0cc1ae1427823ff4c1dd2f420c3
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Error:
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

This is using the latest circleci/picard - released ~an hour ago - https://hub.docker.com/layers/circleci/picard/1.0.95690-99ec9ada/images/sha256-e025cba3f5236a7e3ed9b97dfadce44c824ea0cc1ae1427823ff4c1dd2f420c3?context=explore

We can also see that runc is up to 1.0.3:

docker run --rm --interactive --tty --entrypoint=/bin/sh circleci/picard
/ # cd /opt/circleci
/opt/circleci # grep 'runc@' ./linux/arm64/circleci-agent
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/user/user.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/devices/device_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/configs/namespaces_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/configs/namespaces_syscall.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/userns/userns_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/user/lookup_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/cgroups/utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/cgroups/v1_utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/cgroups/file.go

This is bothersome, since I can no longer test out circleci changes and have to use remote resources to debug.

mjsteinbaugh commented 2 years ago

I'm seeing this also when attempting to test bioconda builds on a MacBook locally.

podplatnikm commented 2 years ago

Any update on this?

circleci local execute --job test
Docker image digest: sha256:eb0c4688f843e6987c29f72c9fdf8c910d59b2a72c7e1559d46a3b1837038cdb
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy
circleci version
0.1.16535+5010eb7 (homebrew)
kienstra commented 2 years ago

Also seeing this with Docker Desktop 4.3.1 (72247) and a 2019 MacBook Pro running macOS 12.0.1 (21A559). It was fine with Docker Desktop 4.2.0.

====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Running the latest CircleCI CLI (v0.1.16535).

vingov commented 2 years ago

Running into the same issue with the latest CircleCI CLI (v0.1.16535) after upgrading the local docker desktop to the latest version 4.3.1 (72247) on Mac.

====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Error:
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
sebastian-lerner commented 2 years ago

Hey folks, I'm a product manager here at CircleCI. Thank you for making us aware of the problem and apologies for this loss of functionality. Our engineering team is investigating a fix. I unfortunately do not have an estimate at this time on when we'll get this fixed but as soon as I have more information I'll update this thread.

We'll be updating the local CLI docs page to call out this limitation in the time being.

Thanks all for your patience.

kienstra commented 2 years ago

Hi Sebastian, thanks a lot for working on this, I really appreciate it!

pete-woods commented 2 years ago

We believe we have a fix for this issue in the CircleCI build-agent (Docker image circleci/picard) now.

To accelerate your rollout of the agent, you can run: rm ~/.circleci/build_agent_settings.json to get the latest version.

mike-hearn commented 2 years ago

After deleting the build agent file and re-running, I'm now getting a different error on Manjaro:


Error:   error starting container circleci/python:3.7: Error response from daemon: cgroup-parent for systemd cgroup should be a valid slice named as "xxx.slice"

Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as circleci/python:3.7, but refreshing...
3.7: Pulling from circleci/python
Digest: sha256:3de62879e0b899d3de2a836957c736e6711c5a3faabd0ea2ce28bb49cf5e4ae2
Status: Image is up to date for circleci/python:3.7
Error: 
Unexpected environment preparation error: Error response from daemon: cgroup-parent for systemd cgroup should be a valid slice named as "xxx.slice"
Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: cgroup-parent for systemd cgroup should be a valid slice named as "xxx.slice"```
pete-woods commented 2 years ago

Seems like we have further debugging work to do on Linux. This was tested primarily on Docker Desktop for Mac.

pete-woods commented 2 years ago

@mike-hearn could you post your version of:

====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

for comparison?

pete-woods commented 2 years ago

(I'm interested in which Cgroup Driver you have)

mike-hearn commented 2 years ago

Sure:

====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.11.22-2-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
pete-woods commented 2 years ago

Thanks - that confirms what I suspected. We'll have to add support for the systemd driver, in addition to the cgroupfs driver.

fanninpm commented 2 years ago

Cgroup Driver: systemd

(Oh boy, Lennart. You made this a fun one.)

Mind posting which version of systemd you have, for extra clarity? The command should be /sbin/init --version.

mike-hearn commented 2 years ago

systemd 248 (248.3-3-manjaro) +PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

kienstra commented 2 years ago

Thanks a lot! It's working now on my Mac with https://github.com/CircleCI-Public/circleci-cli/issues/589#issuecomment-1005865018.

pete-woods commented 2 years ago

I've just released a new version of the CircleCI build-agent. Could anyone running on Linux with the systemd Cgroup driver please try updating and running again?

pete-woods commented 2 years ago

@mike-hearn do you have a moment to check if the latest release has fixed your issue (it seems resolved on an Ubuntu 21.10 machine for me)

pete-woods commented 2 years ago

Resolving this issue, unless we hear back from anyone about it not working.

bhamail commented 2 years ago

@pete-woods Joy! Local CircleCI builds are happy again on MacOS (after rm ~/.circleci/build_agent_settings.json). Thanks for this fix!

jtarchie commented 2 years ago

cc: @pete-woods

I've been following the steps about for MacOS. It is not working for me, though.

Given the .circleci/config.yml

orbs: # declare what orbs we are going to use
  node: circleci/node@2.0.2 # the node orb provides common node-related configuration

version: 2.1 # using 2.1 provides access to orbs and other features

workflows:
  matrix-tests:
    jobs:
      - node/test:
          version: 16.13.2

Then running the circleci CLI.

$ circleci local execute --job "node/test"
Docker image digest: sha256:156a80a311fbad52d73d2391f5951604f041c687d8b4e0f87624255af8f2ef72
====>> Spin up environment
Build-agent version  ()
process panic
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/circleci/build-agent/cmd/internal/setup.PanicHandler()
        /mnt/ramdisk/circleci/project/cmd/internal/setup/handlers.go:23 +0x85
panic({0x145e780, 0x24a3dd0})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/opencontainers/runc/libcontainer/cgroups/fs2.defaultDirPath(0xc000a13840)
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/defaultpath.go:35 +0x18
github.com/opencontainers/runc/libcontainer/cgroups/fs2.NewManager(0x0, {0x0, 0x0})
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/fs2.go:31 +0x2c
github.com/circleci/build-agent/outer/docker/cgroups.cgroupsV2({0xc00063cb00, 0x8})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:51 +0x6e
github.com/circleci/build-agent/outer/docker/cgroups.GetSelf({0xc00063cb00, 0x8})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:25 +0xa5
github.com/circleci/build-agent/outer/docker.NewDockerExecutorWithClient({{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:70 +0x13e
github.com/circleci/build-agent/outer/docker.NewDockerExecutor({{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:103 +0xdc
github.com/circleci/build-agent/outer.newExecutor({_, _}, {{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/run.go:372 +0x265
github.com/circleci/build-agent/outer.ConfigureAndRun({_, _}, {{0xc000726438, 0x15}, {0x0, 0x0}, {0xc0007265a0, 0x16}, {0x0, 0x0}, ...})
        /mnt/ramdisk/circleci/project/outer/run.go:219 +0x17ad
github.com/circleci/build-agent/cmd/internal/outer/local.runExecuteJob(0xc0000a0380, 0x0, {0x19862b0, 0x0})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:125 +0x2b6
github.com/circleci/build-agent/cmd/internal/outer/local.NewExecuteJobCmd.func1(0xc000349400, {0x165b78d, 0x4, 0x4})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:64 +0x29
github.com/spf13/cobra.(*Command).execute(0xc000349400, {0xc000731140, 0x4, 0x4})
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000349180)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/circleci/build-agent/cmd.Execute()
        /mnt/ramdisk/circleci/project/cmd/root.go:64 +0x2cf
main.main()
        /mnt/ramdisk/circleci/project/main.go:14 +0xca

I can create a new issue if need be, but it seems related because of the cgroups in the stacktrace.

CLI Version: 0.1.16535+5010eb7 (homebrew) Docker Desktop Mac: Docker Desktop 4.4.2 (73305)

bhamail commented 2 years ago

@jtarchie Just to make sure, did you clear out the old local cache by first running: rm ~/.circleci/build_agent_settings.json ?

jtarchie commented 2 years ago

I did.

s4bs commented 2 years ago

cc: @pete-woods

I've been following the steps about for MacOS. It is not working for me, though.

Given the .circleci/config.yml

orbs: # declare what orbs we are going to use
  node: circleci/node@2.0.2 # the node orb provides common node-related configuration

version: 2.1 # using 2.1 provides access to orbs and other features

workflows:
  matrix-tests:
    jobs:
      - node/test:
          version: 16.13.2

Then running the circleci CLI.

$ circleci local execute --job "node/test"
Docker image digest: sha256:156a80a311fbad52d73d2391f5951604f041c687d8b4e0f87624255af8f2ef72
====>> Spin up environment
Build-agent version  ()
process panic
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/circleci/build-agent/cmd/internal/setup.PanicHandler()
        /mnt/ramdisk/circleci/project/cmd/internal/setup/handlers.go:23 +0x85
panic({0x145e780, 0x24a3dd0})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/opencontainers/runc/libcontainer/cgroups/fs2.defaultDirPath(0xc000a13840)
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/defaultpath.go:35 +0x18
github.com/opencontainers/runc/libcontainer/cgroups/fs2.NewManager(0x0, {0x0, 0x0})
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/fs2.go:31 +0x2c
github.com/circleci/build-agent/outer/docker/cgroups.cgroupsV2({0xc00063cb00, 0x8})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:51 +0x6e
github.com/circleci/build-agent/outer/docker/cgroups.GetSelf({0xc00063cb00, 0x8})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:25 +0xa5
github.com/circleci/build-agent/outer/docker.NewDockerExecutorWithClient({{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:70 +0x13e
github.com/circleci/build-agent/outer/docker.NewDockerExecutor({{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:103 +0xdc
github.com/circleci/build-agent/outer.newExecutor({_, _}, {{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/run.go:372 +0x265
github.com/circleci/build-agent/outer.ConfigureAndRun({_, _}, {{0xc000726438, 0x15}, {0x0, 0x0}, {0xc0007265a0, 0x16}, {0x0, 0x0}, ...})
        /mnt/ramdisk/circleci/project/outer/run.go:219 +0x17ad
github.com/circleci/build-agent/cmd/internal/outer/local.runExecuteJob(0xc0000a0380, 0x0, {0x19862b0, 0x0})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:125 +0x2b6
github.com/circleci/build-agent/cmd/internal/outer/local.NewExecuteJobCmd.func1(0xc000349400, {0x165b78d, 0x4, 0x4})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:64 +0x29
github.com/spf13/cobra.(*Command).execute(0xc000349400, {0xc000731140, 0x4, 0x4})
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000349180)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/circleci/build-agent/cmd.Execute()
        /mnt/ramdisk/circleci/project/cmd/root.go:64 +0x2cf
main.main()
        /mnt/ramdisk/circleci/project/main.go:14 +0xca

I can create a new issue if need be, but it seems related because of the cgroups in the stacktrace.

CLI Version: 0.1.16535+5010eb7 (homebrew) Docker Desktop Mac: Docker Desktop 4.4.2 (73305)

Same issue for me.

CLI Version: 0.1.16535+5010eb7 (homebrew)

jonpeterson commented 2 years ago

I'm also having the same stacktrace as @jtarchie above.

macOS: 12.1 on M1 (ARM) processor.

CLI version: 0.1.16535+5010eb7 (homebrew)

Docker version:

Client:
 Cloud integration: v1.0.22
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:46:56 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:07 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
ghost commented 2 years ago

Same issue on 2020 i7 Macbook Pro MacOS 12.1 CLI : 0.1.16535+5010eb7 (homebrew) Docker: version 20.10.12, build e91ed57

sokorahen-szk commented 2 years ago

I'm facing a similar bug.

Mac OS: 12.1 Docker version 20.10.12, build e91ed57 CLI version: 0.1.16535+5010eb7 (homebrew)

mike-hearn commented 2 years ago

I've switched from Manjaro to Debian (bullseye) and I'm now seeing essentially the same stack trace as the others:

Docker image digest: sha256:3b40ae3c6445ba88fefdcd6803e132c1a5a0999783d75640568ad20a766f1762
====>> Spin up environment
Build-agent version  ()
process panic
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/circleci/build-agent/cmd/internal/setup.PanicHandler()
        /mnt/ramdisk/circleci/project/cmd/internal/setup/handlers.go:23 +0x85
panic({0x1460aa0, 0x24a6dd0})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/opencontainers/runc/libcontainer/cgroups/fs2.defaultDirPath(0xc000563a08)
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/defaultpath.go:35 +0x18
github.com/opencontainers/runc/libcontainer/cgroups/fs2.NewManager(0x0, {0x0, 0x0})
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/fs2.go:31 +0x2c
github.com/circleci/build-agent/outer/docker/cgroups.cgroupsV2({0xc000c80ba9, 0x7})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:51 +0x6e
github.com/circleci/build-agent/outer/docker/cgroups.GetSelf({0xc000c80ba9, 0x7})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:25 +0xa5
github.com/circleci/build-agent/outer/docker.NewDockerExecutorWithClient({{{{0xc000a86240, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:70 +0x13e
github.com/circleci/build-agent/outer/docker.NewDockerExecutor({{{{0xc000a86240, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:103 +0xdc
github.com/circleci/build-agent/outer.newExecutor({_, _}, {{{{0xc000a86240, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/run.go:369 +0x265
github.com/circleci/build-agent/outer.ConfigureAndRun({_, _}, {{0xc000040c90, 0x15}, {0x0, 0x0}, {0xc000040df8, 0x16}, {0x0, 0x0}, ...})
        /mnt/ramdisk/circleci/project/outer/run.go:219 +0x17ad
github.com/circleci/build-agent/cmd/internal/outer/local.runExecuteJob(0xc0000e2380, 0x0, {0x1988890, 0x0})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:125 +0x2b6
github.com/circleci/build-agent/cmd/internal/outer/local.NewExecuteJobCmd.func1(0xc000a20780, {0x165de4d, 0x4, 0x4})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:64 +0x29
github.com/spf13/cobra.(*Command).execute(0xc000a20780, {0xc00049f680, 0x4, 0x4})
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000a20500)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/circleci/build-agent/cmd.Execute()
        /mnt/ramdisk/circleci/project/cmd/root.go:64 +0x2cf
main.main()
        /mnt/ramdisk/circleci/project/main.go:14 +0xca

Docker version info:

Client:
 Version:           20.10.5+dfsg1
 API version:       1.41
 Go version:        go1.15.15
 Git commit:        55c4c88
 Built:             Sat Dec  4 10:53:03 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.5+dfsg1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.15
  Git commit:       363e9a8
  Built:            Sat Dec  4 10:53:03 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12~ds1
  GitCommit:        1.4.12~ds1-1~deb11u1
 runc:
  Version:          1.0.0~rc93+ds1
  GitCommit:        1.0.0~rc93+ds1-5+b2
 docker-init:
  Version:          0.19.0
  GitCommit:
jtarchie commented 2 years ago

👍 the comments. That’s what they are there for.

mike-hearn commented 2 years ago

I was specifically asked for feedback by Pete. Apologies if it's cluttering your inbox.

sebastian-lerner commented 2 years ago

Hey folks, thank you for letting us know that you're experiencing issues. We believe we have identified the root cause and are working on pushing a fix. Will update when we have more. cc @bjohnso5 @pete-woods

sebastian-lerner commented 2 years ago

We believe we have rolled out a fix for this recently reported issue.

To accelerate your rollout of the agent, you can run: rm ~/.circleci/build_agent_settings.json to get the latest version.

Please let us know if this still does not fix the issue for you.

jtarchie commented 2 years ago

Can confirm, resolved for me. Thank you @sebastian-lerner.

To make sure everyone is one the same page of instructions:

$ rm ~/.circleci/build_agent_settings.json # this is important
$ circleci local execute --job "node/test"

Docker image digest: sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/node:16.13.2
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/node:16.13.2, but refreshing...
cimg/node:16.13.2:
  using image cimg/node@sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
  pull stats: Image was already available so the image was not pulled
  time to create container: 41ms
Creating Docker containers in parallel 
16.13.2: Pulling from cimg/node
Digest: sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
Status: Image is up to date for cimg/node:16.13.2
Time to upload agent and config: 2.026862226s
Time to start containers: 371.585657ms
jmstone617 commented 2 years ago

I've tried removing the build agent file already but am still getting the following error:

Build-agent version  ()
System information:
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64

Starting container hashicorp/terraform:1.0.0
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as hashicorp/terraform:1.0.0, but refreshing...
hashicorp/terraform:1.0.0:
  using image hashicorp/terraform@sha256:6dafbe34b7c18d189ff18e3dfb7cd5fc4144a2714eb2bc7b6dd9352455e8848a
  pull stats: Image was already available so the image was not pulled
  time to create container: 41ms
Creating Docker containers in parallel
1.0.0: Pulling from hashicorp/terraform
Digest: sha256:6dafbe34b7c18d189ff18e3dfb7cd5fc4144a2714eb2bc7b6dd9352455e8848a
Status: Image is up to date for hashicorp/terraform:1.0.0
Time to upload agent and config: 4.498269919s
Error:
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/b73bc306f5f0b63dbbf7e3ca471cb09d1693082ed24b022c18044c13949b87c5/abe49607cf6cebfc32f940ef1e58d1244b0c4cdc7aecb0a2f62cccfb9b394b56" with domain controllers -- it is in an invalid state: unknown
steven-dicristofaro commented 2 years ago

+1 on issue: Running Catalina 10.15.7 and seeing the same error as @jmstone617 . I thought at first it was per my specific config.yml (I was still testing with my original file), but I duplicated the simple config file from @jtarchie, ran the identical commands to remove the build settings file and execute the job locally and still seeing the following error:

rm ~/.circleci/build_agent_settings.json circleci local execute --job "node/test"

Downloading latest CircleCI build agent...
Docker image digest: sha256:6b954f968479e94b5a5a687b467540d9df7359c7e03230052596f0c41c07b3d8
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/node:16.13.2
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image cache not found on this host, downloading cimg/node:16.13.2
16.13.2: Pulling from cimg/node
f3ef4ff62e0d: Pull complete 
2c322a662952: Pull complete 
fb8baef6d8e7: Pull complete 
805b7ba0a256: Pull complete 
63833a96b952: Pull complete 
5d1ffe5f9ba5: Pull complete 
6dbfea31d6de: Pull complete 
44f09c08a855: Pull complete 
Digest: sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
Status: Downloaded newer image for cimg/node:16.13.2
cimg/node:16.13.2:
  using image cimg/node@sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
  pull stats: download 343.3MiB in 43.33s (7.923MiB/s), extract 358.7MiB in 24.413s (14.69MiB/s)
  time to create container: 273ms
Creating Docker containers in parallel 
Time to upload agent and config: 2.0054758s
Error: 
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/3026c5aac7c67fe6af86320edf46e0730deccb4f3e88c2631e3240b976f7b706/e264c13293b89594dc31b307a5b8cc5e183efb2704ce2ac28184e3852a2c7ef0" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/3026c5aac7c67fe6af86320edf46e0730deccb4f3e88c2631e3240b976f7b706/e264c13293b89594dc31b307a5b8cc5e183efb2704ce2ac28184e3852a2c7ef0" with domain controllers -- it is in an invalid state: unknown