Unexpected environment preparation error: ... invalid name or ID supplied: ""

[ptwales]

Meta:

CircleCI CLI Version: 0.1.21812+72528b8 (release)

Operating System:

Build-agent version  ()
System information:
 Server Version: 20.10.18
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.19.11-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64

CircleCI CLI Diagnostic:

Debugger mode: false
Config found: /home/ptwales/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API...
Ok.
Hello, Philip Wales.

---

Current behavior:

for job in (cat .circleci/config.yml | yq '.jobs | keys[]' -r)
    circleci local execute --job $job
end

All iterations output the following:

Fetching latest build environment...
Docker image digest: sha256:c654083264021321481e472e35ce54c1c4403096896ee06904041bb05817cd7b
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.18
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.19.11-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64

Error:
Unexpected environment preparation error: error connecting build-agent to ephemeral network: error connecting build-agent container to ephemeral network: Error response from daemon: invalid name or ID supplied: ""

Step failed
Task failed
Error: Unhandled prepare executor error: error connecting build-agent to ephemeral network: error connecting build-agent container to ephemeral network: Error response from daemon: invalid name or ID supplied: ""

Expected behavior:

Ideally it would run the job locally, just as it would on github. More practically, I'd expect it to identify where in my local setup have I provided an invalid name or ID? Like a jq path to a missing field in .circleci/config.yml or ~/.circleci/cli.yml, if that's the case.

When did this begin / Was this previously working?:

It was working at some point over a year ago. Stopped working months ago, but it wasn't a major concern. I can't say for sure when it started doing this. I haven't needed to use it until I noticed docker images built on circleci had faults that local builds didn't.

Additional Information:

The repo that I'm running this against is in a private organization so that might be an issue.

[ptwales]

Fails against any repo with any public repo with circleci. It's not a private organization issue. Something's wrong with my machine. I also built it from source and still got the same error.

[ptwales]

Was able to get it working out of the box on a different machine with the exact same system information as above. I think it has to do with differences in docker configuration. Possibly interference from gcloud.

This is likely not a bug with circleci and will hopefully I can close it and leave it as documentation for prosperity.

[ptwales]

Issue isn't really with circleci-cli but the picard docker image circelci-cli downloads. The cli executes this docker command which spins up and fails on my machine.

docker run --interactive --tty --rm --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/466880726_circleci_config.yml:/tmp/local_build_config.yml --volume /home/ptwales/Code/Go/circleci-cli:/home/ptwales/Code/Go/circleci-cli --volume /home/ptwales/.circleci:/root/.circleci --workdir /home/ptwales/Code/Go/circleci-cli circleci/picard@sha256:0d6c17bb09da8707eabad913d61a68b28028906d48c176d6bd60ebd36cce222e circleci build --config /tmp/local_build_config.yml --job test

[ptwales]

Made a new user and confirmed it wasn't a user issue. Picard is for some reason trying to disconnect from a network with no name on my machine but it doesn't on other machine.

Oct 28 12:16:31 myhost dockerd[1238]: time="2022-10-28T12:16:31.263770830-05:00" level=error msg="Handler for POST /v1.41/networks/26fa2c1ee779f63f78c16836ce28ae5d44941d4b8e2c3d53e19b43b813fd8517/disconnect returned error: invalid name: "
Oct 28 12:16:31 myhost dockerd[1238]: time="2022-10-28T12:16:31.683610652-05:00" level=info msg="ignoring event" container=52264be0fe6fc74137fac61f31f1031035e9da4048343f527d6e3540cb60d04a module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

[particleflux]

Had the same issue, turns out I was missing some kernel options required by docker (which did not affect all the other images and compose projects I ran in the slightest though):

  CONFIG_NETFILTER_XT_MATCH_IPVS:        is not set when it should be.
  CONFIG_BLK_DEV_THROTTLING:     is not set when it should be.
  CONFIG_NET_CLS_CGROUP:         is not set when it should be.
  CONFIG_CFS_BANDWIDTH: is optional for container statistics gathering
  CONFIG_MACVLAN:        is not set when it should be.

I'd guess it's one or several of the network related options in there^^

[rsyring]

Probably dups #318. No resolution there either. :(

[rsyring]

@particleflux can you give more info about how you tracked down those kernel options where required? I ran Moby's check-config script and it showed everything green on my system. But I still have the same error you reported.

[ptwales]

FWIW I tried the solution to #589 despite it being a different error message and it worked.

To be specific, setting the kernel parameter systemd.unified_cgroup_hierarchy=false is what fixed it for me

(Edit, totally misread @particleflux's answer)

[particleflux]

@rsyring gentoo shows them on install - but here is the part of the ebuild which checks them:

https://github.com/gentoo/gentoo/blob/f6d706c27982bd7271d91a6af98cd08dc586c6b0/app-containers/docker/docker-23.0.4.ebuild#L65-L247