Error docker inspect runner-... failed with exit status: 1

[shandak]

Checklist

• [X] I've looked through the issues and pull requests for similar reports

Describe your issue

A similar issue I found there https://github.com/cross-rs/cross/issues/1153 but CROSS_REMOTE=1 is set and does not help. For some reason build does not work anymore(it worked without problems ~6 months before) and I can't figure out what the issue is. Below some related verbose info and error at the end:

cross -V
cross 0.2.5 (ed713d9 2023-05-23)
[cross] warning: unable to get metadata for package
[cross] note: Falling back to `cargo` on the host.
cargo 1.70.0 (ec8a8a0ca 2023-04-25)

docker info
WARNING: API is accessible on http://0.0.0.0:2375/ without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: No swap limit support
Client: Docker Engine - Community
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-164-generic
 Operating System: Alpine Linux v3.14 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.761GiB
 Name: fbdf80109021
 ID: WM7C:Y4ND:WKWY:3PPC:HLXQ:2XEP:Y7OJ:5RAI:6LHX:C3QB:YY3T:YPSU
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  [127](https://git.slab.pp.ua/root/analytics-cli/-/jobs/599#L127).0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

cross build -v --target x86_64-unknown-linux-musl --release
+ cargo metadata --format-version 1 --filter-platform x86_64-unknown-linux-musl
+ rustc --print sysroot
+ /usr/bin/docker
+ /usr/bin/docker version -f '{{ .Server.Os }},,,{{ .Server.Arch }}'
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup target add x86_64-unknown-linux-musl --toolchain stable-x86_64-unknown-linux-gnu
info: downloading component 'rust-std' for 'x86_64-unknown-linux-musl'
info: installing component 'rust-std' for 'x86_64-unknown-linux-musl'
+ /usr/bin/docker inspect runner-xqkuh56d-project-43-concurrent-0
Error: 
   0: `/usr/bin/docker inspect runner-xqkuh56d-project-43-concurrent-0` failed with exit status: 1
Location:
   /rustc/90c54[180](https://git.slab.pp.ua/root/analytics-cli/-/jobs/591#L180)6f23a127002de5b4038be731ba1458ca/library/core/src/convert/mod.rs:727
Stderr:
   Error: No such object: runner-xqkuh56d-project-43-concurrent-0

This is the dind config with CROSS_REMOTE = 1 for gitlab ci:

  image: vshandak/cross:builder
  services:
    - name: docker:20.10.9-dind
      alias: docker
      command: ["--tls=false"]
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
    GIT_SSL_NO_VERIFY: "1"
    CROSS_REMOTE: 1
    CROSS_REMOTE_COPY_REGISTRY: 1

Dockefile body in use:

FROM debian:buster-20230502-slim

RUN apt-get update && apt-get install -y curl build-essential

# Install rustup
ENV RUSTUP_HOME=/opt/rustup
ENV CARGO_HOME=/opt/cargo
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="${PATH}:/opt/cargo/bin"

ENV RUST_TOOLCHAIN="stable"

RUN rustup toolchain install "${RUST_TOOLCHAIN}"
RUN rustup default "${RUST_TOOLCHAIN}"

RUN rustup component add clippy
RUN rustup component add rustfmt

RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN apt-get install -y software-properties-common apt-transport-https ca-certificates curl gnupg2
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian buster stable"
RUN apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io libssl-dev pkg-config

# set CROSS_CONTAINER_IN_CONTAINER to inform `cross` that it is executed from within a container
ENV CROSS_CONTAINER_IN_CONTAINER=true

RUN cargo install cross --git https://github.com/cross-rs/cross

What target(s) are you cross-compiling for?

x86_64-unknown-linux-musl

Which operating system is the host (e.g computer cross is on) running?

• [ ] macOS
• [ ] Windows
• [X] Linux / BSD
• [ ] other OS (specify in description)

What architecture is the host?

• [X] x86_64 / AMD64
• [ ] arm32
• [ ] arm64 (including Mac M1)

What container engine is cross using?

• [X] docker
• [ ] podman
• [ ] other container engine (specify in description)

cross version

cross 0.2.5 (ed713d9 2023-05-23)

Example

No response

Additional information / notes

No response

[Emilgardis]

this is on a gitlab instance right? has there maybe been an update done in the base container that could've changed something?

[shandak]

it looks like cross tries to get info from not available container docker inspect runner-xqkuh56d-project-43-concurrent-0. I called to check list of containers docker container ls it returned an empty list CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

[Emilgardis]

We get the name using $HOSTNAME here

[shandak]

yeah, I saw it, but Docker does not see any of the containers inside. Seems no one value fits because nothing is available.

[Emilgardis]

can you run docker context inspect please?

[shandak]

sure

docker context inspect
[
    {
        "Name": "default",
        "Metadata": {},
        "Endpoints": {
            "docker": {
                "Host": "tcp://docker:2375/",
                "SkipTLSVerify": false
            }
        },
        "TLSMaterial": {},
        "Storage": {
            "MetadataPath": "\u003cIN MEMORY\u003e",
            "TLSPath": "\u003cIN MEMORY\u003e"
        }
    }
]

[Emilgardis]

Interesting, that seems wrong

[shandak]

What do you expect to see from docker context inspect instead of mine result @Emilgardis ?

[Emilgardis]

Sorry, I meant that it seems correct.

I'm not sure how to proceed with this. docker inspect is used to query about different kinds of objects in docker, and I can't remember what specific thing it's supposed to return with docker inspect $HOSTNAME

[thomas725]

should we maybe try to ask Gitlab to restore that functionality using a bugreport on their tracker? https://gitlab.com/gitlab-org/gitlab/-/issues/new

or create a thread asking about it on their forum? https://forum.gitlab.com

[Rylan-Meilutis]

If you have the gitlab container spawn a new container that runs the cross command it allows the cross container to work correctly

script:
    - printenv | grep -v "^HOSTNAME=" > env.list
    - docker run --rm --env-file env.list -v /var/run/docker.sock:/var/run/docker.sock image:latest bash -c "git clone $CI_REPOSITORY_URL /data && cd /data && chmod +x build.sh && ./build.sh"