Running fails with "sh: write error: File exists" or "sh: can't create ... : Permission denied"

rolandjitsu commented 4 years ago

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind bug

Description:

I'm running this image on a Gitlab CI runner using Kubernetes with Docker in Docker (docker:19.03.12, docker:19:03.12-dind) and I started getting some errors:

Steps to reproduce the issue:

I install some apk packages (because the docker:19.03.12 image is a alpine based image):

- apk update
- |
apk add build-base \
  git \
  gnupg \
  libc6-compat \
  ncurses \
  openssh-client \
  openssl \
  python3 \
  py3-crcmod \
  wget

I install docker buildx

- mkdir -p ~/.docker/cli-plugins
- |
wget -O ~/.docker/cli-plugins/docker-buildx \
  --tries 5 \
  --retry-connrefused \
  --random-wait \
  https://github.com/docker/buildx/releases/download/$BUILDX_VERSION/buildx-$BUILDX_VERSION.linux-amd64
- chmod +x ~/.docker/cli-plugins/docker-buildx

I authenticate the docker CLI with Docker Hub

I install GCP SDK:

- mkdir -p $HOME
- | 
wget -O $HOME/google-cloud-sdk.tar.gz \
  --tries 5 \
  --retry-connrefused \
  --random-wait \
  https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-$GCLOUD_SDK_VERSION-linux-x86_64.tar.gz
- tar -xzf $HOME/google-cloud-sdk.tar.gz -C $HOME
- export PATH=$PATH:$HOME/google-cloud-sdk/bin
- gcloud config set core/disable_usage_reporting true
- gcloud config set component_manager/disable_update_check true
- gcloud config set metrics/environment github_docker_image
- gcloud auth activate-service-account -q --key-file $CI_PROJECT_DIR/secrets/release-ci-key.json
- gcloud --version

I run docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Describe the results you received: These are some of the errors I get:

 Unable to find image 'multiarch/qemu-user-static:latest' locally
 latest: Pulling from multiarch/qemu-user-static
 d9cbbca60e5f: Pulling fs layer
 c5351f770467: Pulling fs layer
 ed8a5179ae11: Pulling fs layer
 1ec39da9c97d: Pulling fs layer
 6cbaefe2d3dc: Pulling fs layer
 1ec39da9c97d: Waiting
 6cbaefe2d3dc: Waiting
 c5351f770467: Verifying Checksum
 c5351f770467: Download complete
 ed8a5179ae11: Verifying Checksum
 ed8a5179ae11: Download complete
 d9cbbca60e5f: Verifying Checksum
 d9cbbca60e5f: Download complete
 1ec39da9c97d: Verifying Checksum
 1ec39da9c97d: Download complete
 6cbaefe2d3dc: Download complete
 d9cbbca60e5f: Pull complete
 c5351f770467: Pull complete
 ed8a5179ae11: Pull complete
 1ec39da9c97d: Pull complete
 6cbaefe2d3dc: Pull complete
 Digest: sha256:96c2efd64db20bc326788773f6fa7cecef71b9d2c3de4c7d95a9910f38295c42
 Status: Downloaded newer image for multiarch/qemu-user-static:latest
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-or1k: Permission denied
 find: /proc/sys/fs/binfmt_misc/qemu-microblazeel: No such file or directory
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-microblaze: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-xtensaeb: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-xtensa: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-riscv64: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-riscv32: Permission denied
 find: /proc/sys/fs/binfmt_misc/qemu-hppa: No such file or directory
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-ppc: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-sparc64: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-sparc32plus: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-sparc: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-armeb: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-arm: Permission denied
 sh: can't create /proc/sys/fs/binfmt_misc/qemu-alpha: Permission denied
 sh: write error: File exists
 Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
 Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
 sh: write error: File exists
 sh: write error: File exists
 Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
 sh: write error: File exists
 Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
 Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
 sh: write error: File exists
 Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
 sh: write error: File exists
 Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
 sh: write error: File exists
 sh: write error: File exists
 Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
 Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
 sh: write error: File exists
 Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
 sh: write error: File exists
 sh: write error: File exists
 Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
 sh: write error: File exists
 Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
 sh: write error: File exists
 Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
 Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
 sh: write error: File exists
 Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
 sh: write error: File exists
 Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
 sh: write error: File exists
 Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
 sh: write error: File exists
 Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
 sh: write error: File exists
 Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
 sh: write error: File exists
 Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
 sh: write error: File exists
 sh: write error: File exists
 Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
 Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
 sh: write error: File exists
 Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
 sh: write error: File exists
 Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
 sh: write error: File exists
 Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
 sh: write error: File exists
 Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
 sh: write error: File exists
 Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
 sh: write error: File exists
 sh: write error: File exists
 Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
 Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
 sh: write error: File exists

But sometimes I get just the sh: write error: File exists error or no error at all. Though, it seems like the binaries I compile are still working, regardless of the error.

Describe the results you expected: I should be getting these errors?

Environment:

QEMU version: (if you can know it):
Container application: Docker

Output of docker version, podman version or singularity version

Note that this is called from the docker container running docker, not from the host as the host is a kubernetes pod that get's destroyed once the CI job is done.

$ docker info
 WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
          Access to the remote API is equivalent to root access on the host. Refer
          to the 'Docker daemon attack surface' section in the documentation for
          more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
 WARNING: No swap limit support
 Client:
  Debug Mode: false
  Plugins:
   buildx: Build with BuildKit (Docker Inc., v0.4.1)
 Server:
  Containers: 0
   Running: 0
   Paused: 0
   Stopped: 0
  Images: 0
  Server Version: 19.03.12
  Storage Driver: overlay2
   Backing Filesystem: extfs
   Supports d_type: true
   Native Overlay Diff: true
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Plugins:
   Volume: local
   Network: bridge host ipvlan macvlan null overlay
   Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
  Swarm: inactive
  Runtimes: runc
  Default Runtime: runc
  Init Binary: docker-init
  containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
  runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
  init version: fec3683
  Security Options:
   apparmor
   seccomp
    Profile: default
  Kernel Version: 5.0.0-1032-gke
  Operating System: Alpine Linux v3.12 (containerized)
  OSType: linux
  Architecture: x86_64
  CPUs: 32
  Total Memory: 31.41GiB
  Name: runner-ahdr3bm-project-14-concurrent-1lwzhj
  ID: 6F7A:DPKC:ORFD:34K7:MDPF:7Y43:7UM4:EEXU:LFG3:OM2C:FUQX:34AF
  Docker Root Dir: /var/lib/docker
  Debug Mode: false
  Username: rolandgroza
  Registry: https://index.docker.io/v1/
  Labels:
  Experimental: false
  Insecure Registries:
   127.0.0.0/8
  Live Restore Enabled: false
  Product License: Community Engine

Additional information optionally:

This setup is running on a GCP Kubernetes Engine cluster that's using the 1.15.12-gke.2 track. And it seems like it only occurs when multiple jobs run concurrently on the same node.

ghost commented 1 year ago

I experience this same error when running on a self-hosted Gitlab Runner, Docker Executor, on an Azure VM. I do NOT experience this issue when running on Gitlab's Shared Runners, probably because I'm getting spread across different runners. When running on my self hosted runner, I can have several concurrent jobs fire that will execute the qemu container and depending on the timing I will get this error across 1 or more of them.

ntrlmt commented 1 year ago

I also get the same error on a self-hosted GitLab Runner. I am dealing with it by retrying the docker run command until success.

QEMU_COMMAND="docker run --rm --privileged multiarch/qemu-user-static --reset -p yes"
until ${QEMU_COMMAND};do sleep 1;done

multiarch / qemu-user-static

Running fails with "sh: write error: File exists" or "sh: can't create ... : Permission denied" #118