buildkit-runc did not terminate successfully

[siennathesane]

With a generic Angular application, Given this task definition:

platform: linux

image_resource:
  type: registry-image
  source:
    repository: vito/oci-build-task

inputs:
  - name: repo

outputs:
  - name: image

params:
  DOCKERFILE: repo/src/subproject/deployments/docker/Dockerfile
  CONTEXT: repo/src/subproject

run:
  path: build

With this Dockerfile:

FROM node:12 as builder

WORKDIR /home/node/app
COPY . .

ARG VAR_URL=${VAR_URL}
ENV API_URL=${VAR_URL}

RUN npm install --loglevel=warn
RUN npm install -g --save-dev @types/node
RUN npm install -g @angular/cli
RUN $(npm bin)/ng build -

# Stage 2
FROM nginx:1.17.1-alpine

COPY nginx.conf /etc/nginx/nginx.conf
COPY --from=builder /home/node/app/dist/subproject/ /usr/share/nginx/html
EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

And this is a generic pipeline definition:

jobs:
  - name: prepare-container-image
    plan:
      - get: repo
        trigger: true
      - task: build-image
        privileged: true
        file: repo/src/subproject/ci/tasks/build-repo-image.yml

This is the error I receive:

#10 [builder 3/7] COPY . .
#10 DONE 0.3s

#11 [builder 4/7] RUN npm install --loglevel=warn
#11 0.247 container_linux.go:367: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: mkdir /sys/fs/cgroup/cpuset/buildkit: read-only file system
#11 ERROR: executor failed running [/bin/sh -c npm install --loglevel=warn]: buildkit-runc did not terminate successfully
------
> [builder 4/7] RUN npm install --loglevel=warn:
------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c npm install --loglevel=warn]: buildkit-runc did not terminate successfully
FATA[0027] failed to build: build: exit status 1
FATA[0027] failed to run task: exit status 1

Additional Info

• Concourse v6.6.0
• Manual worker is using runc, Helm-deployed worker is using the default.
• Builds a different multi-stage image without issues.
• The workers are not out of disk space.
• This error happens in Helm-deployed Concourse workers and manually deployed Concourse workers.
• No other steps fail.
• The docker image build locally.

[holgerstolzenberg]

Can confirm this behaviour, stumbled upon this 10m ago having exactly the same scenario.

[holgerstolzenberg]

Additional info: For us, this problem only occurs when running with containerd runtime.

[mnitchev]

We also noticed this behaviour using both helm and bosh deployed concourse. As far as I can tell they're both running with the garden runtime. Oddly enough when we try to fly execute a one off task it does succeed. This is the yaml we're using for it and this is the Dockerfile

platform: linux
image_resource:
  type: registry-image
  source:
    repository: vito/oci-build-task
inputs:
- name: eirini
  path: .
- name: docker-build-args
outputs:
- name: image
run:
  path: build
params:
  BUILD_ARGS_FILE: docker-build-args/args
  DOCKERFILE: docker/opi/Dockerfile

When run in a pipeline however a job with the same task fails with the above error.

[siennathesane]

I've also noticed it can succeed as a one-off, but fails as part of the pipeline, which was very strange.

[drahnr]

Still present with v6.7.2 on native linux64 workers using the internal containerd backend.