search

docker / build-push-action

GitHub Action to build and push Docker images with Buildx
https://github.com/marketplace/actions/build-and-push-docker-images
Apache License 2.0
4.28k stars 550 forks source link

Build should not fail on corrupted cache #577

Closed Meemaw closed 1 year ago

Meemaw commented 2 years ago

Expected behaviour

Building with build-push-action and buildx works as expected.

Actual behaviour

Yesterday we started seeing our docker build push action fail on every commit. This is not correlated to any specific code change that we did.

We temporary fixed this by removing the buildx from our workflow.

Configuration

name: Build & Push Docker Next Image

on: push

jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-next-prod-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-next-prod
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ secrets.GH_BOT_USER }}
          password: ${{ secrets.GH_PAT_TOKEN }}

      - name: Inject slug/short variables
        uses: rlespinasse/github-slug-action@v4

      - name: Build and push
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          tags: |
            ghcr.io/companyx/next:${{ env.GITHUB_REF_SLUG }}
            ghcr.io/companyx/next:${{ github.sha }}
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new

      - name: Image digest
        run: echo ${{ steps.docker_build.outputs.digest }}

        # Temporary fix
        # https://github.com/docker/build-push-action/issues/252
        # https://github.com/moby/buildkit/issues/1896
      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache
      - name: Trigger release workflow
        if: github.ref == 'refs/heads/master'
        uses: benc-uk/workflow-dispatch@v1
        with:
          workflow: release
          inputs: '{ "commit_sha": "${{ github.sha }}" }'
          token: ${{ secrets.GH_PAT_TOKEN }}

Logs

#9 [builder 1/9] COPY package.json yarn.lock ./
#9 ERROR: failed commit on ref "layer-sha256:0b0af05a4d868593f859eaa5815fc1c3596d77318a4ed756f3865a5fa3f290c6": unexpected commit digest sha256:7097d9f739d4ed17f8875508553264eb5e496d5e07edd35330157cae6d219a2c, expected sha256:0b0af05a4d868593f859eaa5815fc1c3596d77318a4ed756f3865a5fa3f290c6: failed precondition
------
 > [builder 1/9] COPY package.json yarn.lock ./:
------
Dockerfile:9
--------------------
   7 |     FROM base AS builder
   8 |     
   9 | >>> COPY package.json yarn.lock ./
  10 |     
  11 |     # Install production dependencies
--------------------
error: failed to solve: failed to compute cache key: failed commit on ref "layer-sha256:0b0af05a4d868593f859eaa5815fc1c3596d77318a4ed756f3865a5fa3f290c6": unexpected commit digest sha256:7097d9f739d4ed17f8875508553264eb5e496d5e07edd35330157cae6d219a2c, expected sha256:0b0af05a4d868593f859eaa5815fc1c3596d77318a4ed756f3865a5fa3f290c6: failed precondition
Error: buildx failed with: error: failed to solve: failed to compute cache key: failed commit on ref "layer-sha256:0b0af05a4d868593f859eaa5815fc1c3596d77318a4ed756f3865a5fa3f290c6": unexpected commit digest sha256:7097d9f739d4ed17f8875508553264eb5e496d5e07edd35330157cae6d219a2c, expected sha256:0b0af05a4d868593f859eaa5815fc1c3596d77318a4ed756f3865a5fa3f290c6: failed precondition
crazy-max commented 2 years ago

@Meemaw Can you enable debug in the Set up Docker Buildx step:

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          buildkitd-flags: --debug

And post BuildKit container logs available in the Post Set Up Docker Buildx please?

If you have a link to your repo or a simple repro so we can check on our side, that would be great, thanks!

Meemaw commented 2 years ago

Done. Here are the logs:

  /usr/bin/docker logs buildx_buildkit_builder-051c83e6-10bc-4d70-97f2-04c81188f3ae0
  time="2022-03-24T08:20:55Z" level=info msg="auto snapshotter: using overlayfs"
  time="2022-03-24T08:20:55Z" level=warning msg="using host network as the default"
  time="2022-03-24T08:20:55Z" level=info msg="found worker \"topm57vu4yerinbavmjfyb41b\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:221586c425c5 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/arm/v7 linux/arm/v6]"
  time="2022-03-24T08:20:55Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
  time="2022-03-24T08:20:55Z" level=info msg="found 1 workers, default=\"topm57vu4yerinbavmjfyb41b\""
  time="2022-03-24T08:20:55Z" level=warning msg="currently, only the default worker can be used."
  time="2022-03-24T08:20:55Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
  time="2022-03-24T08:21:01Z" level=debug msg="session started"
  time="2022-03-24T08:21:01Z" level=debug msg="session finished: <nil>"
  time="2022-03-24T08:21:01Z" level=debug msg="session started"
  time="2022-03-24T08:21:01Z" level=debug msg="skipping invalid cache export mode: "
  time="2022-03-24T08:21:01Z" level=debug msg="new ref for local: n4zjd8rul06ng8v7zydkpx87x" span="[internal] load build definition from Dockerfile"
  time="2022-03-24T08:21:01Z" level=debug msg="diffcopy took: 30.521205ms" span="[internal] load build definition from Dockerfile"
  time="2022-03-24T08:21:01Z" level=debug msg="saved n4zjd8rul06ng8v7zydkpx87x as dockerfile:dockerfile:companyx-next:35b65b22e5785078" span="[internal] load build definition from Dockerfile"
  time="2022-03-24T08:21:01Z" level=debug msg="new ref for local: xyjw3ta0nznzfxcfnmcjhwrbh" span="[internal] load .dockerignore"
  time="2022-03-24T08:21:01Z" level=debug msg="diffcopy took: 17.165891ms" span="[internal] load .dockerignore"
  time="2022-03-24T08:21:01Z" level=debug msg="saved xyjw3ta0nznzfxcfnmcjhwrbh as context:context-.dockerignore:companyx-next:35b65b22e5785078" span="[internal] load .dockerignore"
  time="2022-03-24T08:21:02Z" level=debug msg=resolving host=registry-1.docker.io
  time="2022-03-24T08:21:02Z" level=debug msg="do request" host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=buildkit/0.0.0+unknown request.method=HEAD url="https://registry-1.docker.io/v2/library/node/manifests/16.11.0-buster"
  time="2022-03-24T08:21:02Z" level=debug msg="fetch response received" host=registry-1.docker.io response.header.content-length=155 response.header.content-type=application/json response.header.date="Thu, 24 Mar 2022 08:21:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=52.191.252.221 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="***"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/node:pull\"" response.status="401 Unauthorized" url="https://registry-1.docker.io/v2/library/node/manifests/16.11.0-buster"
  time="2022-03-24T08:21:02Z" level=debug msg=Unauthorized header="***"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/node:pull\"" host=registry-1.docker.io
  time="2022-03-24T08:21:02Z" level=debug msg="do request" host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=buildkit/0.0.0+unknown request.method=HEAD url="https://registry-1.docker.io/v2/library/node/manifests/16.11.0-buster"
  time="2022-03-24T08:21:02Z" level=debug msg="fetch response received" host=registry-1.docker.io response.header.content-length=1206 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Thu, 24 Mar 2022 08:21:02 GMT" response.header.docker-content-digest="sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=52.191.252.221 response.header.etag="\"sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa\"" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" url="https://registry-1.docker.io/v2/library/node/manifests/16.11.0-buster"
  time="2022-03-24T08:21:02Z" level=debug msg=resolved desc.digest="sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa" host=registry-1.docker.io
  time="2022-03-24T08:21:02Z" level=debug msg=fetch digest="sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1206
  time="2022-03-24T08:21:02Z" level=debug msg="do request" digest="sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=buildkit/0.0.0+unknown request.method=GET size=1206 url="https://registry-1.docker.io/v2/library/node/manifests/sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa"
  time="2022-03-24T08:21:02Z" level=debug msg="fetch response received" digest="sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.content-length=1206 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Thu, 24 Mar 2022 08:21:02 GMT" response.header.docker-content-digest="sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=52.191.252.221 response.header.etag="\"sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa\"" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=1206 url="https://registry-1.docker.io/v2/library/node/manifests/sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa"
  time="2022-03-24T08:21:02Z" level=debug msg=fetch digest="sha256:ae3a1ce333f322728ef3ea12186598067c31248a14a33aaa148bac7c6efc3a14" mediatype=application/vnd.docker.distribution.manifest.v2+json size=2214
  time="2022-03-24T08:21:02Z" level=debug msg="do request" digest="sha256:ae3a1ce333f322728ef3ea12186598067c31248a14a33aaa148bac7c6efc3a14" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=buildkit/0.0.0+unknown request.method=GET size=2214 url="https://registry-1.docker.io/v2/library/node/manifests/sha256:ae3a1ce333f322728ef3ea12186598067c31248a14a33aaa148bac7c6efc3a14"
  time="2022-03-24T08:21:02Z" level=debug msg="fetch response received" digest="sha256:ae3a1ce333f322728ef3ea12186598067c31248a14a33aaa148bac7c6efc3a14" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=2214 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Thu, 24 Mar 2022 08:21:02 GMT" response.header.docker-content-digest="sha256:ae3a1ce333f322728ef3ea12186598067c31248a14a33aaa148bac7c6efc3a14" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=52.191.252.221 response.header.etag="\"sha256:ae3a1ce333f322728ef3ea12186598067c31248a14a33aaa148bac7c6efc3a14\"" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=2214 url="https://registry-1.docker.io/v2/library/node/manifests/sha256:ae3a1ce333f322728ef3ea12186598067c31248a14a33aaa148bac7c6efc3a14"
  time="2022-03-24T08:21:02Z" level=debug msg=fetch digest="sha256:a22d52bf85f0130cb32ba91725b03249e94c8ad99c716a5d04858f7b001b2fd1" mediatype=application/vnd.docker.container.image.v1+json size=7605
  time="2022-03-24T08:21:02Z" level=debug msg="do request" digest="sha256:a22d52bf85f0130cb32ba91725b03249e94c8ad99c716a5d04858f7b001b2fd1" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=buildkit/0.0.0+unknown request.method=GET size=7605 url="https://registry-1.docker.io/v2/library/node/blobs/sha256:a22d52bf85f0130cb32ba91725b03249e94c8ad99c716a5d04858f7b001b2fd1"
  time="2022-03-24T08:21:02Z" level=debug msg="fetch response received" digest="sha256:a22d52bf85f0130cb32ba91725b03249e94c8ad99c716a5d04858f7b001b2fd1" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.age=1457457 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=6f0dfdb1faab81af-IAD response.header.connection=keep-alive response.header.content-length=7605 response.header.content-type=application/octet-stream response.header.date="Thu, 24 Mar 2022 08:21:02 GMT" response.header.etag="\"c74f91a101755bea448a3bf674988dbf\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Thu, 24 Mar 2022 12:21:02 GMT" response.header.last-modified="Mon, 11 Oct 2021 18:34:54 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="ftbjuVU852ISTQL1hGPC+zKUD4UFg5f1Nf81we5P6M8BxS0CVzOZls+AhhPFSxLmTCz6l1LCDRQ=" response.header.x-amz-request-id=MBYQ8ZG4F076N99N response.header.x-amz-version-id=e7LaWluPegkjLrUVsB3gaMRicH.sD28m response.status="200 OK" size=7605 url="https://registry-1.docker.io/v2/library/node/blobs/sha256:a22d52bf85f0130cb32ba91725b03249e94c8ad99c716a5d04858f7b001b2fd1"
  time="2022-03-24T08:21:02Z" level=debug msg=fetch digest="sha256:aa1e997e30f6616f0517f6968982e3cd73bff84529d852cbdf0521592bc582aa" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1206
  time="2022-03-24T08:21:02Z" level=debug msg=fetch digest="sha256:ae3a1ce333f322728ef3ea12186598067c31248a14a33aaa148bac7c6efc3a14" mediatype=application/vnd.docker.distribution.manifest.v2+json size=2214
  time="2022-03-24T08:21:02Z" level=debug msg=fetch digest="sha256:a22d52bf85f0130cb32ba91725b03249e94c8ad99c716a5d04858f7b001b2fd1" mediatype=application/vnd.docker.container.image.v1+json size=7605
  time="2022-03-24T08:21:02Z" level=debug msg="new ref for local: mnco4lvim8gcx8k3nrzgbigzc" span="[internal] load build context"
  time="2022-03-24T08:21:03Z" level=debug msg="diffcopy took: 1.055925172s" span="[internal] load build context"
  time="2022-03-24T08:21:03Z" level=debug msg="saved mnco4lvim8gcx8k3nrzgbigzc as context:context:companyx-next:35b65b22e5785078" span="[internal] load build context"
  time="2022-03-24T08:21:03Z" level=debug msg="load cache for [base 2/2] WORKDIR /usr/app with sha256:7955be70ba2c6cc246ea7ef9fabe8a99fb4e3e218b2a8b257c5fc343846eba3a"
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:81b3aa9536df1989c63621d43c6881bbb4f6bd83eee22a56441683ed795ba82b" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=116
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:2d5feb05857fef73be65a2c0228bbf66fdc86b6ec8733245398960da4b0ad39e" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=283
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:4b19957ec80a370ab74f7397cc90d8952a150257942ecbecd57c29c09b6739f0" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=2268011
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:ca7417c24e969235aaade789b8c85e208e0a7baa5752395d0c987781bd9e1bf5" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=33350248
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:8c33de21d690878efd5d2c7872e0f6afd1316b6173a288bb2e25b6c962c36cfc" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=4197
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:ca4689f0588c1ba01108f5c6d33943d3d542ae339fd526390d07c7a00ded6473" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=192395236
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:0b0af05a4d868593f859eaa5815fc1c3596d77318a4ed756f3865a5fa3f290c6" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=51841311
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:1ae016bc26876abbd5e952133b02b04d4c1dae1bc75a3d9386250e4797ccd87a" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=9997190
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:5e7b6b7bd506c12399d65977c0ba8dd02824dc5d0e65fc55d7382da889bdac7d" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=50436209
  time="2022-03-24T08:21:03Z" level=debug msg=fetch digest="sha256:fd67d668d6911bf21ad4701522e1ed3af416837433fdba3f88cff06a23e23861" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=7833602
  time="2022-03-24T08:21:08Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to compute cache key: failed commit on ref \"layer-sha256:0b0af05a4d868593f859eaa5815fc1c3596d77318a4ed756f3865a5fa3f290c6\": unexpected commit digest sha256:7097d9f739d4ed17f8875508553264eb5e496d5e07edd35330157cae6d219a2c, expected sha256:0b0af05a4d868593f859eaa5815fc1c3596d77318a4ed756f3865a5fa3f290c6: failed precondition\n"
  time="2022-03-24T08:21:08Z" level=debug msg="session finished: <nil>"

Unfortunately I can't link the repo, as its private.

crazy-max commented 2 years ago

Thanks. Can you give the output of Docker info and BuildKit version in the Set up Docker Buildx step please?

Meemaw commented 2 years ago 
Docker info
/usr/bin/docker version
Client:
 Version:           20.10.11+azure-3
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        dea9396e1[8](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:8)42[9](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:9)0f638ea873c76db7c80efd5a1d2
 Built:             Wed Nov 17 23:49:46 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.[10](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:10).[11](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:11)+azure-3
  API version:      1.41 (minimum version 1.[12](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:12))
  Go version:       go1.16.12
  Git commit:       847da184ad5048b27f5bdf9d53d070f731b43180
  Built:            Thu Nov 18 00:21:59 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12+azure-2
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.3
  GitCommit:        f46b6ba2c93[14](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:14)cfc8caae24a32ec5fe9ef1059fe
 docker-init:
  Version:          0.19.0
  GitCommit:        
/usr/bin/docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., 0.7.1+azure-2)
  compose: Docker Compose (Docker Inc., 2.2.3+azure-1)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 20
 Server Version: 20.10.11+azure-3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23[18](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:18)2b9347b4245eb5d
 runc version: f46b6ba2c9314cfc8caae24a32ec5fe9ef1059fe
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.0-1028-azure
 Operating System: Ubuntu [20](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:20).04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 6.785GiB
 Name: fv-az1[22](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:22)-94
 ID: DFUV:3MIO:QJB4:SFAI:CWPP:LEBI:ETH4:AT3U:T5PZ:KRMI:AS3I:HDYW
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: githubactions
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  1[27](https://github.com/companyx/companyx-next/runs/5672982129?check_suite_focus=true#step:4:27).0.0.0/8
 Live Restore Enabled: false

and

BuildKit version
moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.10.0 068cf686a7e5c3254244d0acc54636f4f393611e
tonistiigi commented 2 years ago

@ktock @sipsma Any way you see the blob management or lazy loading changes could cause something like this? Looks like error on blob commit. Possible cases could be that the verification digest is for wrong blob or there are parallel writes happening to the same location, messing up the digest. Assuming whatever base is in that Dockerfile is not an invalid image or registry did not provide invalid layers.

@Meemaw If you think this could be v0.10.0 regression you can set a different version on setup-buildx-action. If you can reproduce this then please provide us a way we could do it as well.

Meemaw commented 2 years ago

@ktock @sipsma Any way you see the blob management or lazy loading changes could cause something like this? Looks like error on blob commit. Possible cases could be that the verification digest is for wrong blob or there are parallel writes happening to the same location, messing up the digest. Assuming whatever base is in that Dockerfile is not an invalid image or registry did not provide invalid layers.

@Meemaw If you think this could be v0.10.0 regression you can set a different version on setup-buildx-action. If you can reproduce this then please provide us a way we could do it as well.

Can you give me an example on how to change the buildkit version in setup-buildx-action? I can check if its related to v0.10.0.

Meemaw commented 2 years ago

Also seeing this error in the Cache Docker layers step in case relevant:

Run actions/cache@v3
Received 1845493[7](https://github.com/ProjectOpenSea/companyx-next/runs/5691537979?check_suite_focus=true#step:5:7)6 of 570426441 (32.4%), 175.5 MBs/sec
Received 461373440 of 570426441 ([8](https://github.com/ProjectOpenSea/companyx-next/runs/5691537979?check_suite_focus=true#step:5:8)0.[9](https://github.com/ProjectOpenSea/companyx-next/runs/5691537979?check_suite_focus=true#step:5:9)%), 219.6 MBs/sec
Received 570426441 of 570426441 ([10](https://github.com/ProjectOpenSea/companyx-next/runs/5691537979?check_suite_focus=true#step:5:10)0.0%), 193.4 MBs/sec
Cache Size: ~544 MB (570426441 B)
/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/b0ef45b7-d47c-4858-8359-cd9ba6c787aa/cache.tzst -P -C /home/runner/work/companyx-next/companyx-next
/*stdin*\ : Decoding error (36) : Restored data doesn't match checksum 
/usr/bin/tar: Child returned status 1
/usr/bin/tar: Error is not recoverable: exiting now
Warning: Tar failed with error: The process '/usr/bin/tar' failed with exit code 2
crazy-max commented 2 years ago

Also seeing this error in the Cache Docker layers step in case relevant:

Interesting so cache seems corrupted in GitHub.

Can you give me an example on how to change the buildkit version in setup-buildx-action? I can check if its related to v0.10.0.

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          version: v0.9.3
Meemaw commented 2 years ago

Also seeing this error in the Cache Docker layers step in case relevant:

Interesting so cache seems corrupted in GitHub.

Can you give me an example on how to change the buildkit version in setup-buildx-action? I can check if its related to v0.10.0.

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          version: v0.9.3

Download and install buildx Error: Cannot find buildx v0.9.3 release

crazy-max commented 2 years ago

@Meemaw

Error: Cannot find buildx v0.9.3 release

My bad:

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          driver-opts: |
            image=moby/buildkit:v0.9.3
Meemaw commented 2 years ago

Fails on that version as well -- same error.

crazy-max commented 2 years ago

@Meemaw With the exact same step in https://github.com/docker/build-push-action/issues/577#issuecomment-1079030579?

Meemaw commented 2 years ago

@Meemaw With the exact same step in #577 (comment)?

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          driver-opts: |
            image=moby/buildkit:v0.9.3
          buildkitd-flags: --debug
crazy-max commented 2 years ago

@Meemaw Works fine for me with this step: https://github.com/crazy-max/buildx-buildkit-tests/blob/43e41154a597cdf1548852266db3daa179901510/.github/workflows/build-push-action-577.yml#L16-L22

Meemaw commented 2 years ago

@crazy-max the setup-buildx step works, but the build/push step fails with the original error.

crazy-max commented 2 years ago

@Meemaw

Fails on that version as well -- same error.

Ok, I thought the setup-buildx step had failed.

So looking at https://github.com/docker/build-push-action/issues/577#issuecomment-1078973976, your GitHub cache looks busted. Looks like something wrong occurs with actions/cache. To be sure can you change your cache key with smth like:

      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-next-prod-v2-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-next-prod-v2
Meemaw commented 2 years ago

@crazy-max changing the cache key fixed the issue. Wondering why that would break the next steps though? I would expect cache action to ignore the cache if corrupted.

crazy-max commented 2 years ago

Yes I think we should just skip and not fail in this case. Will try to repro on my side.

thewilkybarkid commented 1 year ago

https://github.com/moby/buildkit/pull/3430 introduced an ignore-error parameter, which seems to work (https://github.com/moby/buildkit/issues/2836#issuecomment-1543694979):

cache-from: type=gha,ignore-error=true
cache-to: type=gha,mode=max,ignore-error=true
crazy-max commented 1 year ago

@thewilkybarkid Yes indeed, we have added docs for this too: https://docs.docker.com/build/cache/backends/gha/#synopsis

Also ignore-error opt is just handled for cache-to.

crazy-max commented 1 year ago

Closing this one as this is tracked upstream: https://github.com/moby/buildkit/issues/2836