Docker build fails occasionally reporting "ERROR: failed to copy to tar: EOF"

[qishipengqsp]

Contributing guidelines

• [X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• [X] The documentation does not mention anything about my problem
• [X] There are no open or closed issues that are related to my problem

Description

I use docker/build-push-action@v3 in my project to run unit tests. The workflow reports "ERROR: failed to copy to tar: EOF" occasionally after building the docker image successfully.

E.g. Two identical job in this PR: https://github.com/TuGraph-family/tugraph-db/pull/287

• Succeeded: https://github.com/TuGraph-family/tugraph-db/actions/runs/6154380071/job/16699670047?pr=287
• Failed: https://github.com/TuGraph-family/tugraph-db/actions/runs/6154380278/job/16699670537?pr=287

Expected behaviour

Both of the identical jobs succeed.

Actual behaviour

One of the job failed occasionally. And I reproduced it twice.

Repository URL

https://github.com/TuGraph-family/tugraph-db

Workflow run URL

https://github.com/TuGraph-family/tugraph-db/actions/runs/6154380278/job/16699670537?pr=287

YAML workflow

name: ci

on:
  push:
  pull_request:
    branches:
      - '*'

env:
  TEST_TAG: tugraph/tugraph-ut:test

# Issues TODO:
# 1. Seems GitHub limits the output passed among jobs up to 1MB,
#    so we can't break down the entire workflow to multi-jobs.
# 2. For submodules in the repo inside the single job, the git binary in docker image
#    should be upgrade to at least 2.18 in order to use actions/checkout@v3
#    inside docker instead of running ut and it via Dockerfile.
jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          submodules: 'true'
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      - name: Unit Test
        uses: docker/build-push-action@v4
        with:
          file: ci/gh-actions/ubuntu-latest/Dockerfile
          context: .
          load: true
          tags: ${{ env.TEST_TAG }}
          secrets: |
            "CODECOV_TOKEN=${{ secrets.CODECOV_TOKEN }}"

Workflow logs

#11 exporting to docker image format
#11 exporting layers
#11 exporting layers 188.9s done
#11 exporting manifest sha256:d044f3514ef984b08741ba457467d92b7f7bac1642722c9ca7eebd6eb796498d done
#11 exporting config sha256:b1fa63814aabd5811b056a6593babd38931f5f4a3250c0a2b087dd96c5680e2c done
#11 sending tarball
#11 sending tarball 80.3s done
#11 ERROR: failed to copy to tar: EOF

BuildKit logs

No response

Additional info

No response

[qishipengqsp]

And I tried to upgrade it to v4. Got failed again.

[crazy-max]

Looking at the error I would say you don't have enough disk space available on the runner. Can you add this step after "Unit Test" to check that please:

      - name: Check disk space
        if: always()
        run: df -h

Also looking at your build config I don't think you need the "Set up Docker Buildx" step to create container builder. You can either remove this step or select the docker driver as you just want to load the image to the store anyway:

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          driver: docker

[crazy-max]

you just want to load the image to the store anyway:

Actually I'm not sure why you want to load the image to the store if this just run tests right? You can just do:

      - name: Unit Test
        uses: docker/build-push-action@v4
        with:
          file: ci/gh-actions/ubuntu-latest/Dockerfile
          context: .
          tags: ${{ env.TEST_TAG }}
          secrets: |
            "CODECOV_TOKEN=${{ secrets.CODECOV_TOKEN }}"

[qishipengqsp]

Hi, thanks for the suggestion! I didn't notice the load parameter, just want to run the tests. I removed the option and see if it helps with the limited space.

[crazy-max]

Oh btw if you can run your build with

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          buildkitd-flags: --debug

that would give us more logs to understand what's going on, thanks!

[qishipengqsp]

Sure. Added. Let's see what happens. https://github.com/TuGraph-family/tugraph-db/pull/287

[qishipengqsp]

It works. It did not run out of space this time. In that case, do we need to keep the setup-buildx-action, which is for debugging?

[crazy-max]

You can keep it if you want but I think you can just remove it imo.

[qishipengqsp]

Thanks!