search

docker / build-push-action

GitHub Action to build and push Docker images with Buildx
https://github.com/marketplace/actions/build-and-push-docker-images
Apache License 2.0
4.11k stars 526 forks source link

Action hangs after build #1064

Closed jfoclpf closed 4 months ago

jfoclpf commented 4 months ago

Contributing guidelines

I've found a bug, and:

Description

Action hangs after build

Expected behaviour

It should complete

Actual behaviour

It hangs, it does not do anything and then the action is considered as errored

Repository URL

https://github.com/geoapi-pt

Workflow run URL

https://github.com/geoapi-pt/main/blob/main/.github/workflows/docker.yml

YAML workflow

name: Docker build and push

on:
  push:
    branches:
      - "main"

jobs:
  build:
    runs-on: ubuntu-20.04
    steps:
      - 
        name: Check disk space
        run: df . -h
      - 
        name: Free disk space
        run: |
          sudo docker rmi $(docker image ls -aq) >/dev/null 2>&1 || true
          sudo rm -rf \
            /usr/share/dotnet /usr/local/lib/android /opt/ghc \
            /usr/local/share/powershell /usr/share/swift /usr/local/.ghcup \
            /usr/lib/jvm || true
          echo "some directories deleted"
          sudo apt install aptitude -y >/dev/null 2>&1
          sudo aptitude purge aria2 ansible azure-cli shellcheck rpm xorriso zsync \
            esl-erlang firefox gfortran-8 gfortran-9 google-chrome-stable \
            google-cloud-sdk imagemagick \
            libmagickcore-dev libmagickwand-dev libmagic-dev ant ant-optional kubectl \
            mercurial apt-transport-https mono-complete libmysqlclient \
            unixodbc-dev yarn chrpath libssl-dev libxft-dev \
            libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev \
            snmp pollinate libpq-dev postgresql-client powershell ruby-full \
            sphinxsearch subversion mongodb-org azure-cli microsoft-edge-stable \
            -y -f >/dev/null 2>&1
          sudo aptitude purge google-cloud-sdk -f -y >/dev/null 2>&1
          sudo aptitude purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
          sudo apt purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
          sudo aptitude purge '~n ^mysql' -f -y >/dev/null 2>&1
          sudo aptitude purge '~n ^php' -f -y >/dev/null 2>&1
          sudo aptitude purge '~n ^dotnet' -f -y >/dev/null 2>&1
          sudo apt-get autoremove -y >/dev/null 2>&1
          sudo apt-get autoclean -y >/dev/null 2>&1
          echo "some packages purged"
      - 
        name: Check disk space
        run: |
          sudo dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -nr | head
          df . -h
          sudo du /usr/ -hx -d 4 --threshold=1G | sort -hr | head
      - 
        name: Checkout Node
        uses: actions/setup-node@v4
        with:
          node-version: 16
      - 
        name: Checkout main root repo
        uses: actions/checkout@v4
        with:
          repository: geoapi-pt/root
          token: ${{ secrets.GH_PAT }}
      -
        name: Main Checkout
        uses: actions/checkout@v4
        with:
          path: main
      - 
        name: Checkout resources private repo
        uses: actions/checkout@v4
        with:
          repository: geoapi-pt/resources
          token: ${{ secrets.GH_PAT }}
          path: resources
      - 
        name: Check working space directory
        run: du ${GITHUB_WORKSPACE} -h -d 1
      - 
        name: Get more space
        run: |
          df . -h
          sudo rm -rf ${GITHUB_WORKSPACE}/main/.git
          sudo rm -rf ${GITHUB_WORKSPACE}/resources/.git
          df . -h
      -  
        name: List project root direcotory
        run: ls ${GITHUB_WORKSPACE} -lh
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      -
        name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ${{ secrets.DOCKER_USERNAME }}/geoapi.pt:latest

Workflow logs

Run docker/build-push-action@v5
GitHub Actions runtime token ACs
Docker info
Proxy configuration
Buildx version
/usr/bin/docker buildx build --file ./Dockerfile --iidfile /home/runner/work/_temp/docker-actions-toolkit-ldu8Dw/iidfile --provenance mode=max,builder-id=https://github.com/geoapi-pt/main/actions/runs/8078724413 --tag ***/geoapi.pt:latest --metadata-file /home/runner/work/_temp/docker-actions-toolkit-ldu8Dw/metadata-file --push .
#0 building with "builder-613e5f59-037b-4c4b-8f5f-64dfd3fbcfb1" instance using docker-container driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 634B done
#1 DONE 0.0s
#2 [auth] library/node:pull token for registry-1.docker.io
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/node:16-slim
#3 DONE 0.3s
#4 [internal] load .dockerignore
#4 transferring context: 442B done
#4 DONE 0.0s
#5 [1/8] FROM docker.io/library/node:16-slim@sha256:3ebf2875c188d22939c6ab080cfb1a4a6248cc86bae600ea8e2326aa03acdb8f
#5 resolve docker.io/library/node:16-slim@sha256:3ebf2875c188d22939c6ab080cfb1a4a6248cc86bae600ea8e2326aa03acdb8f done
#5 sha256:a9db6797916ed95aa70dba9828c955247db171807e704ae7b1fb37b7c7e4b0e6 450B / 450B 0.0s done
#5 sha256:ecdd21ad91a4e69294555b3e56a71a7382a58a9c54b02d9038bcd3821ba3d072 4.18kB / 4.18kB 0.1s done
#5 sha256:6d65359ecb29ff24f2322d5ad308f0a8a95980706ae2e76315a267474402aa3a 2.73MB / 2.73MB 0.1s done
#5 sha256:72f5a0bbd6d5f85fce35070bd624cf69c2ceb91424afe8a5ed2a02442cdd7[11](https://github.com/geoapi-pt/main/actions/runs/8078724413/job/22071633735#step:14:12)c 10.49MB / 35.25MB 0.2s
#5 sha256:72f5a0bbd6d5f85fce35070bd624cf69c2ceb91424afe8a5ed2a02442cdd711c 26.21MB / 35.25MB 0.3s
#5 sha256:91f01557fe0da558070d4f24631c94e91a80877a24621b52b8b[13](https://github.com/geoapi-pt/main/actions/runs/8078724413/job/22071633735#step:14:15)009b62d8d96 27.19MB / 27.19MB 0.3s done
#5 sha256:72f5a0bbd6d5f85fce35070bd624cf69c2ceb91424afe8a5ed2a02442cdd711c 35.25MB / 35.25MB 0.4s done
#5 extracting sha256:91f01557fe0da558070d4f24631c94e91a80877a24621b52b8b[130](https://github.com/geoapi-pt/main/actions/runs/8078724413/job/22071633735#step:14:135)09b62d8d96
#5 extracting sha256:91f01557fe0da558070d4f24631c94e91a80877a24621b52b8b13009b62d8d96 1.8s done
#5 extracting sha256:ecdd21ad91a4e69294555b3e56a71a7382a58a9c54b02d9038bcd3821ba3d072
#5 extracting sha256:ecdd21ad91a4e69294555b3e56a71a7382a58a9c54b02d9038bcd3821ba3d072 0.0s done
#5 extracting sha256:72f5a0bbd6d5f85fce35070bd624cf69c2ceb9[142](https://github.com/geoapi-pt/main/actions/runs/8078724413/job/22071633735#step:14:147)4afe8a5ed2a02442cdd711c
#5 extracting sha256:72f5a0bbd6d5f85fce35070bd624cf69c2ceb91424afe8a5ed2a02442cdd711c 2.5s done
#5 DONE 4.7s
#6 [internal] load build context
#6 ...
#5 [1/8] FROM docker.io/library/node:16-slim@sha256:3ebf2875c188d22939c6ab080cfb1a4a6248cc86bae600ea8e2326aa03acdb8f
#5 extracting sha256:6d65359ecb29ff24f2322d5ad308f0a8a95980706ae2e76315a[267](https://github.com/geoapi-pt/main/actions/runs/8078724413/job/22071633735#step:14:272)474402aa3a 0.2s done
#5 extracting sha256:a9db6797916ed95aa70dba9828c955247db171807e704ae7b1fb37b7c7e4b0e6 0.0s done
#5 DONE 4.9s
#6 [internal] load build context
#6 transferring context: 91.79MB 5.1s
#6 ...
#7 [2/8] WORKDIR /usr/src/app
#7 DONE 0.6s
#6 [internal] load build context
#6 transferring context: 180.43MB 10.2s
#6 transferring context: 293.10MB 15.2s
#11 4.770 found 0 vulnerabilities
#11 DONE 4.9s
#12 [7/8] RUN npm install pm2 -g
#12 3.030 npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
#12 3.624 
#12 3.624 added 162 packages, and audited 163 packages in 3s
#12 3.624 
#12 3.624 13 packages are looking for funding
#12 3.624   run `npm fund` for details
#12 3.626 
#12 3.626 found 0 vulnerabilities
#12 DONE 3.7s
#13 [8/8] COPY . .

BuildKit logs

No response

Additional info

No response

crazy-max commented 4 months ago

Your context is really big, almost 14GB:

#7 [2/8] WORKDIR /usr/src/app
#7 DONE 0.6s
#6 [internal] load build context
#6 transferring context: 180.43MB 10.2s
#6 transferring context: 293.10MB 15.2s
#6 transferring context: 407.39MB 20.3s
#6 transferring context: 513.48MB 25.4s
#6 transferring context: 617.43MB 30.5s
#6 transferring context: 806.25MB 35.5s
#6 transferring context: 949.40MB 40.7s
#6 transferring context: 1.11GB 45.9s
#6 transferring context: 1.22GB 51.0s
#6 transferring context: 1.32GB 56.0s
#6 transferring context: 1.41GB 61.0s
#6 transferring context: 1.51GB 66.0s
#6 transferring context: 1.79GB 71.1s
#6 transferring context: 2.19GB 76.2s
#6 transferring context: 2.61GB 81.2s
#6 transferring context: 2.99GB 86.2s
#6 transferring context: 3.47GB 91.3s
#6 transferring context: 3.97GB 96.4s
#6 transferring context: 4.47GB 101.5s
#6 transferring context: 4.95GB 106.5s
#6 transferring context: 5.04GB 111.5s
#6 transferring context: 5.19GB 116.6s
#6 transferring context: 5.31GB 121.8s
#6 transferring context: 5.46GB 126.9s
#6 transferring context: 5.57GB 132.0s
#6 transferring context: 5.72GB 137.0s
#6 transferring context: 5.86GB 142.0s
#6 transferring context: 5.99GB 147.0s
#6 transferring context: 6.10GB 152.2s
#6 transferring context: 6.24GB 157.3s
#6 transferring context: 6.36GB 162.4s
#6 transferring context: 6.48GB 167.4s
#6 transferring context: 6.82GB 172.6s
#6 transferring context: 7.33GB 177.7s
#6 transferring context: 7.92GB 182.8s
#6 transferring context: 8.48GB 187.8s
#6 transferring context: 9.07GB 192.9s
#6 transferring context: 9.61GB 198.1s
#6 transferring context: 10.20GB 203.1s
#6 transferring context: 10.75GB 208.2s
#6 transferring context: 11.19GB 213.3s
#6 transferring context: 11.47GB 218.4s
#6 transferring context: 11.79GB 223.4s
#6 transferring context: 12.08GB 228.4s
#6 transferring context: 12.29GB 233.5s
#6 transferring context: 12.57GB 238.5s
#6 transferring context: 12.84GB 243.6s
#6 transferring context: 13.09GB 248.6s
#6 transferring context: 13.30GB 255.7s done

I guess you're running out of space on the runner.

I also don't have access to https://github.com/orgs/geoapi-pt/root which you're cloning in https://github.com/geoapi-pt/main/blob/bd95fd9240e64ad12a105c92e41eaaa5538195cd/.github/workflows/docker.yml#L55-L60 to check the Dockerfile and I suspect this repo takes more than 14GB. So cloning + uploading context + cache handling would require more than 42GB from what I see.

It "might" work if you're using the docker driver:

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver: docker

And disable cache:

      -
        name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ${{ secrets.DOCKER_USERNAME }}/geoapi.pt:latest
          no-cache: true

But there might be an anti-pattern or bad practice in your Dockerfile and workflow as this is unsual to have a context that big. If you can share https://github.com/orgs/geoapi-pt/root that would help to understand.

@tonistiigi Related to COPY command hanging, I don't recall this issue being reported already. Rings a bell?

jfoclpf commented 4 months ago

Thanks a lot for the quick reply

Yes, the repo is big, thus the top commands to release some space.

The geoapi-pt/root repo is private though very basic, it has only few files like credentials and Dockerfile, which is:

FROM node:16-slim

# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY resources/package*.json ./resources/
COPY main/package*.json ./main/

# RUN npm install
# If you are building your code for production
RUN cd resources && npm ci --omit=dev
RUN cd main && npm ci --omit=dev

RUN npm install pm2 -g

# Bundle app source
COPY . .

ENV NODE_ENV production
EXPOSE 8080
CMD [ "pm2-runtime", "main/src/server/index.js", "--", "--buildFeAssets", "--port", "8080" ]

I will try again with driver: docker and no cache

Thanks again

jfoclpf commented 4 months ago

Thank you, your suggestion made it work

https://github.com/geoapi-pt/main/actions/runs/8080969130/job/22078477115