ARMv7 Build is not working

amLiux commented 9 months ago

Contributing guidelines

[X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

[X] The documentation does not mention anything about my problem
[X] There are no open or closed issues that are related to my problem

Description

I had a workflow that looked something like this for my qa branch:

on:
  pull_request:
    types:
      - closed
    branches:
      - qa
jobs:
  deploy-docker-qa:
    runs-on: ubuntu-latest
    steps:
      - name: BTC (build to cache)
        uses: docker/setup-buildx-action@v1
      - name: Login to Docker Hub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKER_HUB_USERNAME }}
          password: ${{ secrets.DOCKER_HUB_TOKEN }}
      - name: Build :qa tag and push to Docker Hub
        uses: docker/build-push-action@v3
        with:
          push: true
          tags: marceliux/jardinbinario:qa
          build-args: |
            "NEXT_PUBLIC_BACKEND_URL=${{ secrets.NEXT_PUBLIC_BACKEND_URL }}"
            "NEXT_PUBLIC_PLACEHOLDER_IMAGE=${{ secrets.NEXT_PUBLIC_PLACEHOLDER_IMAGE }}"
            "NEXT_PUBLIC_UPLOAD_IMAGE=${{ secrets.NEXT_PUBLIC_UPLOAD_IMAGE }}"
            "NEXT_PUBLIC_CLOUDINARY_UPLOAD_URL=${{ secrets.NEXT_PUBLIC_CLOUDINARY_UPLOAD_URL }}"
          cache-from: type=gha
          cache-to: type=gha,mode=max

I tried adding a new workflow which implemented a new arch as we have a couple of RPI clusters to load QA efforts into them which use armv7l for each branch and ended up looking something like this:

But it's taking too long, not sure if I leave it for 5+ it will complete.

Expected behaviour

To complete armv7 and amd64 builds and push them.

Actual behaviour

Taking up to 5h for building and not really sure if working.

Repository URL

https://github.com/amLiux/JardinBinario

Workflow run URL

https://github.com/amLiux/JardinBinario/actions/runs/6366029508

YAML workflow

on:
  pull_request_target:
    types:
      - closed
  pull_request:
    branches:
      - 'dev'
jobs:
  dev_branches:
      if: github.event.pull_request.merged == true
      runs-on: ubuntu-latest
      env:
        DOCKER_TAG:
      steps:
       - name: Extract branch name
         shell: bash
         run: echo "branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> $GITHUB_OUTPUT
         id: extract_branch
       - name: Process branch name to meet valid sintax (lowercase and remove -)
         id: process_branch
         run: |
           branch_name="${{ steps.extract_branch.outputs.branch }}"
           lowercase_branch_name="${branch_name,,}"
           underscored_branch_name="${lowercase_branch_name//-/_}"
           echo "processed_branch=$underscored_branch_name" >> $GITHUB_OUTPUT
       - name: Set up QEMU
         id: qemu
         uses: docker/setup-qemu-action@v2
         with:
           image: tonistiigi/binfmt:latest
           platforms: armv7l, amd64
       - name: BTC (build to cache)
         uses: docker/setup-buildx-action@v2
       - name: Login to Docker Hub
         uses: docker/login-action@v1
         with:
          username: ${{ secrets.DOCKER_HUB_USERNAME }}
          password: ${{ secrets.DOCKER_HUB_TOKEN }}
       - name: Set dynamic branch Docker tag
         id: set_docker_tag
         run: echo "DOCKER_TAG=marceliux/jardinbinario:${{ steps.process_branch.outputs.processed_branch }}" >> $GITHUB_ENV
       - name: Build dev branch tag and push to Docker Hub
         uses: docker/build-push-action@v3
         with:
          push: true
          platforms: linux/amd64,linux/arm/v7 
          tags: ${{ env.DOCKER_TAG }}
          build-args: |
              "NEXT_PUBLIC_BACKEND_URL=${{ secrets.NEXT_PUBLIC_BACKEND_URL }}"
              "NEXT_PUBLIC_PLACEHOLDER_IMAGE=${{ secrets.NEXT_PUBLIC_PLACEHOLDER_IMAGE }}"
              "NEXT_PUBLIC_UPLOAD_IMAGE=${{ secrets.NEXT_PUBLIC_UPLOAD_IMAGE }}"
              "NEXT_PUBLIC_CLOUDINARY_UPLOAD_URL=${{ secrets.NEXT_PUBLIC_CLOUDINARY_UPLOAD_URL }}"
          cache-from: type=gha
          cache-to: type=gha,mode=max

Workflow logs

logs_86.zip

BuildKit logs

No response

Additional info

No response

KevinRohn commented 9 months ago

Hi @amLiux,

I have the same issue. For me it looks like, it has something to do with the node:current-alpine Container. I checked your workflow, and it is hanging at the npm install execution.

On my side the issue occurred after I switched from Node v18 to Node v20. Can you please try to use node v18 and give me feedback if the build process is still stuck? I try to fix this for hours, because I see some huge performance increasements in my application by using Node v20 compared to v18.

Thank you in advance.

crazy-max commented 9 months ago

Taking up to 5h for building and not really sure if working.

Looking at your workflow https://github.com/amLiux/JardinBinario/blob/83a4b500744829132b611ae9070b76e7c3344913/.github/workflows/devBranchDockerDeploy.yml#L11

You're building on ubuntu-latest which is a x86 host and therefore building against a non-native platform such as linux/arm/v7 in your case is expected to be quite resource intensive as it will be built using emulation (QEMU). More info: https://docs.docker.com/build/building/multi-platform/.

If you cannot use cross-compilation with this RUN step https://github.com/amLiux/JardinBinario/blob/83a4b500744829132b611ae9070b76e7c3344913/Dockerfile#L28, then you need a native BuildKit node to remove emulation penalty by configuring nodes like https://docs.docker.com/build/ci/github-actions/configure-builder/#append-additional-nodes-to-the-builder

Or you could distribute platform builds across multiple runners if that can help reducing build time: https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners

KevinRohn commented 9 months ago

You're building on ubuntu-latest which is a x86 host and therefore building against a non-native platform such as linux/arm/v7 in your case is expected to be quite resource intensive as it will be built using emulation (QEMU). More info: https://docs.docker.com/build/building/multi-platform/.

I think it is the issue in this case. It looks like there is a bug in qemu (https://gitlab.com/qemu-project/qemu/-/issues/1729)

You can also check these issues: https://github.com/nodejs/docker-node/issues/1798 https://github.com/nodejs/docker-node/issues/1973

tboerger commented 7 months ago

Maybe this issue and my currently stuck build are really related to the QEMU issue? Builds with an Alpine image v3.18 worked fine like https://github.com/toolhippie/commitlint/actions/runs/7083830830/job/19276929005 and the pull request to upgrade Alpine to v3.19 results in canceled builds after running for 6 hours https://github.com/toolhippie/commitlint/actions/runs/7173386897/job/19532635612 which obviously worked fine with older Alpine image versions.

docker / build-push-action