Hang after build and before "pushing layers"

[Fish1]

Troubleshooting

Behaviour

Steps to reproduce this issue

1. checkout@v3
2. login-action@v2
3. metadata-action@v3
4. build-push-action@v4

Expected behaviour

The image should be pushed to ghcr.io

Actual behaviour

After the build and before "pushing layers" step.

I have a nearly identical workflow here that builds another image. And it works fine and doesn't hang. https://github.com/yapms/yapms/actions/runs/5872278682

Configuration

• Repository URL (if public): https://github.com/yapms/yapms
• Build URL (if public): https://github.com/yapms/yapms/actions/runs/5872278683/job/15926291141

name: Build YAPms Image

on:
  push:
    branches:
      - main

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}-yapms

jobs:
  build:
    runs-on: ubuntu-latest

    permissions:
      contents: read
      packages: write

    steps:

      - uses: actions/checkout@v3

      - uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - uses: docker/metadata-action@v4
        id: metadata
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      - uses: docker/build-push-action@v4
        with:
          context: .
          file: ./Dockerfile.yapms
          build-args: |
            DOTENV_VAULT_KEY=${{ secrets.DOTENV_VAULT_KEY }}
          push: true
          tags: ${{ steps.metadata.outputs.tags }}
          labels: ${{ steps.metadata.outputs.labels }}

Logs

logs_876.zip

[crazy-max]

Don't think this is related to the action but your Dockerfile.

Looks to hang here https://github.com/yapms/yapms/blob/ebd40c7e5728ffac01f8760884103e874996af23/Dockerfile.yapms#L18. Do you repro locally?

Also looking at https://github.com/yapms/yapms/blob/ebd40c7e5728ffac01f8760884103e874996af23/.github/workflows/build-yapms-image.yml#L39-L40, this is a bad practice to use build arg to pass secrets. You should use the secrets input for this: https://docs.docker.com/build/ci/github-actions/secrets/

[Fish1]

Thanks for the tips. I'll look into the secrets input.

I wasn't able to reproduce this locally. I've used nektos/act. I get past the build and past the "pushing layers", up until I didn't have the correct github token to push to the image to ghcr.io. In which I get an error, but I don't hang. I can try getting a more permissive token tomorrow.

This only started happening 4 days ago... and only for this image.

[crazy-max]

What I meant by repro locally is with a simple build command like:

$ docker build --build-arg DOTENV_VAULT_KEY=foo -f ./Dockerfile.yapms .

Without pushing to see if it's linked to GHCR registry or not. Also can you enable BuildKit debug logs? See https://github.com/docker/build-push-action/blob/master/TROUBLESHOOTING.md#cannot-push-to-a-registry and https://docs.docker.com/build/ci/github-actions/configure-builder/#buildkit-container-logs

[Fish1]

Yes, I am able to build the image locally. Will try to enable BuildKit debug logs now.

[Fish1]

I've added the following to my CI.

Here are the logs, I stopped the CI after 4 minutes.

logs_881.zip

[crazy-max]

Looks like it works now https://github.com/yapms/yapms/actions/runs/5898280503/job/15999107870#step:6:707?

[Fish1]

Yeah, it's going through intermittently. Because it's working sometimes, and not others, I feel like it might just be a github server error? You can close this, if you feel like it isn't an issue with the action. Thanks!

[crazy-max]

I don't think this is an issue with the GitHub registry but your build as it hangs in https://github.com/yapms/yapms/actions/runs/5898078668/job/15998722706#step:6:746:

#17 45.68 yapms:build:   ✔ done
#17 45.68 yapms:build: ✓ built in 43.18s
Error: The operation was canceled.

When it works you have: https://github.com/yapms/yapms/actions/runs/5898280503/job/15999107870#step:6:707

#17 39.80 yapms:build:   ✔ done
#17 39.80 yapms:build: ✓ built in 37.43s
#17 39.89 
#17 39.89  Tasks:    1 successful, 1 total
#17 39.89 Cached:    0 cached, 1 total
#17 39.89   Time:    39.337s 
#17 39.89 
#17 DONE 40.0s

So this is before pushing anything to the registry. BuildKit logs also show the same behavior and hangs at pnpm run build: https://github.com/yapms/yapms/actions/runs/5898078668/job/15998722706#step:10:80

  time="2023-08-18T02:26:35Z" level=debug msg="> creating z4ke98lqj68xr2qu9vw6sx6ss [/bin/sh -c npm install turbo --global]" span="[builder 4/5] RUN npm install turbo --global" spanID=f2ad299bbdb7b08b traceID=49efa9f74cd81e3c81ee60fb6091e9ab
  time="2023-08-18T02:26:37Z" level=debug msg="> creating aaz0w8rs1i3sog2po12o9rath [/bin/sh -c turbo prune --scope=yapms --docker]" span="[builder 5/5] RUN turbo prune --scope=yapms --docker" spanID=fd2427a3336d8c00 traceID=49efa9f74cd81e3c81ee60fb6091e9ab
  time="2023-08-18T02:26:38Z" level=debug msg="> creating mc83cgbyvpvketztirgslr4f4 [/bin/sh -c pnpm install]" span="[installer 7/9] RUN pnpm install" spanID=83d64e966d5e7ad6 traceID=49efa9f74cd81e3c81ee60fb6091e9ab
  time="2023-08-18T02:26:56Z" level=debug msg="> creating qreq4n4u2pwyqyf34zoawusg1 [/bin/sh -c pnpm dotenv-vault decrypt ${DOTENV_VAULT_KEY} > .env]" span="[installer 8/9] RUN pnpm dotenv-vault decrypt *** > .env" spanID=3fc8a1cc5284acac traceID=49efa9f74cd81e3c81ee60fb6091e9ab
  time="2023-08-18T02:26:57Z" level=debug msg="> creating zw61xwy6oqe0f0nhw65a3v3zg [/bin/sh -c pnpm run build]" span="[installer 9/9] RUN pnpm run build" spanID=d87d0768018315cf traceID=49efa9f74cd81e3c81ee60fb6091e9ab

Unfortunately I can't run this build locally as it needs DOTENV_VAULT_KEY.

Closing since it doesn't seem related to the action but your Dockerfile but keep us posted if you have more information. Thanks.