Bumped docker/build-push-action to 5 and got "no disk space"

mskyttner commented 10 months ago

Contributing guidelines

[X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

[X] The documentation does not mention anything about my problem
[X] There are no open or closed issues that are related to my problem

Description

I just merged a dependabot PR to upgrade docker/build-push-action to 5 at https://github.com/KTH-Library/kontarion/blob/master/.github/workflows/push-kontarion.yml.

The previous build status was green but after merging the PR, the job fails with a "no disk space" annotation.

Expected behaviour

Bumping the version number for docker/build-push-action to 5 (dependabot also wanted to bump docker/setup-buildx-action, docker/metadata-action and docker/login-action one step up) was the only change since the previous succesful/green build, see https://github.com/KTH-Library/kontarion/actions/runs/4754371976.

So the expectation was to not see "no disk space" reported.

Actual behaviour

This annotation was shown:

System.IO.IOException: No space left on device : '/home/runner/runners/2.309.0/_diag/Worker_20230915-063024-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.309.0/_diag/Worker_20230915-063024-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Common.Tracing.Error(Exception exception)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.309.0/_diag/Worker_20230915-063024-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at System.Diagnostics.TraceSource.Flush()
   at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing)
   at GitHub.Runner.Common.TraceManager.Dispose()
   at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing)
   at GitHub.Runner.Common.HostContext.Dispose()
   at GitHub.Runner.Worker.Program.Main(String[] args)

Repository URL

https://github.com/KTH-Library/kontarion

Workflow run URL

https://github.com/KTH-Library/kontarion/actions/runs/6194704763/job/16818117829

YAML workflow

on:
  workflow_dispatch:
  push:
    branches:
      - master
#    paths:
#      - '.github/workflows/push-kontarion.yml'

jobs:
  kontarion-push:
    runs-on: ubuntu-latest
    steps:
    -
      name: Checkout
      uses: actions/checkout@v4
    -
      name: Docker meta
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: |
          ghcr.io/${{ github.repository_owner }}/kontarion
        tags: |
          type=raw,value=1.0.${{ github.run_number }},priority=1000
          type=ref,event=branch
          type=sha
          type=raw,value=latest
    -
      name: Set up QEMU
      uses: docker/setup-qemu-action@v2
    -
      name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3
    -
      name: Login to Container Registry
      uses: docker/login-action@v3
      with:
        registry: ghcr.io
        username: ${{ github.repository_owner }}
        password: ${{ secrets.CR_PAT }}
    -
      name: Docker build and push
      uses: docker/build-push-action@v5
      with:
        context: .
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        push: true
        cache-from: type=gha
        cache-to: type=gha,mode=max

Workflow logs

logs_5.zip

BuildKit logs

No response

Additional info

The "no disk space" issue is similar to https://github.com/docker/build-push-action/issues/238 - ie no logs but an "annotation" describes the issue.

Unclear what to do - can I clear out disk space somehow? Clear some caches?

How can I understand how to fix the issue?

Can/should my YAML workflow file be improved to avoid the "no disk space" issue?

crazy-max commented 10 months ago

This annotation was shown:

This error is returned by the runner, not the action. Like https://github.com/actions/runner/issues/1807

I just merged a dependabot PR to upgrade docker/build-push-action to 5

I don't think this is related. Your last build was 5 months ago so maybe now there is less space left on runner version 2.309.0. You can try to revert to v4 if yo uwant but I think you will have the same issue. You can try freeing up disk space as suggested in https://github.com/actions/runner/issues/1807#issuecomment-1537175089.

mskyttner commented 10 months ago

Confirming that it does not seem related to docker/build-push-action. My issue seems to have been caused by either working off a upstream base image which may have grown(?) retrospectively even though I used a pinned version, or maybe more likely due to lower disk space available these days on the latest ubuntu runners when compared to five months ago.

I used this 4 minute step to free up 23 Gb disk space on the runner after the checkout step:

    - 
      name: Free disk space on Ubuntu runner 🔧
      uses: kfir4444/free-disk-space@main
      with:
        # frees about 6 GB, warning: may remove required tools
        tool-cache: false
        android: true
        dotnet: true
        haskell: true
        large-packages: true
        swap-storage: true

@crazy-max thanks so much for those pointers, you saved my day (again), the build is green and now I can get out and mill some real logs instead of slicing through text logs :)

Thinking that an official "docker/buildx-ubunturunner-freespace" step that removes all extra stuff not required for running a build-x job on a standard runner would be nice, 23 Gb is a substantial portion of the total available space!

docker / build-push-action