Exporting to GitHub Actions Cache fails with TLS handshake timeout

[franpog859]

Contributing guidelines

• [X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• [X] The documentation does not mention anything about my problem
• [X] There are no open or closed issues that are related to my problem

Description

A couple of days ago this Action started to fail on all of my PRs to my private repository. It says it's due to Error: buildx failed with: ERROR: failed to solve: Get "https://asdf.actions.githubusercontent.com/qwer1234/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%qwer1234&version=1234": net/http: TLS handshake timeout. The step configuration is below and it's consistent with the documentation. I did not change anything on my side. What could go wrong and how can I fix it? I observed it first time on a Dependabot PR that bumps this very Action to @v5 version

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          # https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md#github-cache
          cache-from: type=gha
          cache-to: type=gha,mode=max
          push: true
          tags: |
            ${{ steps.meta.outputs.tags }}
          build-args: |
            GIT_HASH=${{ github.sha }}
            ENTRYPOINT_DIR=./cmd
            VERSION=${{ steps.version.outputs.value }}

Expected behaviour

The Action works as it always did

Actual behaviour

The Action fails with this error every time. I did not change anything in the configuration:

...
#20 exporting to GitHub Actions Cache
#20 preparing build cache for export
#20 preparing build cache for export 11.9s done
#20 ERROR: Get "https://asdf.actions.githubusercontent.com/qwer1234/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%qwer1234&version=1234": net/http: TLS handshake timeout
------
 > importing cache manifest from gha:1234:
------
------
 > exporting to GitHub Actions Cache:
------
ERROR: failed to solve: Get "https://asdf.actions.githubusercontent.com/qwer1234/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%qwer1234&version=1234": net/http: TLS handshake timeout
Error: buildx failed with: ERROR: failed to solve: Get "https://asdf.actions.githubusercontent.com/qwer1234/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%qwer1234&version=1234": net/http: TLS handshake timeout

Repository URL

it is private

Workflow run URL

it is private

YAML workflow

provided in the description

Workflow logs

provided in the actual behavior

BuildKit logs

No response

Additional info

No response

[mrclrchtr]

Same here..

#13 exporting to GitHub Actions Cache
#13 preparing build cache for export
#13 preparing build cache for export 14.6s done
#13 ERROR: Get "https://acghubeus1.actions.githubusercontent.com/fNeSvcg.../_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A17d24ed60...": net/http: TLS handshake timeout
------
 > importing cache manifest from gha:17849619333386471842:
------
------
 > exporting to GitHub Actions Cache:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
ERROR: failed to solve: Get "https://acghubeus1.actions.githubusercontent.com/fNeSvcg3D2e.../_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A17d24ed6036f479...": net/http: TLS handshake timeout
Error: buildx failed with: ERROR: failed to solve: Get "https://acghubeus1.actions.githubusercontent.com/fNeSvcg3D2e.../_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A17d24ed6...&version=693bb...": net/http: TLS handshake timeout

      - name: Set up QEMU
        uses: docker/setup-qemu-action@68827325e0b33c7199eb31dd4e31fbe9023e06e3 # v3
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@f95db51fddba0c2d1ec667646a06c2ce06100226 # v3
      - name: Build and push Docker image
        uses: docker/build-push-action@0565240e2d4ab88bba5387d719585280857ece09 # v5.0.0
        with:
          context: ${{ matrix.config.module_dir }}
          platforms: linux/amd64,linux/arm64
          push: ${{ env.IMAGE_PUBLISH }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

[KenBurk]

ditto

[obinna240]

same here, started a couple of days ago

[mrclrchtr]

I removed

cache-from: type=gha
cache-to: type=gha,mode=max

Then it worked again.

[franpog859]

I did the same, @mrclrchtr . It's fine as a workaround to enable the pipeline. It's not a solution to the real problem though

[mrclrchtr]

Absolutely right. I just wanted to share a workaround to help others.

[crazy-max]

Get "https://asdf.actions.githubusercontent.com/qwer1234/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%qwer1234&version=1234": net/http: TLS handshake timeout

This is a transient network error with GitHub infrastructure and nothing much we can do about it unfortunately. Suggest to contact them. Also check https://www.githubstatus.com/.

[crazy-max]

Otoh you can prevent build to fail by setting ignore-error: https://docs.docker.com/build/cache/backends/gha/#synopsis

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          cache-from: type=gha
          cache-to: type=gha,mode=max,ignore-error=true

[aequitas]

@crazy-max ignoring errors is only a partial solution for me as I've seen builds not fail directly but hang for hours during pushing of layers to cache until the entire job is timed out (https://github.com/internetstandards/Internet.nl/actions/runs/6262993085/job/17104020359#step:8:293), with no clear indicator of the core issue. It might be that the core of that problem is separate from the TLS timeout, but I've been seeing those same errors on builds as well so my idea is that they are related. I'm experiencing these issues on a workflow that pushes multiple images at the same time in a job matrix (https://github.com/internetstandards/Internet.nl/blob/main/.github/workflows/docker.yml). For now my solution has been to disable gha caching. Should I open a separate issue for this?

[ohbriansung]

To those who still encountered this issue, if you didn't find any useful information, consider this:

This is likely not related to docker/build-push-action but the issue with Maximum Transmission Unit (MTU) setting of the docker0 network interface instead. Try running the ip a command in your workflow to print your networking settings. If you see the MTU of the docker0 being greater than the MTU of the outgoing interface. See below code block, in this case, the outgoing interface is ens3. This would be your root cause of the TLS handshake timeout on docker push and cache operations. The MTU of docker0 needs to be equal or lower to the MTU of the outgoing interface.

2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fb:1a:3e:47:80:62 brd ff:ff:ff:ff:ff:ff
    inet 10.17.228.5/24 brd 10.35.228.255 scope global ens3
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:83:20:dc:cc brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever

There are a lot of sources out there for configuring the MTU of docker. However, if you are using actions-runner-controller (ARC) to host a self-hosted runner for your GitHub Action, see this for how to configure that for your ARC.

References:

• https://www.civo.com/learn/fixing-networking-for-docker