go-gitea / gitea

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD
https://gitea.com
MIT License
45k stars 5.48k forks source link

Gitea Runner Fails To Push To The Registry On Certain Images #31865

Open TheDarkula opened 2 months ago

TheDarkula commented 2 months ago

Description

I have a Gitea action that pulls docker images and pushes them to the Gitea registry.

On some images, the push does not succeed.

The actions job shows this:

546ad1cf1b01: Preparing
5f70bf18a086: Preparing
55b70ce9d016: Preparing
327bd3ff3130: Preparing
d6955580f40d: Preparing
f44d4557638f: Preparing
32bf7eabde06: Preparing
646a59710f2a: Preparing
b262c6329a45: Preparing
78011e011ca1: Preparing
882acb3fbb9d: Preparing
9375032f1539: Preparing
ccf17877232e: Preparing
bed3270bc1c6: Preparing
809d3bb9c80f: Preparing
5136ffc45974: Preparing
2ecbf7829cb7: Preparing
cd76869b72ab: Preparing
106e8431b412: Preparing
f44d4557638f: Waiting
882acb3fbb9d: Waiting
646a59710f2a: Waiting
32bf7eabde06: Waiting
b262c6329a45: Waiting
5136ffc45974: Waiting
9375032f1539: Waiting
78011e011ca1: Waiting
2ecbf7829cb7: Waiting
ccf17877232e: Waiting
809d3bb9c80f: Waiting
bed3270bc1c6: Waiting
106e8431b412: Waiting
5f70bf18a086: Layer already exists
d6955580f40d: Pushed
55b70ce9d016: Pushed
646a59710f2a: Pushed
b262c6329a45: Pushed
f44d4557638f: Pushed
882acb3fbb9d: Pushed
9375032f1539: Pushed
ccf17877232e: Pushed
32bf7eabde06: Pushed
809d3bb9c80f: Pushed
5136ffc45974: Pushed
2ecbf7829cb7: Pushed
cd76869b72ab: Pushed
106e8431b412: Pushed
327bd3ff3130: Retrying in 5 seconds
546ad1cf1b01: Retrying in 5 seconds
78011e011ca1: Retrying in 5 seconds
327bd3ff3130: Retrying in 4 seconds
546ad1cf1b01: Retrying in 4 seconds
bed3270bc1c6: Retrying in 5 seconds
78011e011ca1: Retrying in 4 seconds
327bd3ff3130: Retrying in 3 seconds
546ad1cf1b01: Retrying in 3 seconds
bed3270bc1c6: Retrying in 4 seconds
78011e011ca1: Retrying in 3 seconds
327bd3ff3130: Retrying in 2 seconds
546ad1cf1b01: Retrying in 2 seconds
bed3270bc1c6: Retrying in 3 seconds
78011e011ca1: Retrying in 2 seconds
327bd3ff3130: Retrying in 1 second
546ad1cf1b01: Retrying in 1 second
bed3270bc1c6: Retrying in 2 seconds
78011e011ca1: Retrying in 1 second
bed3270bc1c6: Retrying in 1 second
327bd3ff3130: Retrying in 10 seconds
546ad1cf1b01: Retrying in 10 seconds
78011e011ca1: Retrying in 10 seconds
327bd3ff3130: Retrying in 9 seconds
546ad1cf1b01: Retrying in 9 seconds
bed3270bc1c6: Retrying in 10 seconds
78011e011ca1: Retrying in 9 seconds
327bd3ff3130: Retrying in 8 seconds
546ad1cf1b01: Retrying in 8 seconds
bed3270bc1c6: Retrying in 9 seconds
78011e011ca1: Retrying in 8 seconds
327bd3ff3130: Retrying in 7 seconds
546ad1cf1b01: Retrying in 7 seconds
bed3270bc1c6: Retrying in 8 seconds
78011e011ca1: Retrying in 7 seconds
327bd3ff3130: Retrying in 6 seconds
546ad1cf1b01: Retrying in 6 seconds
bed3270bc1c6: Retrying in 7 seconds
78011e011ca1: Retrying in 6 seconds
327bd3ff3130: Retrying in 5 seconds
546ad1cf1b01: Retrying in 5 seconds
bed3270bc1c6: Retrying in 6 seconds
78011e011ca1: Retrying in 5 seconds
327bd3ff3130: Retrying in 4 seconds
546ad1cf1b01: Retrying in 4 seconds
bed3270bc1c6: Retrying in 5 seconds
78011e011ca1: Retrying in 4 seconds
327bd3ff3130: Retrying in 3 seconds
546ad1cf1b01: Retrying in 3 seconds
bed3270bc1c6: Retrying in 4 seconds
78011e011ca1: Retrying in 3 seconds
327bd3ff3130: Retrying in 2 seconds
546ad1cf1b01: Retrying in 2 seconds
bed3270bc1c6: Retrying in 3 seconds
78011e011ca1: Retrying in 2 seconds
327bd3ff3130: Retrying in 1 second
546ad1cf1b01: Retrying in 1 second
bed3270bc1c6: Retrying in 2 seconds
78011e011ca1: Retrying in 1 second
bed3270bc1c6: Retrying in 1 second
327bd3ff3130: Retrying in 15 seconds
546ad1cf1b01: Retrying in 15 seconds
78011e011ca1: Retrying in 15 seconds
327bd3ff3130: Retrying in 14 seconds
546ad1cf1b01: Retrying in 14 seconds
bed3270bc1c6: Retrying in 15 seconds
78011e011ca1: Retrying in 14 seconds
327bd3ff3130: Retrying in 13 seconds
546ad1cf1b01: Retrying in 13 seconds
bed3270bc1c6: Retrying in 14 seconds
78011e011ca1: Retrying in 13 seconds
327bd3ff3130: Retrying in 12 seconds
546ad1cf1b01: Retrying in 12 seconds
bed3270bc1c6: Retrying in 13 seconds
78011e011ca1: Retrying in 12 seconds
327bd3ff3130: Retrying in 11 seconds
546ad1cf1b01: Retrying in 11 seconds
bed3270bc1c6: Retrying in 12 seconds
78011e011ca1: Retrying in 11 seconds
327bd3ff3130: Retrying in 10 seconds
546ad1cf1b01: Retrying in 10 seconds
bed3270bc1c6: Retrying in 11 seconds
78011e011ca1: Retrying in 10 seconds
327bd3ff3130: Retrying in 9 seconds
546ad1cf1b01: Retrying in 9 seconds
bed3270bc1c6: Retrying in 10 seconds
78011e011ca1: Retrying in 9 seconds
327bd3ff3130: Retrying in 8 seconds
546ad1cf1b01: Retrying in 8 seconds
bed3270bc1c6: Retrying in 9 seconds
78011e011ca1: Retrying in 8 seconds
327bd3ff3130: Retrying in 7 seconds
546ad1cf1b01: Retrying in 7 seconds
bed3270bc1c6: Retrying in 8 seconds
78011e011ca1: Retrying in 7 seconds
327bd3ff3130: Retrying in 6 seconds
546ad1cf1b01: Retrying in 6 seconds
bed3270bc1c6: Retrying in 7 seconds
78011e011ca1: Retrying in 6 seconds
327bd3ff3130: Retrying in 5 seconds
546ad1cf1b01: Retrying in 5 seconds
bed3270bc1c6: Retrying in 6 seconds
78011e011ca1: Retrying in 5 seconds
327bd3ff3130: Retrying in 4 seconds
546ad1cf1b01: Retrying in 4 seconds
bed3270bc1c6: Retrying in 5 seconds
78011e011ca1: Retrying in 4 seconds
327bd3ff3130: Retrying in 3 seconds
546ad1cf1b01: Retrying in 3 seconds
bed3270bc1c6: Retrying in 4 seconds
78011e011ca1: Retrying in 3 seconds
327bd3ff3130: Retrying in 2 seconds
546ad1cf1b01: Retrying in 2 seconds
bed3270bc1c6: Retrying in 3 seconds
78011e011ca1: Retrying in 2 seconds
327bd3ff3130: Retrying in 1 second
546ad1cf1b01: Retrying in 1 second
bed3270bc1c6: Retrying in 2 seconds
78011e011ca1: Retrying in 1 second
bed3270bc1c6: Retrying in 1 second
327bd3ff3130: Retrying in 20 seconds
546ad1cf1b01: Retrying in 20 seconds
78011e011ca1: Retrying in 20 seconds
bed3270bc1c6: Retrying in 20 seconds
327bd3ff3130: Retrying in 19 seconds
546ad1cf1b01: Retrying in 19 seconds
78011e011ca1: Retrying in 19 seconds
bed3270bc1c6: Retrying in 19 seconds
327bd3ff3130: Retrying in 18 seconds
546ad1cf1b01: Retrying in 18 seconds
78011e011ca1: Retrying in 18 seconds
bed3270bc1c6: Retrying in 18 seconds
327bd3ff3130: Retrying in 17 seconds
546ad1cf1b01: Retrying in 17 seconds
78011e011ca1: Retrying in 17 seconds
bed3270bc1c6: Retrying in 17 seconds
327bd3ff3130: Retrying in 16 seconds
546ad1cf1b01: Retrying in 16 seconds
78011e011ca1: Retrying in 16 seconds
bed3270bc1c6: Retrying in 16 seconds
327bd3ff3130: Retrying in 15 seconds
546ad1cf1b01: Retrying in 15 seconds
78011e011ca1: Retrying in 15 seconds
bed3270bc1c6: Retrying in 15 seconds
327bd3ff3130: Retrying in 14 seconds
546ad1cf1b01: Retrying in 14 seconds
78011e011ca1: Retrying in 14 seconds
bed3270bc1c6: Retrying in 14 seconds
327bd3ff3130: Retrying in 13 seconds
546ad1cf1b01: Retrying in 13 seconds
78011e011ca1: Retrying in 13 seconds
bed3270bc1c6: Retrying in 13 seconds
327bd3ff3130: Retrying in 12 seconds
546ad1cf1b01: Retrying in 12 seconds
78011e011ca1: Retrying in 12 seconds
bed3270bc1c6: Retrying in 12 seconds
327bd3ff3130: Retrying in 11 seconds
546ad1cf1b01: Retrying in 11 seconds
78011e011ca1: Retrying in 11 seconds
bed3270bc1c6: Retrying in 11 seconds
327bd3ff3130: Retrying in 10 seconds
546ad1cf1b01: Retrying in 10 seconds
78011e011ca1: Retrying in 10 seconds
bed3270bc1c6: Retrying in 10 seconds
327bd3ff3130: Retrying in 9 seconds
546ad1cf1b01: Retrying in 9 seconds
78011e011ca1: Retrying in 9 seconds
bed3270bc1c6: Retrying in 9 seconds
327bd3ff3130: Retrying in 8 seconds
546ad1cf1b01: Retrying in 8 seconds
78011e011ca1: Retrying in 8 seconds
bed3270bc1c6: Retrying in 8 seconds
327bd3ff3130: Retrying in 7 seconds
546ad1cf1b01: Retrying in 7 seconds
78011e011ca1: Retrying in 7 seconds
bed3270bc1c6: Retrying in 7 seconds
327bd3ff3130: Retrying in 6 seconds
546ad1cf1b01: Retrying in 6 seconds
78011e011ca1: Retrying in 6 seconds
bed3270bc1c6: Retrying in 6 seconds
327bd3ff3130: Retrying in 5 seconds
546ad1cf1b01: Retrying in 5 seconds
78011e011ca1: Retrying in 5 seconds
bed3270bc1c6: Retrying in 5 seconds
327bd3ff3130: Retrying in 4 seconds
546ad1cf1b01: Retrying in 4 seconds
78011e011ca1: Retrying in 4 seconds
bed3270bc1c6: Retrying in 4 seconds
327bd3ff3130: Retrying in 3 seconds
546ad1cf1b01: Retrying in 3 seconds
78011e011ca1: Retrying in 3 seconds
bed3270bc1c6: Retrying in 3 seconds
327bd3ff3130: Retrying in 2 seconds
546ad1cf1b01: Retrying in 2 seconds
78011e011ca1: Retrying in 2 seconds
bed3270bc1c6: Retrying in 2 seconds
327bd3ff3130: Retrying in 1 second
546ad1cf1b01: Retrying in 1 second
78011e011ca1: Retrying in 1 second
bed3270bc1c6: Retrying in 1 second
use of closed network connection

I also have these errors in the Gitea logs:

2024/08/18 16:34:20 ...eb/routing/logger.go:102:func1() [I] router: completed POST /username/repository-name/actions/runs/134/jobs/0 for user.ip.address.here:0, 200 OK in 17.8ms @ actions/view.go:131(actions.ViewPost)
2024/08/18 16:34:21 ...ntainer/container.go:90:apiError() [E] unexpected EOF
2024/08/18 16:34:21 ...eb/routing/logger.go:102:func1() [I] router: completed PATCH /v2/username/image-name/image-name/blobs/uploads/qb3ijuvhrtbi1romay5uha4dj for 10.0.0.125:0, 500 Internal Server Error in 60000.2ms @ packages/api.go:700(packages.ContainerRoutes.func2.2)
2024/08/18 16:34:21 ...ntainer/container.go:90:apiError() [E] unexpected EOF
2024/08/18 16:34:21 ...eb/routing/logger.go:102:func1() [I] router: completed PATCH /v2/username/image-name/image-name/blobs/uploads/keysexeohf0hsyohs61eanuck for 10.0.0.125:0, 500 Internal Server Error in 60000.0ms @ packages/api.go:700(packages.ContainerRoutes.func2.2)
2024/08/18 16:34:21 ...ntainer/container.go:90:apiError() [E] unexpected EOF
2024/08/18 16:34:21 ...eb/routing/logger.go:102:func1() [I] router: completed PATCH /v2/username/image-name/image-name/blobs/uploads/vvtqnhu0h1nfk7qarp61nvamu for 10.0.0.125:0, 500 Internal Server Error in 59124.5ms @ packages/api.go:700(packages.ContainerRoutes.func2.2)
2024/08/18 16:34:21 ...ntainer/container.go:90:apiError() [E] unexpected EOF
2024/08/18 16:34:21 ...eb/routing/logger.go:102:func1() [I] router: completed PATCH /v2/username/image-name/image-name/blobs/uploads/dkdkplfkajb0i2jyacfetffnp for 10.0.0.125:0, 500 Internal Server Error in 59869.9ms @ packages/api.go:700(packages.ContainerRoutes.func2.2)

Gitea Version

1.22.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

Linux

How are you running Gitea?

I have Gitea and the ACT Runner running in kubernetes.

Database

PostgreSQL

lunny commented 2 months ago

Can you try logout from docker and login again? Could you also try to upgrade docker's version?

TheDarkula commented 1 month ago

@lunny This is from a Gitea Actions Runner, so it logs in on every run. Is there some configuration setting that limits/sets the maximum docker image size?

lunny commented 1 month ago

No limitation by default. Can you list the images you try to upload?

TheDarkula commented 1 month ago

Examples that fail:

lunny commented 1 month ago

I cannot reproduce this problem in my local machine with main branch. Can you upgrade to v1.22.2 and try again?

图片
TheDarkula commented 1 month ago

I just upgraded gitea to v1.22.2 and upgraded the act_runner to 0.2.11-dind-rootless, but the issue is still happening.

I just tried with ollama/ollama:0.3.12, and I see this in the logs:

2024/10/07 08:20:40 ...eb/routing/logger.go:68:func1() [W] router: slow      PATCH /v2/username/ollama/ollama/blobs/uploads/kvr9kfbso38cq77ddkghylcbl for 10.0.0.67:0, elapsed 3088.0ms @ packages/api.go:700(packages.ContainerRoutes.func2.2)
2024/10/07 08:21:37 ...ntainer/container.go:92:apiError() [E] unexpected EOF
2024/10/07 08:21:37 ...eb/routing/logger.go:102:func1() [I] router: completed PATCH /v2/username/ollama/ollama/blobs/uploads/kvr9kfbso38cq77ddkghylcbl for 10.0.0.67:0, 500 Internal Server Error in 60000.5ms @ packages/api.go:700(packages.ContainerRoutes.func2.2)
lunny commented 1 month ago

Can you paste your related workflow file?

TheDarkula commented 1 week ago

Sure thing!

This is the workflow:

name: Push Docker Images

on:
  push:
    branches:
      - master

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Login to Gitea Container Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ vars.REGISTRY_URL }}
        username: ${{ github.actor }}
        password: ${{ secrets.REGISTRY_TOKEN }}

    - name: Run Images Script
      env:
        REGISTRY_URL: ${{ vars.REGISTRY_URL }}
        REGISTRY_OWNER: ${{ vars.REGISTRY_OWNER }}
      run: bash download-images-retag-and-push.sh

And here is the respective script

#!/bin/bash

set -eu

cat image_list.conf | while read image_name_and_tag || [ -n "$image_name_and_tag" ];
do
    if [[ "$image_name_and_tag" == \#* ]]; then
        echo "Skipping comment."
    else
        FULL_IMAGE_URL="$REGISTRY_URL/$REGISTRY_OWNER/$image_name_and_tag"

        set +e
        docker manifest inspect "$FULL_IMAGE_URL" > /dev/null
        EXIT_CODE=$?
        set -e

        if [ "$EXIT_CODE" -eq 0 ]; then
            echo "$FULL_IMAGE_URL already exists exists in the registry, skipping"
        else
            echo "$FULL_IMAGE_URL does not exist in the registry."    

            docker images

            echo "Pulling image: $image_name_and_tag"

            docker pull "$image_name_and_tag"

            echo "Retagging image: $image_name_and_tag to: $FULL_IMAGE_URL"
            echo "New image tag: $FULL_IMAGE_URL"

            docker tag "$image_name_and_tag" "$FULL_IMAGE_URL"

            echo "Removing original image: $image_name_and_tag"

            docker rmi "$image_name_and_tag"

            echo "Pushing new image: $FULL_IMAGE_URL"

            docker push "$FULL_IMAGE_URL"
        fi
    fi
done

You will need a file called image_list.conf with an image in it like we discussed:

ollama/ollama:0.3.12

Also, you will need two environment variables set in the repository: