Git clone fails with "Connection reset by peer"

Eragonfr commented 1 year ago

Component

agent, other

Describe the bug

I'm pretty sure that this is a bug in the plugin-git image or in my networking setup on my computer.

My git server is only available in ipv6 on my local network (as there is a bug on my router that prevent connexion to itself in ipv4). Clonning from a empty alpine container works, clonning from the host works too. Is this related to the other ipv6 issues ?

Sorry if this is just an error in my setup but I can't find any way to get this working.

System Info

Server build from source "https://github.com/woodpecker-ci/woodpecker"
version "next-328f1044" (I'm pretty sure that I build that on the v0.15.5 git tag)
The server is running on an alpine linux aarch64.

The agent is running on my main computer.
OS: Archlinux AMD64
Output of `woodpecker-agent -v`: `woodpecker-agent version dev` (I just took the package from the community repo)
I use podman as a drop-in replacement for docker

Additional context

+ git init -b main
  Initialized empty Git repository in /woodpecker/src/git.eragon.re/eragon/woodpecker-test/.git/
  + git remote add origin https://git.eragon.re/eragon/woodpecker-test.git
  + git fetch --no-tags origin +refs/heads/main:   fatal: unable to access 'https://git.eragon.re/eragon/woodpecker-test.git/': OpenSSL SSL_connect: Connection reset by peer in connection to git.eragon.re:443

Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

6543 commented 1 year ago

we will publish plugin-git v2.0.0 soon ... I'm alos gona bump it in the v0.15.x release branch...

if it's not fixed by then ... we can revisit that issue

Eragonfr commented 1 year ago

+ git init -b main
Initialized empty Git repository in /woodpecker/src/git.eragon.re/eragon/woodpecker-test/.git/
+ git remote add origin https://git.eragon.re/eragon/woodpecker-test.git
+ git fetch --no-tags origin +refs/heads/main:
fatal: unable to access 'https://git.eragon.re/eragon/woodpecker-test.git/': Could not resolve host: git.eragon.re
exit status 128

I pulled the last image and re-run the tests, now it doesn't even resolve the host.

A blank alpine docker is still able to resolve and clone from git.eragon.re

EDIT: This is not the same as #1171 as my host has public DNS records (and you can access it if you want)

6543 commented 1 year ago

@Eragonfr can you check if it works now?

Eragonfr commented 1 year ago

I can't get it to work...

Also I can't find how to test with the woodpecker-agent:next docker image, it seems that the server doesn't allow the agent to register, so I can't even test if enabling IPv6 networking in the agent is enough to fix it.

I'm almost certain that the problem could be solved by using IPv6 in the bridge networks made by the agent.

6543 commented 1 year ago

hmm if you are on next ... try export WOODPECKER_BACKEND_DOCKER_ENABLE_IPV6=true

Eragonfr commented 1 year ago

I tried the server and the agent on next withWOODPECKER_BACKEND_DOCKER_ENABLE_IPV6 and I still got the same error. Is it possible that the git binary in the container does not use IPv6 by default and fail because of the broken IPv4 ?

At this point I don't understand what's happening in the container.

Result of woodpecker-agent --version : woodpecker-agent version next-f1e9c841

Here is the log of the agent when run with WOODPECKER_BACKEND_DOCKER_ENABLE_IPV6=true :

6:28PM INF ../../data/projects/woodpecker/cmd/agent/agent.go:180 > Starting Woodpecker agent with version 'next-f1e9c841' and backend 'docker' running up to 1 pipelines in parallel
6:28PM DBG ../../data/projects/woodpecker/cmd/agent/agent.go:164 > loaded docker backend engine
6:28PM DBG ../../data/projects/woodpecker/cmd/agent/agent.go:171 > polling new steps
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:61 > request next execution
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:97 > received execution id=4 pipeline=2 repo=eragon/woodpecker-test
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:111 > listen for cancel signal id=4 pipeline=2 repo=eragon/woodpecker-test
6:28PM DBG ../../data/projects/woodpecker/pipeline/pipeline.go:75 > Executing 2 stages, in order of: ID=4 Pipeline=2 Repo=eragon/woodpecker-test
6:28PM DBG ../../data/projects/woodpecker/pipeline/pipeline.go:85 > stage ID=4 Pipeline=2 Repo=eragon/woodpecker-test Stage=wp_0_314432866923000364_clone_0 Steps=wp_0_314432866923000364_clone_0
6:28PM DBG ../../data/projects/woodpecker/pipeline/pipeline.go:85 > stage ID=4 Pipeline=2 Repo=eragon/woodpecker-test Stage=wp_0_314432866923000364_stage_0 Steps=wp_0_314432866923000364_step_0
6:28PM DBG ../../data/projects/woodpecker/pipeline/pipeline.go:157 > Prepare ID=4 Pipeline=2 Repo=eragon/woodpecker-test Step=wp_0_314432866923000364_clone_0
6:28PM DBG ../../data/projects/woodpecker/agent/tracer.go:51 > update step status exit_code=0 exited=false id=4 image=woodpeckerci/plugin-git:2 pipeline=2 repo=eragon/woodpecker-test stage=git
6:28PM DBG ../../data/projects/woodpecker/agent/tracer.go:59 > update step status complete exit_code=0 exited=false id=4 image=woodpeckerci/plugin-git:2 pipeline=2 repo=eragon/woodpecker-test stage=git
6:28PM DBG ../../data/projects/woodpecker/pipeline/pipeline.go:184 > Executing ID=4 Pipeline=2 Repo=eragon/woodpecker-test Step=wp_0_314432866923000364_clone_0
6:28PM DBG ../../data/projects/woodpecker/agent/logger.go:53 > log stream opened id=4 image=woodpeckerci/plugin-git:2 pipeline=2 repo=eragon/woodpecker-test stage=git
6:28PM TRC ../../data/projects/woodpecker/pipeline/rpc/line.go:69 > grpc write line: + git init -b main
 ID=4 name=git
6:28PM TRC ../../data/projects/woodpecker/pipeline/rpc/line.go:69 > grpc write line: Initialized empty Git repository in /woodpecker/src/git.eragon.re/eragon/woodpecker-test/.git/
 ID=4 name=git
6:28PM TRC ../../data/projects/woodpecker/pipeline/rpc/line.go:69 > grpc write line: + git remote add origin https://git.eragon.re/eragon/woodpecker-test.git
 ID=4 name=git
6:28PM TRC ../../data/projects/woodpecker/pipeline/rpc/line.go:69 > grpc write line: + git fetch --no-tags --depth=1 --filter=tree:0 origin +refs/heads/main:
 ID=4 name=git
6:28PM TRC ../../data/projects/woodpecker/pipeline/rpc/line.go:69 > grpc write line: fatal: unable to access 'https://git.eragon.re/eragon/woodpecker-test.git/': Could not resolve host: git.eragon.re
 ID=4 name=git
6:28PM TRC ../../data/projects/woodpecker/pipeline/rpc/line.go:69 > grpc write line: exit status 128
 ID=4 name=git
6:28PM ERR ../../data/projects/woodpecker/agent/logger.go:58 > copy limited logStream part error="io: read/write on closed pipe"
6:28PM DBG ../../data/projects/woodpecker/agent/logger.go:61 > log stream copied id=4 image=woodpeckerci/plugin-git:2 pipeline=2 repo=eragon/woodpecker-test stage=git
6:28PM DBG ../../data/projects/woodpecker/agent/logger.go:77 > log stream uploading id=4 image=woodpeckerci/plugin-git:2 pipeline=2 repo=eragon/woodpecker-test stage=git
6:28PM DBG ../../data/projects/woodpecker/agent/logger.go:81 > log stream upload complete id=4 image=woodpeckerci/plugin-git:2 pipeline=2 repo=eragon/woodpecker-test stage=git
6:28PM DBG ../../data/projects/woodpecker/agent/logger.go:85 > log stream closed id=4 image=woodpeckerci/plugin-git:2 pipeline=2 repo=eragon/woodpecker-test stage=git
6:28PM DBG ../../data/projects/woodpecker/pipeline/pipeline.go:190 > Complete ID=4 Pipeline=2 Repo=eragon/woodpecker-test Step=wp_0_314432866923000364_clone_0
6:28PM DBG ../../data/projects/woodpecker/agent/tracer.go:51 > update step status exit_code=128 exited=true id=4 image=woodpeckerci/plugin-git:2 pipeline=2 repo=eragon/woodpecker-test stage=git
6:28PM DBG ../../data/projects/woodpecker/agent/tracer.go:59 > update step status complete exit_code=128 exited=true id=4 image=woodpeckerci/plugin-git:2 pipeline=2 repo=eragon/woodpecker-test stage=git
6:28PM DBG ../../data/projects/woodpecker/pipeline/pipeline.go:157 > Prepare ID=4 Pipeline=2 Repo=eragon/woodpecker-test Step=wp_0_314432866923000364_step_0
6:28PM DBG ../../data/projects/woodpecker/pipeline/pipeline.go:164 > Skipped due to OnFailure=false error="wp_0_314432866923000364_clone_0 : exit code 128" ID=4 Pipeline=2 Repo=eragon/woodpecker-test Step=wp_0_314432866923000364_step_0
6:28PM ERR ../../data/projects/woodpecker/pipeline/backend/docker/docker.go:238 > could not kill container 'wp_0_314432866923000364_clone_0' error="Error response from daemon: can only kill running containers. dcf0160133783e6afdf6fd75bf169ee861246143da714aaf699787e8c8a34c7a is in state exited: container state improper"
6:28PM ERR ../../data/projects/woodpecker/pipeline/backend/docker/docker.go:238 > could not kill container 'wp_0_314432866923000364_stage_0' error="Error response from daemon: no container with name or ID \"wp_0_314432866923000364_step_0\" found: no such container"
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:185 > pipeline complete error= canceled=false exit_code=128 id=4 pipeline=2 repo=eragon/woodpecker-test
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:187 > uploading logs id=4 pipeline=2 repo=eragon/woodpecker-test
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:189 > uploading logs complete id=4 pipeline=2 repo=eragon/woodpecker-test
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:194 > updating pipeline status error= exit_code=128 id=4 pipeline=2 repo=eragon/woodpecker-test
6:28PM ERR ../../data/projects/woodpecker/pipeline/rpc/client_grpc.go:97 > grpc error: wait(): code: Unknown: rpc error: code = Unknown desc = Step finished with exitcode 128,  error="rpc error: code = Unknown desc = Step finished with exitcode 128, "
6:28PM WRN ../../data/projects/woodpecker/agent/runner.go:115 > cancel signal received error="rpc error: code = Unknown desc = Step finished with exitcode 128, " id=4 pipeline=2 repo=eragon/woodpecker-test
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:127 > pipeline done id=4 pipeline=2 repo=eragon/woodpecker-test
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:199 > updating pipeline status complete id=4 pipeline=2 repo=eragon/woodpecker-test
6:28PM DBG ../../data/projects/woodpecker/cmd/agent/agent.go:171 > polling new steps
6:28PM DBG ../../data/projects/woodpecker/agent/runner.go:61 > request next execution

(I was running the binary from my home folder but the git clone is in /data/projects/woodpecker that's why there is a ../.. prefix everywhere)

6543 commented 1 year ago

hmm could be related to dns

often docker images use external dns servers ...

for example https://codeberg.org/woodpecker-plugins/plugin-docker-buildx/pulls/45 was added, to use the internal dns

Eragonfr commented 1 year ago

I don't think that it is the issue here, except if for some reason the dns does not replicate the content of the OVH dns servers.

6543 commented 6 months ago

does it still exist?

Eragonfr commented 6 months ago

Don't know, I moved to forgejo and act_runner, no longer using podman nor docker too.

voidcontext commented 2 weeks ago

It seems I bumped into this issue. I am also trying out podman instead of docker, and the git clone step fails on a aarch64/arm64 host:

+ git init -b main
Initialized empty Git repository in /woodpecker/src/git.vdx.hu/voidcontext/nix-config/.git/
+ git config --global --replace-all safe.directory /woodpecker/src/git.vdx.hu/voidcontext/nix-config
+ git remote add origin https://git.vdx.hu/voidcontext/nix-config.git
+ git fetch --no-tags --depth=1 --filter=tree:0 origin +a4b83a869d6366bf24921b8f4f3f5836f806f57e:fatal: unable to access 'https://git.vdx.hu/voidcontext/nix-config.git/': Could not resolve host: git.vdx.hu
exit status 128

Build: https://woodpecker.ci.vdx.hu/repos/109/pipeline/119/5

6543 commented 2 weeks ago

please open a new issue and link to old issues if you think they are related - thanks :)

PS: do you run the woodpecker server on the same host as the arm agent? -> https://woodpecker-ci.org/docs/administration/forges/forgejo#forgejo-on-the-same-host-with-containers

woodpecker-ci / woodpecker