search

dokku / ci-docker-image

A Docker Image meant for use with CI/CD pipelines
MIT License
35 stars 24 forks source link

git push stuck #82

Closed andreroggeri closed 11 months ago

andreroggeri commented 11 months ago

If you're using Dokku - especially for commercial purposes - consider donating to project development via OpenCollective or Patreon. Funds go to general development, support, and infrastructure costs.

If you'd like to sponsor specific functionality, see the project's Sponsoring document.

If you need support for a version of Dokku that is more than a year old, your issue may be closed without an answer. Please upgrade to a recent version before filing an issue.

Description of problem

I'm using the Github Action to deploy my app, and when using it together with a VPN (Wireguard), the git push command hangs with no output (Only when running in GHA inside this container image).

Posting here because the issue only happens inside this image.

How reproducible

100% of the time in GHA, but not locally (Using the same VPN config)

Steps to Reproduce

  1. Configure and connect to VPN
  2. SSH into the GHA runner (Used the VPN for this)
  3. Run docker run --rm -v $(pwd):/app -w /app -i dokku/ci-docker-image:0.7.0
  4. Push into repo git push -v ssh://dokku@10.8.0.11/backend ffbd7b782de0b25d17b273026b54c590b675695f:refs/heads/main <-- Stuck, no output
  5. Do the same outside the container and it works.

Actual Results

Git push command stuck

Expected Results

Successful deploy

Environment Information

Details

ubuntu@agendaodonto-staging:~$ dokku report -----> uname: Linux agendaodonto-staging 5.15.0-1045-oracle #51-Ubuntu SMP Fri Sep 22 10:56:53 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux -----> memory: total used free shared buff/cache available Mem: 11940 4949 5567 44 1423 6724 Swap: 0 0 0 -----> docker version: Client: Docker Engine - Community Version: 24.0.6 API version: 1.43 Go version: go1.20.7 Git commit: ed223bc Built: Mon Sep 4 12:31:57 2023 OS/Arch: linux/arm64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.6 API version: 1.43 (minimum version 1.12) Go version: go1.20.7 Git commit: 1a79695 Built: Mon Sep 4 12:31:57 2023 OS/Arch: linux/arm64 Experimental: false containerd: Version: 1.6.24 GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523 runc: Version: 1.1.9 GitCommit: v1.1.9-0-gccaecfc docker-init: Version: 0.19.0 GitCommit: de40ad0 -----> docker daemon info: Client: Docker Engine - Community Version: 24.0.6 Context: default Debug Mode: true Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.11.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.21.0 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 7 Running: 7 Paused: 0 Stopped: 0 Images: 20 Server Version: 24.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523 runc version: v1.1.9-0-gccaecfc init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-1045-oracle Operating System: Ubuntu 22.04.3 LTS OSType: linux Architecture: aarch64 CPUs: 2 Total Memory: 11.66GiB Name: agendaodonto-staging ID: 16580e0c-186b-4144-ac5c-5674e1337ec5 Docker Root Dir: /var/lib/docker Debug Mode: false File Descriptors: 68 Goroutines: 68 System Time: 2023-10-08T23:24:56.53655536Z EventsListeners: 1 Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false -----> git version: git version 2.34.1 -----> sigil version: 0.10.1build+e443be0 -----> herokuish version: ! herokuish not supported on arm64 architecture -----> dokku version: dokku version 0.31.4 -----> plugn version: plugn: 0.13.0build+fd5297a -----> dokku plugins: 00_dokku-standard 0.31.4 enabled dokku core standard plugin 20_events 0.31.4 enabled dokku core events logging plugin app-json 0.31.4 enabled dokku core app-json plugin apps 0.31.4 enabled dokku core apps plugin builder 0.31.4 enabled dokku core builder plugin builder-dockerfile 0.31.4 enabled dokku core builder-dockerfile plugin builder-herokuish 0.31.4 enabled dokku core builder-herokuish plugin builder-lambda 0.31.4 enabled dokku core builder-lambda plugin builder-null 0.31.4 enabled dokku core builder-null plugin builder-pack 0.31.4 enabled dokku core builder-pack plugin buildpacks 0.31.4 enabled dokku core buildpacks plugin caddy-vhosts 0.31.4 enabled dokku core caddy-vhosts plugin certs 0.31.4 enabled dokku core certificate management plugin checks 0.31.4 enabled dokku core checks plugin common 0.31.4 enabled dokku core common plugin config 0.31.4 enabled dokku core config plugin cron 0.31.4 enabled dokku core cron plugin docker-options 0.31.4 enabled dokku core docker-options plugin domains 0.31.4 enabled dokku core domains plugin enter 0.31.4 enabled dokku core enter plugin git 0.31.4 enabled dokku core git plugin haproxy-vhosts 0.31.4 enabled dokku core haproxy-vhosts plugin letsencrypt 0.20.3 enabled Automated installation of let's encrypt TLS certificates logs 0.31.4 enabled dokku core logs plugin network 0.31.4 enabled dokku core network plugin nginx-vhosts 0.31.4 enabled dokku core nginx-vhosts plugin openresty-vhosts 0.31.4 enabled dokku core openresty-vhosts plugin plugin 0.31.4 enabled dokku core plugin plugin ports 0.31.4 enabled dokku core ports plugin postgres 1.36.0 enabled dokku postgres service plugin proxy 0.31.4 enabled dokku core proxy plugin ps 0.31.4 enabled dokku core ps plugin rabbitmq 1.36.3 enabled dokku rabbitmq service plugin redis 1.37.1 enabled dokku redis service plugin registry 0.31.4 enabled dokku core registry plugin repo 0.31.4 enabled dokku core repo plugin resource 0.31.4 enabled dokku core resource plugin run 0.31.4 enabled dokku core run plugin scheduler 0.31.4 enabled dokku core scheduler plugin scheduler-docker-local 0.31.4 enabled dokku core scheduler-docker-local plugin scheduler-null 0.31.4 enabled dokku core scheduler-null plugin shell 0.31.4 enabled dokku core shell plugin ssh-keys 0.31.4 enabled dokku core ssh-keys plugin storage 0.31.4 enabled dokku core storage plugin trace 0.31.4 enabled dokku core trace plugin traefik-vhosts 0.31.4 enabled dokku core traefik-vhosts plugin

How (deb/make) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

Ansible playbook in OCI

Additional information

ssh -vvv -T dokku@10.8.0.11 OpenSSH_9.1p1, OpenSSL 3.0.7 1 Nov 2022 debug1: Reading configuration data /etc/ssh/ssh_config debug2: resolve_canonicalize: hostname 10.8.0.11 is address debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts' debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2' debug3: ssh_connect_direct: entering debug1: Connecting to 10.8.0.11 [10.8.0.11] port 22. debug3: set_sock_tos: set socket 3 IP_TOS 0x48 debug1: Connection established. debug1: identity file /root/.ssh/id_rsa type -1 debug1: identity file /root/.ssh/id_rsa-cert type -1 debug1: identity file /root/.ssh/id_ecdsa type -1 debug1: identity file /root/.ssh/id_ecdsa-cert type -1 debug1: identity file /root/.ssh/id_ecdsa_sk type -1 debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /root/.ssh/id_ed25519 type -1 debug1: identity file /root/.ssh/id_ed25519-cert type -1 debug1: identity file /root/.ssh/id_ed25519_sk type -1 debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /root/.ssh/id_xmss type -1 debug1: identity file /root/.ssh/id_xmss-cert type -1 debug1: identity file /root/.ssh/id_dsa type -1 debug1: identity file /root/.ssh/id_dsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_9.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.4 debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.4 pat OpenSSH* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to 10.8.0.11:22 as 'dokku' debug1: load_hostkeys: fopen /root/.ssh/known_hosts: No such file or directory debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug3: order_hostkeyalgs: no algorithms matched; accept original debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,zlib@openssh.com,zlib debug2: compression stoc: none,zlib@openssh.com,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,zlib@openssh.com debug2: compression stoc: none,zlib@openssh.com debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com debug1: kex: host key algorithm: ssh-ed25519 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

josegonzalez commented 11 months ago

How am I supposed to replicate your setup? Does the GitHub runner have a vpn connection to the host in question, and have you verified that works?

andreroggeri commented 11 months ago

Yeah, that's working.

I did some additional debugging and this seems to be related to Wireguard and the MTU configuration inside the container (Totally unrelated to this repo).

Updating the MTU config inside the container does fix it, but it's not very usable that way. I'll try some different configurations to see what works and will update here, but there's no action from Dokku.

Useful links

andreroggeri commented 11 months ago

Gosh, network issues are so hard to debug 😭 Turn out I just had to configure the MTU on my VPN server to a lower value, and everything seems to be working 🙏

josegonzalez commented 11 months ago

Maybe a jumbo frames issue?

andreroggeri commented 11 months ago

I believe the actual issue lies in the misconfiguration of MTU between the Wireguard/Docker/WAN interfaces.

But the side effects seem to be well known 😅