search

dokku / dokku-pro-issues

Temporary location for issues with installing or using dokku pro
1 stars 0 forks source link

Cannot deploy via git HTTP due to crash / timeout in dokku-pro #10

Closed gamedevsam closed 1 year ago

gamedevsam commented 1 year ago

Description of problem

When I attempt to deploy via git push with HTTP remote, I get this error:

error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: the remote end hung up unexpectedly

It's

Steps to reproduce

  1. Install dokku-pro on Digital Ocean (1CPU + 1GB of Ram + 2GB swap)
  2. Create new app called t-1
  3. Attempt to push to it and witness timeout
samuel@samuel-zorin-ssd:~/Dev/Personal/t-1$ git remote add dokku http://root@---.---.---:5000/@git/t-1
samuel@samuel-zorin-ssd:~/Dev/Personal/t-1$ git push dokku main:main
Password for 'http://root@157.245.94.123:5000': 
Enumerating objects: 491, done.
Counting objects: 100% (491/491), done.
Delta compression using up to 16 threads
Compressing objects: 100% (466/466), done.
Writing objects: 100% (491/491), 6.04 MiB | 11.98 MiB/s, done.
Total 491 (delta 71), reused 0 (delta 0)
remote: Resolving deltas: 100% (71/71), done.
-----> Cleaning up...
-----> Building t-1 from herokuish
-----> Adding BUILD_ENV to build environment...
       BUILD_ENV added successfully
-----> Node.js app detected

-----> Creating runtime environment

       NPM_CONFIG_LOGLEVEL=error
       NODE_VERBOSE=false
       NODE_ENV=production
       NODE_MODULES_CACHE=true

-----> Installing binaries
       engines.node (package.json):  18.x
       engines.npm (package.json):   unspecified (use default)

       Resolving node version 18.x...
       Downloading and installing node 18.15.0...
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
Everything up-to-date

dokku-pro version

1.2.0

dokku report

root@ubuntu-s-1vcpu-1gb-nyc1-01:~# dokku report -----> uname: Linux ubuntu-s-1vcpu-1gb-nyc1-01 5.4.0-146-generic #163-Ubuntu SMP Fri Mar 17 18:26:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux -----> memory: total used free shared buff/cache available Mem: 976 809 73 3 93 45 Swap: 1999 378 1621 -----> docker version: Client: Docker Engine - Community Version: 23.0.2 API version: 1.42 Go version: go1.19.7 Git commit: 569dd73 Built: Mon Mar 27 16:16:18 2023 OS/Arch: linux/amd64 Context: default

   Server: Docker Engine - Community
    Engine:
     Version:          23.0.2
     API version:      1.42 (minimum version 1.12)
     Go version:       go1.19.7
     Git commit:       219f21b
     Built:            Mon Mar 27 16:16:18 2023
     OS/Arch:          linux/amd64
     Experimental:     false
    containerd:
     Version:          1.6.20
     GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
    runc:
     Version:          1.1.5
     GitCommit:        v1.1.5-0-gf19387a
    docker-init:
     Version:          0.19.0
     GitCommit:        de40ad0

-----> docker daemon info: Client: Context: default Debug Mode: true Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.10.4 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.17.2 Path: /usr/libexec/docker/cli-plugins/docker-compose scan: Docker Scan (Docker Inc.) Version: v0.23.0 Path: /usr/libexec/docker/cli-plugins/docker-scan

   Server:
    Containers: 7
     Running: 4
     Paused: 0
     Stopped: 3
    Images: 15
    Server Version: 23.0.2
    Storage Driver: overlay2
     Backing Filesystem: extfs
     Supports d_type: true
     Using metacopy: false
     Native Overlay Diff: true
     userxattr: false
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    Cgroup Version: 1
    Plugins:
     Volume: local
     Network: bridge host ipvlan macvlan null overlay
     Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
    Swarm: inactive
    Runtimes: runc io.containerd.runc.v2
    Default Runtime: runc
    Init Binary: docker-init
    containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
    runc version: v1.1.5-0-gf19387a
    init version: de40ad0
    Security Options:
     apparmor
     seccomp
      Profile: builtin
    Kernel Version: 5.4.0-146-generic
    Operating System: Ubuntu 20.04.6 LTS
    OSType: linux
    Architecture: x86_64
    CPUs: 1
    Total Memory: 976.8MiB
    Name: ubuntu-s-1vcpu-1gb-nyc1-01
    ID: f5eeedc3-4a64-4d27-8a8b-059345a70fca
    Docker Root Dir: /var/lib/docker
    Debug Mode: false
    Registry: https://index.docker.io/v1/
    Experimental: false
    Insecure Registries:
     127.0.0.0/8
    Live Restore Enabled: false

WARNING: No swap limit support -----> git version: git version 2.25.1 -----> sigil version: 0.9.0build+bc921b7 -----> herokuish version: herokuish: v0.5.41 buildpacks: heroku-buildpack-multi v1.2.0 heroku-buildpack-ruby v252 heroku-buildpack-nodejs v203 heroku-buildpack-clojure v90 heroku-buildpack-python v226 heroku-buildpack-java v72 heroku-buildpack-gradle v39 heroku-buildpack-scala v96 heroku-buildpack-play v26 heroku-buildpack-php v231 heroku-buildpack-go v171 heroku-buildpack-nginx v22 buildpack-null v3 -----> dokku version: dokku version 0.30.2 -----> plugn version: plugn: 0.12.0build+3a27594 -----> dokku plugins: 00_dokku-standard 0.30.2 enabled dokku core standard plugin 20_events 0.30.2 enabled dokku core events logging plugin app-json 0.30.2 enabled dokku core app-json plugin apps 0.30.2 enabled dokku core apps plugin builder 0.30.2 enabled dokku core builder plugin builder-dockerfile 0.30.2 enabled dokku core builder-dockerfile plugin builder-herokuish 0.30.2 enabled dokku core builder-herokuish plugin builder-lambda 0.30.2 enabled dokku core builder-lambda plugin builder-null 0.30.2 enabled dokku core builder-null plugin builder-pack 0.30.2 enabled dokku core builder-pack plugin buildpacks 0.30.2 enabled dokku core buildpacks plugin caddy-vhosts 0.30.2 enabled dokku core caddy-vhosts plugin certs 0.30.2 enabled dokku core certificate management plugin checks 0.30.2 enabled dokku core checks plugin common 0.30.2 enabled dokku core common plugin config 0.30.2 enabled dokku core config plugin cron 0.30.2 enabled dokku core cron plugin docker-options 0.30.2 enabled dokku core docker-options plugin domains 0.30.2 enabled dokku core domains plugin enter 0.30.2 enabled dokku core enter plugin git 0.30.2 enabled dokku core git plugin haproxy-vhosts 0.30.2 enabled dokku core haproxy-vhosts plugin logs 0.30.2 enabled dokku core logs plugin network 0.30.2 enabled dokku core network plugin nginx-vhosts 0.30.2 enabled dokku core nginx-vhosts plugin plugin 0.30.2 enabled dokku core plugin plugin postgres 1.33.0 enabled dokku postgres service plugin proxy 0.30.2 enabled dokku core proxy plugin ps 0.30.2 enabled dokku core ps plugin registry 0.30.2 enabled dokku core registry plugin repo 0.30.2 enabled dokku core repo plugin resource 0.30.2 enabled dokku core resource plugin run 0.30.2 enabled dokku core run plugin scheduler 0.30.2 enabled dokku core scheduler plugin scheduler-docker-local 0.30.2 enabled dokku core scheduler-docker-local plugin scheduler-null 0.30.2 enabled dokku core scheduler-null plugin shell 0.30.2 enabled dokku core shell plugin ssh-keys 0.30.2 enabled dokku core ssh-keys plugin storage 0.30.2 enabled dokku core storage plugin trace 0.30.2 enabled dokku core trace plugin traefik-vhosts 0.30.2 enabled dokku core traefik-vhosts plugin

Additional information

This appears to be an issue with the git pre-push hook or something. The actual build continues, as my dropplet's CPU usage remains high even after the git push command errors out.

josegonzalez commented 1 year ago

Does this reproduce on that server with an ssh push instead of http(s) push? If it does, it's probably not a Dokku Pro issue (though I can certainly help in either case).

gamedevsam commented 1 year ago

I don't know, I wouldn't pay the hefty pricetag for dokku-pro if I wanted to deal with setting up all my SSH keys correctly for all the devices I use. The Git HTTP / HTTPS server is a big part of the reason I purchased dokku-pro.

I was able to get around the issue however, the problem was that the default values for SERVER_WRITE_TIMEOUT and SERVER_READ_TIMEOUT in /etc/default/dokku-pro were configured too low and that caused the server to reboot before the push / deployment completed.

I set these two values to 1 hour and now things are working better:

SERVER_WRITE_TIMEOUT=3600
SERVER_READ_TIMEOUT=3600
josegonzalez commented 1 year ago

Gotcha no worries. Was a bit late when I asked so my fault here!

The settings you set were meant exactly for this situation. On larger servers, there is more progress more often, so the server doesn't disconnect for inactivity.

I'm going to increase that default timeout for the next release to something more reasonable. Thanks for reporting!

gamedevsam commented 1 year ago

This issue seems more like an opportunity to improve on the docs, as I suspect most people are probably not going to run dokku pro in VPS types of environments, but I found the instructions on setting up swap memory very useful, so it might be worth calling out these two vars there.