search

dokku / dokku

A docker-powered PaaS that helps you build and manage the lifecycle of applications
https://dokku.com
MIT License
29.3k stars 1.92k forks source link

Apps wrongly deployed after host reboot (v0.11.4) #3080

Closed HillLiu closed 6 years ago

HillLiu commented 6 years ago

same issue with https://github.com/dokku/dokku/issues/2999

but happen on v0.11.4, and my environment.

-----> docker version: 
Client:
 Version:   18.02.0-ce
 API version:   1.36
 Go version:    go1.9.3
 Git commit:    fc4de44
 Built: Wed Feb  7 21:16:38 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.02.0-ce
  API version:  1.36 (minimum version 1.12)
  Go version:   go1.9.3
  Git commit:   fc4de44
  Built:    Wed Feb  7 21:15:09 2018
  OS/Arch:  linux/amd64
  Experimental: false
-----> docker daemon info: 
Containers: 51
 Running: 16
 Paused: 0
 Stopped: 35
Images: 159
Server Version: 18.02.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 272
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.13.0-36-generic
Operating System: Ubuntu 17.10
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 581.8MiB
Name: startup
ID: PQLT:L63S:JTZ7:4UOX:YYRM:RT3M:33OS:TYOV:GXFN:YKJS:RR2L:VNCL
Docker Root Dir: /var/lib/docker
Debug Mode (client): true
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

-----> sigil version: 0.4.0
-----> herokuish version: 
herokuish: 0.3.34
buildpacks:
  heroku-buildpack-multi     v1.0.0
  heroku-buildpack-ruby      v173
  heroku-buildpack-nodejs    v116
  heroku-buildpack-clojure   v78
  heroku-buildpack-python    v123
  heroku-buildpack-java      v59
  heroku-buildpack-gradle    v25
  heroku-buildpack-scala     v79
  heroku-buildpack-play      v26
  heroku-buildpack-php       v130
  heroku-buildpack-go        v81
  heroku-buildpack-erlang    fa17af9
  buildpack-nginx            v9
-----> dokku version: 0.11.4
-----> dokku plugins: 
plugn: 0.3.0
  00_dokku-standard    0.11.4 enabled    dokku core standard plugin
  20_events            0.11.4 enabled    dokku core events logging plugin
  apps                 0.11.4 enabled    dokku core apps plugin
  build-env            0.11.4 enabled    dokku core build-env plugin
  certs                0.11.4 enabled    dokku core certificate management plugin
  checks               0.11.4 enabled    dokku core checks plugin
  common               0.11.4 enabled    dokku core common plugin
  config               0.11.4 enabled    dokku core config plugin
  docker-options       0.11.4 enabled    dokku core docker-options plugin
  domains              0.11.4 enabled    dokku core domains plugin
  enter                0.11.4 enabled    dokku core enter plugin
  git                  0.11.4 enabled    dokku core git plugin
  logs                 0.11.4 enabled    dokku core logs plugin
  named-containers     0.11.4 enabled    dokku core named containers plugin
  network              0.11.4 enabled    dokku core network plugin
  nginx-vhosts         0.11.4 enabled    dokku core nginx-vhosts plugin
  plugin               0.11.4 enabled    dokku core plugin plugin
  proxy                0.11.4 enabled    dokku core proxy plugin
  ps                   0.11.4 enabled    dokku core ps plugin
  repo                 0.11.4 enabled    dokku core repo plugin
  shell                0.11.4 enabled    dokku core shell plugin
  ssh-keys             0.11.4 enabled    dokku core ssh-keys plugin
  storage              0.11.4 enabled    dokku core storage plugin
  tags                 0.11.4 enabled    dokku core tags plugin
  tar                  0.11.4 enabled    dokku core tar plugin
josegonzalez commented 6 years ago

Can you give more detail as to:

That will help me replicate the issue so I can fix it.

josegonzalez commented 6 years ago

bump

HillLiu commented 6 years ago

be apologetic to miss this.

What you are doing before you reboot

Just reboot, nothing else.

How you are confirming apps are "wrongly deployed"

I have 5 apps in one dokku, and after reboot these apps were pointed to wrong dns (domain name).

frontend (PHP)

middleware (PHP)

dimension configure server (PHP)

internal proxy (use .static)

static server (PHP)

What your current fix steps are

I create a custom rc.local service, and force each apps restart. 

[Unit]
 Description=/etc/rc.local Compatibility
 ConditionPathExists=/etc/rc.local

[Service]
 Type=forking
 ExecStart=/etc/rc.local start
 TimeoutSec=0
 StandardOutput=tty
 RemainAfterExit=yes
 SysVStartPriority=99

[Install]
 WantedBy=multi-user.target

And the rc.local are

/usr/bin/dokku apps:list | awk 'NR >= 2' | xargs -I{} /usr/bin/dokku ps:restart {}
josegonzalez commented 6 years ago

So we already do a dokku ps:restore on reboot. This will trigger rebuilds for the container ip addresses for all applications - docker doesn't require that they stay the same across reboots - but can take a bit of time.

Can you show the log output of the dokku-redeploy service when you see this issue?

josegonzalez commented 6 years ago

bump

HillLiu commented 6 years ago

@josegonzalez after upgrade to v0.11.6 and install dokku-update. I also remove my rc.local hack. When I try reboot twice, everything working perfect now.

We can close this now.

Following are my reboot logs.

Mar 11 12:13:24 startup dokku[2093]: -----> Restarting in parallel
Mar 11 12:14:03 startup dokku[2093]: App force already running
Mar 11 12:14:03 startup dokku[2093]: -----> Ensuring network configuration is in sync for force
Mar 11 12:14:03 startup dokku[2093]: -----> Configuring force.dokku-srver.xxx.com...(using built-in template)
Mar 11 12:14:03 startup dokku[2093]: -----> Creating http nginx.conf
Mar 11 12:14:03 startup dokku[2093]: -----> Running nginx-pre-reload
Mar 11 12:14:03 startup dokku[2093]:        Reloading nginx
Mar 11 12:17:09 startup dokku[2093]: -----> Releasing homecoming (dokku/homecoming:latest)...
Mar 11 12:17:09 startup dokku[2093]: -----> Deploying homecoming (dokku/homecoming:latest)...
Mar 11 12:17:09 startup dokku[2093]: -----> Attempting to run scripts.dokku.predeploy from app.json (if defined)
Mar 11 12:17:09 startup dokku[2093]: -----> App Procfile file found (/home/dokku/homecoming/DOKKU_PROCFILE)
Mar 11 12:17:09 startup dokku[2093]: -----> DOKKU_SCALE file found (/home/dokku/homecoming/DOKKU_SCALE)
Mar 11 12:17:09 startup dokku[2093]: =====> web=1
Mar 11 12:17:09 startup dokku[2093]: -----> Attempting pre-flight checks
Mar 11 12:17:09 startup dokku[2093]:        For more efficient zero downtime deployments, create a file CHECKS.
Mar 11 12:17:09 startup dokku[2093]:        See http://dokku.viewdocs.io/dokku/deployment/zero-downtime-deploys/ for examples
Mar 11 12:17:09 startup dokku[2093]:        CHECKS file not found in container: Running simple container check...
Mar 11 12:17:09 startup dokku[2093]: -----> Waiting for 10 seconds ...
Mar 11 12:17:09 startup dokku[2093]: -----> Default container check successful!
Mar 11 12:17:09 startup dokku[2093]: -----> Running post-deploy
Mar 11 12:17:09 startup dokku[2093]: -----> Found previous container(s) (bf126c61dae7) named homecoming.web.1
Mar 11 12:17:09 startup dokku[2093]: =====> renaming container (bf126c61dae7) homecoming.web.1 to homecoming.web.1.1520770590
Mar 11 12:17:09 startup dokku[2093]: =====> renaming container (2765edc03b15) gifted_kirch to homecoming.web.1
Mar 11 12:17:09 startup dokku[2093]: -----> Configuring homecoming.dokku-srver.xxx.com...(using built-in template)
Mar 11 12:17:09 startup dokku[2093]: -----> Configuring usw1.middleware.xxx.com...(using built-in template)
Mar 11 12:17:09 startup dokku[2093]: -----> Configuring usw1.warcraft.xxx.com...(using built-in template)
Mar 11 12:17:09 startup dokku[2093]: -----> Creating http nginx.conf
Mar 11 12:17:09 startup dokku[2093]: -----> Running nginx-pre-reload
Mar 11 12:17:09 startup dokku[2093]:        Reloading nginx
Mar 11 12:17:09 startup dokku[2093]: -----> Setting config vars
Mar 11 12:17:09 startup dokku[2093]:        DOKKU_APP_RESTORE:  1
Mar 11 12:17:09 startup dokku[2093]: -----> Attempting to run scripts.dokku.postdeploy from app.json (if defined)
Mar 11 12:17:09 startup dokku[2093]: -----> Shutting down old containers in 60 seconds
Mar 11 12:17:09 startup dokku[2093]: =====> bf126c61dae7a7cc07d1a5be1ac4ddabb81dde3c2e0303315c28b70b1ea1ca9c
Mar 11 12:17:09 startup dokku[2093]: =====> Application deployed:
Mar 11 12:17:09 startup dokku[2093]:        http://homecoming.dokku-srver.xxx.com
Mar 11 12:17:09 startup dokku[2093]:        http://usw1.middleware.xxx.com
Mar 11 12:17:09 startup dokku[2093]:        http://usw1.warcraft.xxx.com
Mar 11 12:17:09 startup dokku[2093]: -----> Ensuring network configuration is in sync for homecoming
Mar 11 12:17:09 startup dokku[2093]: -----> Configuring homecoming.dokku-srver.xxx.com...(using built-in template)
Mar 11 12:17:09 startup dokku[2093]: -----> Configuring usw1.middleware.xxx.com...(using built-in template)
Mar 11 12:17:09 startup dokku[2093]: -----> Configuring usw1.warcraft.xxx.com...(using built-in template)
Mar 11 12:17:09 startup dokku[2093]: -----> Creating http nginx.conf
Mar 11 12:17:09 startup dokku[2093]: -----> Running nginx-pre-reload
Mar 11 12:17:09 startup dokku[2093]:        Reloading nginx
Mar 11 12:17:32 startup dokku[2093]: App ht already running
Mar 11 12:17:32 startup dokku[2093]: -----> Ensuring network configuration is in sync for ht
Mar 11 12:17:32 startup dokku[2093]: -----> Overriding default nginx.conf with detected nginx.conf.sigil
Mar 11 12:17:32 startup dokku[2093]: -----> Configuring ht.dokku-srver.xxx.com...(using app-supplied template)
Mar 11 12:17:32 startup dokku[2093]: -----> Configuring market.prod.usw1.bag.xxx.com...(using app-supplied template)
Mar 11 12:17:32 startup dokku[2093]: -----> Configuring market.prod.usw1.ht.xxx.com...(using app-supplied template)
Mar 11 12:17:32 startup dokku[2093]: -----> Creating http nginx.conf
Mar 11 12:17:32 startup dokku[2093]: -----> Running nginx-pre-reload
Mar 11 12:17:32 startup dokku[2093]:        Reloading nginx
Mar 11 12:17:55 startup dokku[2093]: App static already running
Mar 11 12:17:55 startup dokku[2093]: -----> Ensuring network configuration is in sync for static
Mar 11 12:17:55 startup dokku[2093]: -----> Overriding default nginx.conf with detected nginx.conf.sigil
Mar 11 12:17:55 startup dokku[2093]: -----> Configuring i.xxx.com...(using app-supplied template)
Mar 11 12:17:55 startup dokku[2093]: -----> Configuring static.dokku-srver.xxx.com...(using app-supplied template)
Mar 11 12:17:55 startup dokku[2093]: -----> Creating http nginx.conf
Mar 11 12:17:55 startup dokku[2093]: -----> Running nginx-pre-reload
Mar 11 12:17:55 startup dokku[2093]:        Reloading nginx
Mar 11 12:18:22 startup dokku[2093]: App warcraft already running
Mar 11 12:18:22 startup dokku[2093]: -----> Ensuring network configuration is in sync for warcraft
Mar 11 12:18:22 startup dokku[2093]: -----> Configuring warcraft.dokku-srver.xxx.com...(using built-in template)
Mar 11 12:18:22 startup dokku[2093]: -----> Creating http nginx.conf
Mar 11 12:18:22 startup dokku[2093]: -----> Running nginx-pre-reload
Mar 11 12:18:22 startup dokku[2093]:        Reloading nginx