search

dokku / dokku

A docker-powered PaaS that helps you build and manage the lifecycle of applications
https://dokku.com
MIT License
29.27k stars 1.92k forks source link

"ConcurrentMigrationError" with 2 or more web instances #7242

Closed dirkzado closed 1 month ago

dirkzado commented 1 month ago

Description of problem

It seems when you have scaled apps like dokku ps:scale app_name web=2 worker=4 then you have 2 web instances running (on a server). If you now deploy a new version of the app and you have database migrations it seems that the migrations run twice and throw this error: ActiveRecord::ConcurrentMigrationError: (ActiveRecord::ConcurrentMigrationError) ! Cannot run migrations because another migration process is currently running. If I only have 1 web instance everything is fine (because the migration only runs once).

Am I using something wrong or could it be a bug?

Or should I use something like dokku ps:scale app_name web=2 worker=4 release=1 ?

Steps to reproduce

run a deploy with database migrations and have at least two web instances running

dokku report $APP_NAME


dokku report app_name

-----> uname: Linux v220231043610241132 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
                      total        used        free      shared  buff/cache   available
       Mem:           15988        1139       10290          21        4558       14489
       Swap:              0           0           0
-----> disk utilization:
       Filesystem      Size  Used Avail Use% Mounted on
       /dev/vda3       314G   36G  266G  12% /
       /dev/vda2       974M  242M  665M  27% /boot
-----> disk inode utilization:
       Filesystem     Inodes IUsed IFree IUse% Mounted on
       /dev/vda3         20M  1.4M   19M    7% /
       /dev/vda2         64K   320   64K    1% /boot
-----> docker version:
       Client: Docker Engine - Community
        Version:           27.3.1
        API version:       1.47
        Go version:        go1.22.7
        Git commit:        ce12230
        Built:             Fri Sep 20 11:41:00 2024
        OS/Arch:           linux/amd64
        Context:           default

       Server: Docker Engine - Community
        Engine:
         Version:          27.3.1
         API version:      1.47 (minimum version 1.24)
         Go version:       go1.22.7
         Git commit:       41ca978
         Built:            Fri Sep 20 11:41:00 2024
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.7.22
         GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
        runc:
         Version:          1.1.14
         GitCommit:        v1.1.14-0-g2c9f560
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info:
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
       Client: Docker Engine - Community
        Version:    27.3.1
        Context:    default
        Debug Mode: true
        Plugins:
         buildx: Docker Buildx (Docker Inc.)
           Version:  v0.17.1
           Path:     /usr/libexec/docker/cli-plugins/docker-buildx
         compose: Docker Compose (Docker Inc.)
           Version:  v2.29.7
           Path:     /usr/libexec/docker/cli-plugins/docker-compose

       Server:
        Containers: 3
         Running: 3
         Paused: 0
         Stopped: 0
        Images: 5
        Server Version: 27.3.1
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Using metacopy: false
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
        runc version: v1.1.14-0-g2c9f560
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: builtin
         cgroupns
        Kernel Version: 5.15.0-122-generic
        Operating System: Ubuntu 22.04.5 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 6
        Total Memory: 15.61GiB
        Name: v220231043610241132
        ID: ef68bd34-3262-44b4-bbc1-6f2378b7d580
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
         File Descriptors: 44
         Goroutines: 57
         System Time: 2024-10-07T12:13:28.318191205+02:00
         EventsListeners: 1
        Username: docker-images
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false

-----> herokuish version:
 !     Herokuish image gliderlabs/herokuish:latest-24 is not available
-----> dokku version: dokku version 0.35.4
-----> dokku-event-listener version: v0.17.0
-----> dokku-update version: dokku-update v0.9.5
-----> docker-container-healthchecker version: v0.11.3
-----> docker-image-labeler version: v0.8.0
-----> git version: git version 2.34.1
-----> lambda-builder version:        v0.8.0
-----> netrc version: v0.10.0
 !     pack binary is not available
-----> plugn version: plugn: v0.16.0
-----> sigil version: v0.11.0
-----> sshcommand version: sshcommand v0.18.1
-----> dokku plugins:
         00_dokku-standard    0.35.4 enabled    dokku core standard plugin
         20_events            0.35.4 enabled    dokku core events logging plugin
         app-json             0.35.4 enabled    dokku core app-json plugin
         apps                 0.35.4 enabled    dokku core apps plugin
         builder              0.35.4 enabled    dokku core builder plugin
         builder-dockerfile   0.35.4 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.35.4 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.35.4 enabled    dokku core builder-lambda plugin
         builder-nixpacks     0.35.4 enabled    dokku core builder-nixpacks plugin
         builder-null         0.35.4 enabled    dokku core builder-null plugin
         builder-pack         0.35.4 enabled    dokku core builder-pack plugin
         buildpacks           0.35.4 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.35.4 enabled    dokku core caddy-vhosts plugin
         certs                0.35.4 enabled    dokku core certificate management plugin
         checks               0.35.4 enabled    dokku core checks plugin
         common               0.35.4 enabled    dokku core common plugin
         config               0.35.4 enabled    dokku core config plugin
         cron                 0.35.4 enabled    dokku core cron plugin
         docker-options       0.35.4 enabled    dokku core docker-options plugin
         domains              0.35.4 enabled    dokku core domains plugin
         enter                0.35.4 enabled    dokku core enter plugin
         git                  0.35.4 enabled    dokku core git plugin
         haproxy-vhosts       0.35.4 enabled    dokku core haproxy-vhosts plugin
         logs                 0.35.4 enabled    dokku core logs plugin
         network              0.35.4 enabled    dokku core network plugin
         nginx-vhosts         0.35.4 enabled    dokku core nginx-vhosts plugin
         openresty-vhosts     0.35.4 enabled    dokku core openresty-vhosts plugin
         plugin               0.35.4 enabled    dokku core plugin plugin
         ports                0.35.4 enabled    dokku core ports plugin
         proxy                0.35.4 enabled    dokku core proxy plugin
         ps                   0.35.4 enabled    dokku core ps plugin
         registry             0.35.4 enabled    dokku core registry plugin
         repo                 0.35.4 enabled    dokku core repo plugin
         resource             0.35.4 enabled    dokku core resource plugin
         run                  0.35.4 enabled    dokku core run plugin
         scheduler            0.35.4 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.35.4 enabled    dokku core scheduler-docker-local plugin
         scheduler-k3s        0.35.4 enabled    dokku core scheduler-k3s plugin
         scheduler-null       0.35.4 enabled    dokku core scheduler-null plugin
         shell                0.35.4 enabled    dokku core shell plugin
         ssh-keys             0.35.4 enabled    dokku core ssh-keys plugin
         storage              0.35.4 enabled    dokku core storage plugin
         trace                0.35.4 enabled    dokku core trace plugin
         traefik-vhosts       0.35.4 enabled    dokku core traefik-vhosts plugin
=====> app_name app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:
=====> app_name app information
       App created at:                1721504947
       App deploy source:             docker-image
       App deploy source metadata:    dockerimages/app_name:app_name
       App dir:                       /home/dokku/app_name
       App locked:                    true
=====> app_name builder information
       Builder build dir:
       Builder computed build dir:
       Builder computed selected:
       Builder global build dir:
       Builder global selected:
       Builder selected:
=====> app_name builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile
       Builder dockerfile global dockerfile path: Dockerfile
       Builder dockerfile dockerfile path:
=====> app_name builder-herokuish information
       Builder herokuish computed allowed: true
       Builder herokuish global allowed: true
       Builder herokuish allowed:
=====> app_name builder-lambda information
       Builder lambda computed lambdayml path: lambda.yml
       Builder lambda global lambdayml path: lambda.yml
       Builder lambda lambdayml path:
=====> app_name builder-nixpacks information
       Builder nixpacks computed nixpackstoml path: nixpacks.toml
       Builder nixpacks global nixpackstoml path: nixpacks.toml
       Builder nixpacks nixpackstoml path:
       Builder nixpacks computed no cache: false
       Builder nixpacks global no cache: false
       Builder nixpacks no cache:
=====> app_name builder-pack information
       Builder pack computed projecttoml path: project.toml
       Builder pack global projecttoml path: project.toml
       Builder pack projecttoml path:
=====> app_name buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-24
       Buildpacks global stack:
       Buildpacks list:
       Buildpacks stack:
=====> app_name caddy information
       Caddy image:                   lucaslorentz/caddy-docker-proxy:2.9
       Caddy letsencrypt email:
       Caddy letsencrypt server:      https://acme-v02.api.letsencrypt.org/directory
       Caddy log level:               ERROR
       Caddy polling interval:        5s
       Caddy tls internal:            false
=====> app_name ssl information
       Ssl dir:                       /home/dokku/app_name/tls
       Ssl enabled:                   true
       Ssl hostnames:                 *.in.app_name.eu *.app_name.eu app_name.eu
       Ssl expires at:                Nov 20 07:18:50 2024 GMT
       Ssl issuer:                    C = US, O = Let's Encrypt, CN = R10
       Ssl starts at:                 Aug 22 07:18:51 2024 GMT
       Ssl subject:                   subject=CN = app_name.eu
       Ssl verified:                  self signed
=====> app_name checks information
       Checks disabled list:          none
       Checks skipped list:           none
       Checks computed wait to retire: 10
       Checks global wait to retire:  60
       Checks wait to retire:         10
=====> app_name docker options information
       Docker options build:
       Docker options deploy:         --restart=on-failure:10
       Docker options run:
=====> app_name domains information
       Domains app enabled:           true
       Domains app vhosts:            app_name.eu *.app_name.eu
       Domains global enabled:        true
       Domains global vhosts:         app_name.eu
=====> app_name git information
       Git deploy branch:             master
       Git global deploy branch:      master
       Git keep git dir:              false
       Git rev env var:               GIT_REV
       Git sha:                       46b7d126bdbee736f8e1acd321beff90ee49299f
       Git source image:              dockerimages/app_name:app_name
       Git last updated at:           1728295978
=====> app_name haproxy information
       Haproxy image:                 byjg/easy-haproxy:4.4.0
       Haproxy letsencrypt email:
       Haproxy letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Haproxy log level:             ERROR
=====> app_name logs information
       Logs computed max size:        1000m
       Logs global max size:          10m
       Logs global vector sink:
       Logs max size:                 1000m
       Logs vector global image:      timberio/vector:0.41.1-debian
       Logs vector sink:
=====> app_name network information
       Network attach post create:
       Network attach post deploy:
       Network bind all interfaces:          false
       Network computed attach post create:
       Network computed attach post deploy:
       Network computed bind all interfaces: false
       Network computed initial network:
       Network computed tld:
       Network global attach post create:
       Network global attach post deploy:
       Network global bind all interfaces:   false
       Network global initial network:
       Network global tld:
       Network initial network:
       Network static web listener:
       Network tld:
       Network web listeners:                172.17.0.4:5000
=====> app_name nginx information
       Nginx access log format:
       Nginx computed access log format:
       Nginx global access log format:
       Nginx access log path:
       Nginx computed access log path: /var/log/nginx/app_name-access.log
       Nginx global access log path:  /var/log/nginx/app_name-access.log
       Nginx bind address ipv4:
       Nginx computed bind address ipv4:
       Nginx global bind address ipv4:
       Nginx bind address ipv6:
       Nginx computed bind address ipv6: ::
       Nginx global bind address ipv6: ::
       Nginx client max body size:
       Nginx computed client max body size: 1m
       Nginx global client max body size: 1m
       Nginx disable custom config:
       Nginx computed disable custom config: false
       Nginx global disable custom config: false
       Nginx error log path:
       Nginx computed error log path: /var/log/nginx/app_name-error.log
       Nginx global error log path:   /var/log/nginx/app_name-error.log
       Nginx hsts include subdomains:
       Nginx computed hsts include subdomains: true
       Nginx global hsts include subdomains: true
       Nginx hsts max age:
       Nginx computed hsts max age:   15724800
       Nginx global hsts max age:     15724800
       Nginx hsts preload:
       Nginx computed hsts preload:   false
       Nginx global hsts preload:     false
       Nginx hsts:
       Nginx computed hsts:           true
       Nginx global hsts:             true
       Nginx last visited at:
       Nginx nginx conf sigil path:
       Nginx computed nginx conf sigil path: nginx.conf.sigil
       Nginx global nginx conf sigil path: nginx.conf.sigil
       Nginx proxy buffer size:
       Nginx computed proxy buffer size: 4k
       Nginx global proxy buffer size: 4k
       Nginx proxy buffering:
       Nginx computed proxy buffering: on
       Nginx global proxy buffering:  on
       Nginx proxy buffers:
       Nginx computed proxy buffers:  8 4k
       Nginx global proxy buffers:    8 4k
       Nginx proxy busy buffers size:
       Nginx computed proxy busy buffers size: 8k
       Nginx global proxy busy buffers size: 8k
       Nginx proxy read timeout:
       Nginx computed proxy read timeout: 60s
       Nginx global proxy read timeout: 60s
       Nginx underscore in headers:
       Nginx computed underscore in headers: off
       Nginx global underscore in headers: off
       Nginx x forwarded for value:
       Nginx computed x forwarded for value: $remote_addr
       Nginx global x forwarded for value: $remote_addr
       Nginx x forwarded port value:
       Nginx computed x forwarded port value: $server_port
       Nginx global x forwarded port value: $server_port
       Nginx x forwarded proto value:
       Nginx computed x forwarded proto value: $scheme
       Nginx global x forwarded proto value: $scheme
       Nginx x forwarded ssl:
       Nginx computed x forwarded ssl:
       Nginx global x forwarded ssl:
=====> app_name openresty information
       Openresty access log format:
       Openresty access log path:     /var/log/nginx/app_name-access.log
       Openresty allowed letsencrypt domains func base64: cmV0dXJuIHRydWUK
       Openresty bind address ipv4:
       Openresty bind address ipv6:   ::
       Openresty client max body size:
       Openresty error log path:      /var/log/nginx/app_name-error.log
       Openresty global hsts:         true
       Openresty computed hsts:       true
       Openresty hsts:
       Openresty hsts include subdomains: true
       Openresty hsts max age:        15724800
       Openresty hsts preload:        false
       Openresty image:               dokku/openresty-docker-proxy:0.9.1
       Openresty letsencrypt email:
       Openresty letsencrypt server:  https://acme-v02.api.letsencrypt.org/directory
       Openresty proxy buffer size:   4k
       Openresty proxy buffering:     on
       Openresty proxy buffers:       8 4k
       Openresty proxy busy buffers size: 8k
       Openresty proxy read timeout:  60s
       Openresty underscore in headers: off
       Openresty x forwarded for value: $remote_addr
       Openresty x forwarded port value: $server_port
       Openresty x forwarded proto value: $scheme
       Openresty x forwarded ssl:
=====> app_name ports information
       Ports map:
       Ports map detected:            http:80:5000 https:443:5000
=====> app_name proxy information
       Proxy computed type:           nginx
       Proxy enabled:                 true
       Proxy global type:             nginx
       Proxy type:                    nginx
=====> app_name ps information
       Deployed:                      true
       Processes:                     2
       Ps can scale:                  true
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:
       Ps restart policy:             on-failure:10
       Restore:                       true
       Running:                       true
       Status web 1:                  running (CID: 2f1c34a2a80)
       Status worker 1:               running (CID: a4573501916)
=====> app_name registry information
       Registry computed image repo:        dokku/app_name
       Registry computed push on release:   false
       Registry computed server:
       Registry global image repo template:
       Registry global push on release:
       Registry global server:
       Registry image repo:
       Registry push extra tags:
       Registry push on release:
       Registry server:
       Registry tag version:
=====> app_name resource information
       _default_ limit memory:          4096
=====> app_name scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:
=====> app_name scheduler-docker-local information
       Scheduler docker local init process: true
       Scheduler docker local parallel schedule count: 8
=====> app_name scheduler-k3s information
       Scheduler k3s computed deploy timeout:       300s
       Scheduler k3s computed image pull secrets:
       Scheduler k3s computed letsencrypt server:   prod
       Scheduler k3s computed namespace:            default
       Scheduler k3s computed rollback on failure:  false
       Scheduler k3s deploy timeout:
       Scheduler k3s global deploy timeout:         300s
       Scheduler k3s global image pull secrets:
       Scheduler k3s global ingress class:          nginx
       Scheduler k3s global kube context:
       Scheduler k3s global kubeconfig path:        /etc/rancher/k3s/k3s.yaml
       Scheduler k3s global letsencrypt email prod:
       Scheduler k3s global letsencrypt email stag:
       Scheduler k3s global letsencrypt server:     prod
       Scheduler k3s global namespace:              default
       Scheduler k3s global network interface:      eth0
       Scheduler k3s global rollback on failure:    false
       Scheduler k3s image pull secrets:
       Scheduler k3s letsencrypt server:
       Scheduler k3s namespace:
       Scheduler k3s rollback on failure:
=====> app_name storage information
       Storage build mounts:
       Storage deploy mounts:
       Storage run mounts:
=====> app_name traefik information
       Traefik api enabled:           false
       Traefik api vhost:             traefik.dokku.me
       Traefik basic auth password:
       Traefik basic auth username:
       Traefik dashboard enabled:     false
       Traefik image:                 traefik:3.1.4
       Traefik letsencrypt email:
       Traefik letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Traefik log level:             ERROR
       Traefik http entry point:      http
       Traefik https entry point:     https

Additional information

No response

Output of failing commands after running: dokku trace:off

No response

Output of failing commands after running: dokku trace:on

No response

josegonzalez commented 1 month ago

I would suggest not running migrations on web process start. What you want to do is use a deployment task, which you can configure either in your Procfile or an app.json file in the root of your repo. If you place it in your Procfile, you don't need to scale that release process up, Dokku will just pull the command automatically.

Hope that helps.

dirkzado commented 1 month ago

Thank you for your quick answer! I already use Procfile with "release" in it. But somehow I miss something. I will investigate ;)