A way to restore service container

ypadlyak commented 3 years ago

Hi @josegonzalez

Is there a way to restore the service container, which I've lost when switching the docker storage driver? Something like ps:rebuild for the apps.

Currently, it fails because the container is gone: $ dokku ps:rebuild t*g -----> Cleaning up... -----> Building ticketco-blog from herokuish... -----> Checking for pre-build script in hooks -----> Adding BUILD_ENV to build environment... -----> Adding shared deployment-keys to build environment ... -----> Adding host-keys to build environment ... Adding shared keys Error response from daemon: could not get container for **: No such container: **** Error response from daemon: page not found Error: failed to start containers: Error: No such container: 2021/10/07 16:54:49 exit status 1 2021/10/07 16:54:49 exit status 1

josegonzalez commented 3 years ago

Can you include all of the information we ask for in our issue template? It is helpful for debugging problems. Thanks.

ypadlyak commented 3 years ago

It is not the issue, but the question about functionality. Ok, I will consider updating it, please close it for now if you decide so

josegonzalez commented 3 years ago

I'd like to know what version you are using because that will impact the answer... The easiest way is for you to fill out the issue template.

ypadlyak commented 3 years ago

I see, ok

ypadlyak commented 3 years ago

Description of problem

When dokku mysql service loses a related docker container for any reason, it is impossible to restore it without creating a new service

How reproducible

Every time

Steps to Reproduce

Create test service
remove related docker container
try restarting it

Actual Results

it fails with the error about non-esisting service container

Expected Results

It might try to restore the container or suggest how to restore it. If not, there are could be a command to restore it, as containers are kind of ephemeral thing in my mind, so it might be useful

Environment Information

docker version 19.03.11 dokku version 0.21.4 dokku mysql service plugin 1.11.5

How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

AWS EC2 instance

josegonzalez commented 3 years ago

Can you check the output of docker ps -a and also the output of git -C /var/lib/dokku/plugins/enabled/mysql show? I recently committed something that may help fix this but I wonder if you are using it/if the container exists in a weird way that is blocking it from working...

ypadlyak commented 3 years ago

$ git -C /var/lib/dokku/plugins/enabled/mysql show
commit 296607be93b7ed71eedb3286c50ce6c0e8ab0b20 (HEAD -> master, origin/master, origin/HEAD)
Merge: 2405b7e 49c00a0
Author: Jose Diaz-Gonzalez <email@josediazgonzalez.com>
Date:   Sat May 16 04:52:46 2020 -0400

   Merge pull request #117 from dokku/configurable-config

   fix: allow config directory to be configurable 

$ docker ps -a
CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS                      PORTS                     NAMES
dd5bd16b1c6d        bbed8b023ff5                            "/bin/bash -c 'cat >…"   57 minutes ago      Exited (0) 57 minutes ago                             zealous_goodall
af25e6e858bb        aaa391a12c5c                            "/bin/bash -c 'cat >…"   57 minutes ago      Exited (0) 57 minutes ago                             funny_carson
cc3cc14bdaf4        2c3a6682a752                            "/bin/bash -c 'mkdir…"   57 minutes ago      Exited (0) 57 minutes ago                             vigorous_lichterman
95d7e60f64b9        09bca0f8861e                            "/bin/bash -c 'cat >…"   57 minutes ago      Exited (0) 57 minutes ago                             awesome_dewdney
8841f85a15c9        7358895eadfa                            "/bin/bash -c 'cat >…"   57 minutes ago      Exited (0) 57 minutes ago                             zen_poincare
b0068b59b3f6        3c9eefcd2b28                            "/bin/bash -c 'cat >…"   57 minutes ago      Exited (0) 57 minutes ago                             peaceful_morse
023e47d56668        bdbb930fb142                            "/bin/bash -c 'mkdir…"   57 minutes ago      Exited (0) 57 minutes ago                             jolly_feistel
1d1fb15296ee        ceaebbeeffca                            "/bin/bash -c 'cat >…"   57 minutes ago      Exited (0) 57 minutes ago                             compassionate_thompson
08c42430e90e        7742775ee35d                            "/bin/bash -c 'mkdir…"   57 minutes ago      Exited (0) 57 minutes ago                             compassionate_lovelace
5cc515b91984        gliderlabs/herokuish:latest             "/bin/bash -c 'mkdir…"   57 minutes ago      Exited (0) 57 minutes ago                             loving_wilson
77a843e9f762        dokku/*-staging:latest        "bundle exec unicorn…"   6 hours ago         Up 6 hours                                            sidekiq-web-staging.web.1
8229c06678d3        dokku/*-prod:latest           "bundle exec unicorn…"   6 hours ago         Up 6 hours                                            sidekiq-web-prod.web.1
ab709daa47e5        dokku/*-devc:latest           "bundle exec unicorn…"   6 hours ago         Up 6 hours                                            sidekiq-web-devc.web.1
934111cbedb1        dokku/*-devb:latest           "bundle exec unicorn…"   6 hours ago         Up 6 hours                                            sidekiq-web-devb.web.1
4decda71a409        dokku/*-dev:latest            "bundle exec unicorn…"   6 hours ago         Up 6 hours                                            *-dev.web.1
17d4bc471ef3        dokku*-demo:latest           "bundle exec unicorn…"   6 hours ago         Up 6 hours                                            *-demo.web.1
f1e33c67b84f        dokku/*-prod:latest             "/bin/bash -c 'set -…"   6 hours ago         Exited (0) 6 hours ago                                dazzling_shamir
1317479f0778        dokku/*-prod:latest             "/start web"             6 hours ago         Up 6 hours                                            retwitter-prod.web.1
2c3efe2e76d9        dokku/*-prod:latest             "/bin/bash -c 'find …"   6 hours ago         Exited (0) 6 hours ago                                eloquent_archimedes
17f5619289ac        69e394b2fad7                            "/bin/bash -c 'mkdir…"   6 hours ago         Exited (0) 6 hours ago                                objective_kowalevski
f1aab0f45c02        2cf4bae11a12                            "/bin/bash -c 'mkdir…"   6 hours ago         Exited (0) 6 hours ago                                quizzical_wiles
df84f94b9de8        dokku/*:latest           "/start web"             7 hours ago         Up 7 hours                                            *.web.1
186a5170dc9e        d8d67335bd40                            "/bin/bash -c 'mkdir…"   7 hours ago         Exited (0) 7 hours ago                                serene_sutherland
3e43d06aea1b        825061d04a0f                            "/bin/bash -c 'mkdir…"   7 hours ago         Exited (0) 7 hours ago                                crazy_neumann
9d5ab94f947d        dokku/*:latest                "bundle exec unicorn…"   7 hours ago         Up 7 hours                                            *.web.1
69481d64da89        dokku/*-prod:latest     "/start web"             23 hours ago        Exited (0) 23 hours ago                               *-prod.web.1
63f958532d79        7fe23f487c86                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               suspicious_mclean
3caa5b886745        88691479fe9e                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               modest_haibt
053e5272e842        dokku/*-prod:latest        "bundle exec sidekiq…"   23 hours ago        Exited (0) 23 hours ago                               ticketco-vault-prod.worker.1
69f22d3aae09        dokku/*-testing:latest   "bundle exec sidekiq…"   23 hours ago        Exited (0) 23 hours ago                               *-testing.worker.1
c53808645d56        dokku/*-prod:latest      "bundle exec sidekiq…"   23 hours ago        Exited (0) 23 hours ago                               *-prod.worker.1
306524bac65c        dokku/*:latest     "/start worker"          23 hours ago        Exited (0) 23 hours ago                               *.worker.1
05352fcf158d        dokku/*:latest     "/start web"             23 hours ago        Exited (1) 23 hours ago                               *.web.1
10b43ef5e05b        eba662b47c47                            "/build"                 23 hours ago        Exited (0) 23 hours ago                               focused_carson
b21dd6aa16e5        393cdd49f540                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               pedantic_hodgkin
7be348f09aaf        674dd64407c0                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               angry_darwin
fd1f793f6ca2        8a0943dde5aa                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               beautiful_maxwell
56a468da658c        149fb002ba89                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               epic_heisenberg
138f41cecec4        75b8279e634f                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               amazing_rhodes
ff5d503f4cdf        ee774b02744c                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               festive_driscoll
90bb3ea5df24        340e798fd570                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               suspicious_grothendieck
46c6ac994b00        310f3f64d784                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               tender_brahmagupta
872f7c081c3a        9ad3993da1d8                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               nostalgic_lichterman
e901d0cd57ff        524ec706ff74                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               priceless_noyce
4a37839939be        90bae2e9a796                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               angry_mccarthy
b2306c2e527e        ed13677e5a61                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               eager_chatelet
a93d6787cffd        1a5d0dc8fae2                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               modest_bhabha
a525a44fb12a        2498102c8436                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               amazing_tesla
14d654940974        gliderlabs/herokuish:latest             "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               stupefied_saha
582e81c655fb        f9123671c048                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               silly_elbakyan
d5e32d817797        031744887b16                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               brave_goldwasser
20e9eb5ad02e        1bb6b12949dc                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               amazing_ardinghelli
5491ea57804b        c5354d4d5ce1                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               great_mendeleev
85f933cf4bde        21fdab522bb7                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               great_goldwasser0
dccf5bbcd5bb        a4504907b2ae                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               peaceful_bell
e64aaf5b6800        gliderlabs/herokuish:latest             "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               priceless_grothendieck
43d76c682f2b        76cd99fe2e09                            "/build"                 23 hours ago        Exited (0) 23 hours ago                               eloquent_roentgen
3bbb3ceb67ba        99848cfe97ba                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               adoring_wilbur
882212d80939        b0c298ca6b3a                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               confident_satoshi
a3a31a852932        7b5f398c29aa                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               romantic_mayer
359ca8779e9e        37a0e7957eaf                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               pedantic_nightingale
ae603c9b5124        ff3d6ec86236                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               confident_mayer
ef4c4a9cc973        c244c12bb2f7                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               dazzling_varahamihira
4c5bb4d7a056        57fb49030c6c                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               upbeat_lamport
e1e30b96fd0e        04fa415690ce                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               eloquent_fermi
64961bfc4325        519761552db7                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               vibrant_lichterman
3fca34bb7abc        gliderlabs/herokuish:latest             "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               wonderful_easley
366ce041d6e9        2cf4bae11a12                            "/bin/bash -c 'set -…"   23 hours ago        Exited (0) 23 hours ago                               agitated_mestorf
5a9ee0d100a0        2cf4bae11a12                            "/bin/bash -c 'find …"   23 hours ago        Exited (0) 23 hours ago                               sharp_dijkstra
0a6691c2a74e        2f7a0606265f                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               nice_jackson
7e11a92e503a        60bc12f53de1                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               silly_colden
a41c3c682651        dokku/*-deva:latest           "bundle exec unicorn…"   23 hours ago        Exited (0) 23 hours ago                               *-deva.web.1
e64a1bf1d5d6        0bed4774adab                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               tender_booth
c095e2b68767        65f7e8463f19                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               sweet_jang
49a86240d689        dokku/*-prod:latest   "bundle exec unicorn…"   23 hours ago        Exited (10) 23 hours ago                              thirsty_knuth
18117d195ab5        0b116d413849                            "/build"                 23 hours ago        Exited (0) 23 hours ago                               admiring_hodgkin
b887d94c6596        d7bc6a903ab0                            "/build"                 23 hours ago        Exited (0) 23 hours ago                               practical_nightingale
0e10e82bf149        5b5fa5472d0d                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               elegant_roentgen
27d89f3a84e6        6e414a7cb28c                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               sharp_saha
b08f6b3b6423        751bccc2f1e8                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               festive_mestorf
48a6c030b454        d04fdcab7597                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               flamboyant_dijkstra
8588a0e11014        224acc97994f                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               hardcore_keldysh
88082c76239c        a074dc075a3d                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               eager_bartik
78f7ad7addaa        8d4b1453dea1                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               modest_benz
400ed91c689e        207287dfe276                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               festive_cartwright
3d7b49201a31        60e3e9605ef0                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               nice_blackburn
efc3b3609425        47a4208dfe6a                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               hungry_fermat
a13cc972ccd6        452eed927641                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               sweet_bose
c36efdedd574        d91bfe2f059f                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               nifty_visvesvaraya
b346f4331c3d        f99ae391f119                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               nervous_diffie
162b0113b6f3        1b329eaa38c5                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               competent_tesla
a93608244ac0        c1ae355f4a40                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               beautiful_germain
3b7e94815550        971fade1650b                            "/bin/bash -c 'cat >…"   23 hours ago        Exited (0) 23 hours ago                               great_newton
369bf0e52528        3752ac44003a                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               festive_shannon
fca35d3ea8f1        04d042799a14                            "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               interesting_wilson
b0f73fc1f11d        gliderlabs/herokuish:latest             "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               bold_brown
3fa86dfdb305        gliderlabs/herokuish:latest             "/bin/bash -c 'mkdir…"   23 hours ago        Exited (0) 23 hours ago                               trusting_leakey
1e368bb96619        gliderlabs/logspout:v3.1                "/bin/logspout syslo…"   23 hours ago        Up 23 hours                 127.0.0.1:18000->80/tcp   logspout

I have masked only project specific names in above aoutput

josegonzalez commented 3 years ago

Running dokku mysql:start $SERVICE should start the container, but your version of the plugin is from two years ago and your version of Dokku is a bit out of date as well so I can't be sure.

ypadlyak commented 3 years ago

So, I've had to: 1) update the plugin to the latest 2) tried starting with no success: $ dokku mysql:start ticketco-blog-database =====> Starting container Waiting for container to be ready ERROR: unable to connect 3) tried starting with --trace and have got the following: try starting - it didn't work, because of conflicting container: ++ docker run --name dokku.mysql.-database -v /var/lib/dokku/services/mysql/-database/data:/var/lib/mysql -v /var/lib/dokku/services/mysql/-database/config:/etc/mysql/conf.d -e MYSQL_ROOT_PASSWORD= -e MYSQL_USER=mysql -e MYSQL_PASSWORD= -e MYSQL_DATABASE=-database --env-file=/var/lib/dokku/services/mysql//ENV -d --restart always --label dokku=service --label dokku.service=mysql mysql:8.0.26 docker: Error response from daemon: Conflict. The container name "/dokku.mysql.-database" is already in use by container "2400eda7e985fdb037e0fd099c328cf3749160c8af6555a2b2b08dc3dcd841ae". You have to remove (or rename) that container to be able to reuse that name.

4) tried removing that container and start again - it had failed. But this time I've spotted in docker logs, it fails because of MySQL version mismatch. It seems like it had lost version information for some reason, as mysql:info showed no version information and it was complaining about missing config: cat: /var/lib/dokku/services/mysql/*-database/CONFIG_OPTIONS: No such file or directory

5) I have set proper version and image with mysql:upgrade -i -I 6) It has started successfully!

I haven't tested the data integrity, but it seems to be fine now! The service image/version issue might be because of the difference in how different plugin versions store that information, I guess.

Thanks a lot for your help!

josegonzalez commented 3 years ago

The plugin:update call should have pinned the version on disk, but the :info command gets the version from the running service. Might be good for me to update that.

That said, I think the stop command should remove the container, and we should add a pause command that does what stop currently does.

ypadlyak commented 3 years ago

I see. Also, I have had to pull image for mysql: manually for step 6, forgot to mention that

josegonzalez commented 3 years ago

Yeah the image has to exist - newer versions of the plugin should complain if it doesn't.

dokku / dokku-mysql