dokku / dokku-redis

a redis plugin for dokku
MIT License
256 stars 39 forks source link

Destroying an Exposed Service #136

Closed kbuilds closed 4 years ago

kbuilds commented 5 years ago

Description of problem

Using dokku version 0.13.4, when I create a new service from a plugin (i.e. redis, postgres), and expose a port on that service, I cannot destroy that service without first unexposing the service. Fails with a docker error.

How reproducible

Seems to happen every time.

Steps to Reproduce

Create a new service

dokku postgres:create test_db

Expose that service

dokku postgres:expose test_db

Try to destroy the service

dokku postgres:destroy test_db

=====> Deleting test_db
       Deleting container data
 !     Service is already stopped
       Removing container
Error response from daemon: You cannot remove a running container 405e265a6508cf9dbd5d57929a5d85ec298b695f00bf15d9fd9c59d35051a460. Stop the container before attempting removal or force remove

Actual Results

The container is not removed. It appears that there is a second container which is created to expose the service to the outside world.

Expected Results

The container should be removed, along with the container which proxies the connection into the service container.

Environment Information

Ubuntu Server 16.04

dokku report APP_NAME output

-----> uname: Linux ip-172-31-4-136 4.4.0-1074-aws #84-Ubuntu SMP Thu Dec 6 08:57:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
-----> memory: 
                     total        used        free      shared  buff/cache   available
       Mem:          15583        4327        1265         148        9990       10724
       Swap:             0           0           0
-----> docker version: 
       Client:
        Version:           18.09.0
        API version:       1.39
        Go version:        go1.10.4
        Git commit:        4d60db4
        Built:             Wed Nov  7 00:48:57 2018
        OS/Arch:           linux/amd64
        Experimental:      false

       Server: Docker Engine - Community
        Engine:
         Version:          18.09.0
         API version:      1.39 (minimum version 1.12)
         Go version:       go1.10.4
         Git commit:       4d60db4
         Built:            Wed Nov  7 00:16:44 2018
         OS/Arch:          linux/amd64
         Experimental:     false
-----> docker daemon info: 
       Containers: 4
        Running: 4
        Paused: 0
        Stopped: 0
       Images: 12
       Server Version: 18.09.0
       Storage Driver: overlay2
        Backing Filesystem: extfs
        Supports d_type: true
        Native Overlay Diff: true
       Logging Driver: json-file
       Cgroup Driver: cgroupfs
       Plugins:
        Volume: local
        Network: bridge host macvlan null overlay
        Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
       Swarm: inactive
       Runtimes: runc
       Default Runtime: runc
       Init Binary: docker-init
       containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
       runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
       init version: fec3683
       Security Options:
        apparmor
        seccomp
         Profile: default
       Kernel Version: 4.4.0-1074-aws
       Operating System: Ubuntu 16.04.5 LTS
       OSType: linux
       Architecture: x86_64
       CPUs: 4
       Total Memory: 15.22GiB
       Name: ip-172-31-4-136
       ID: VPMU:KL3M:TJYM:CXB4:5CHX:5J4V:7DII:UFBV:LQRM:GBUO:EAMW:B5B5
       Docker Root Dir: /var/lib/docker
       Debug Mode (client): true
       Debug Mode (server): false
       Registry: https://index.docker.io/v1/
       Labels:
       Experimental: false
       Insecure Registries:
        127.0.0.0/8
       Live Restore Enabled: false
       Product License: Community Engine

WARNING: No swap limit support
-----> sigil version: 0.4.0
-----> herokuish version: 
       herokuish: 0.4.6
       buildpacks:
         heroku-buildpack-multi     v1.0.0
         heroku-buildpack-ruby      v197
         heroku-buildpack-nodejs    v134
         heroku-buildpack-clojure   v82
         heroku-buildpack-python    v146
         heroku-buildpack-java      v62
         heroku-buildpack-gradle    v27
         heroku-buildpack-scala     v83
         heroku-buildpack-play      v26
         heroku-buildpack-php       v148
         heroku-buildpack-go        v98
         buildpack-nginx            v10
-----> dokku version: 0.13.4
-----> dokku plugins: 
       plugn: 0.3.0
         00_dokku-standard    0.13.4 enabled    dokku core standard plugin
         20_events            0.13.4 enabled    dokku core events logging plugin
         apps                 0.13.4 enabled    dokku core apps plugin
         build-env            0.13.4 enabled    dokku core build-env plugin
         certs                0.13.4 enabled    dokku core certificate management plugin
         checks               0.13.4 enabled    dokku core checks plugin
         common               0.13.4 enabled    dokku core common plugin
         config               0.13.4 enabled    dokku core config plugin
         docker-options       0.13.4 enabled    dokku core docker-options plugin
         domains              0.13.4 enabled    dokku core domains plugin
         enter                0.13.4 enabled    dokku core enter plugin
         git                  0.13.4 enabled    dokku core git plugin
         logs                 0.13.4 enabled    dokku core logs plugin
         mongo                1.4.10 enabled    dokku mongo service plugin
         network              0.13.4 enabled    dokku core network plugin
         nginx-vhosts         0.13.4 enabled    dokku core nginx-vhosts plugin
         plugin               0.13.4 enabled    dokku core plugin plugin
         postgres             1.4.12 enabled    dokku postgres service plugin
         proxy                0.13.4 enabled    dokku core proxy plugin
         ps                   0.13.4 enabled    dokku core ps plugin
         rabbitmq             1.4.10 enabled    dokku rabbitmq service plugin
         redis                1.4.11 enabled    dokku redis service plugin
         repo                 0.13.4 enabled    dokku core repo plugin
         scheduler-docker-local 0.13.4 enabled    dokku core scheduler-docker-local plugin
         shell                0.13.4 enabled    dokku core shell plugin
         ssh-keys             0.13.4 enabled    dokku core ssh-keys plugin
         storage              0.13.4 enabled    dokku core storage plugin
         tags                 0.13.4 enabled    dokku core tags plugin
         tar                  0.13.4 enabled    dokku core tar plugin

How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

Installed from the install script on the main documentation site.

Moved from https://github.com/dokku/dokku/issues/3399

josegonzalez commented 5 years ago

What is that container? Is that the proxy container? Can you run:

dokku --force --trace postgres:destroy SERVICE

On a new, exposed service and gist the output?

kbuilds commented 5 years ago

https://gist.github.com/kbuilds/3031543e6a8c52494e41f043c33075c8

josegonzalez commented 5 years ago

What is the output of docker inspect 85818d0c36fd10d446fe3a6b9dcf9c818efe307e511388c7e4d334322a53aeff?

kbuilds commented 5 years ago

https://gist.github.com/kbuilds/578b9f625e8fe70f3c17689e5dc6de20

josegonzalez commented 5 years ago

Okay the heuristic for a stopped container here is wrong:

docker ps -f status=running --no-trunc

I'll see if I have time to investigate later.

kbuilds commented 5 years ago

@josegonzalez Thanks for taking the time to look into this.

In the meantime, a workaround I have been using is to simply unexpose the service:

dokku postgres:unexpose test_db

then destroy it

dokku postgres:destroy test_db

josegonzalez commented 4 years ago

This should have been fixed in https://github.com/dokku/dokku-redis/commit/28ff40a507afd43aac96be100cdb2e71208686c4 (and related commits for each service plugin). Example output:

vagrant@dokku:~$ dokku postgres:create test_db
       Waiting for container to be ready
       Creating container database
       Securing connection to database
=====> Postgres container created: test_db
=====> Container Information
       Config dir:          /var/lib/dokku/services/postgres/test_db/config
       Data dir:            /var/lib/dokku/services/postgres/test_db/data
       Dsn:                 postgres://postgres:65d8d7b206747bba0102b4d9a0b7be24@dokku-postgres-test-db:5432/test_db
       Exposed ports:       -
       Id:                  ca887889ba8e6173180c4bfa8bc1a7fe37338b25b208459d771100fd55a7398d
       Internal ip:         172.17.0.2
       Links:               -
       Service root:        /var/lib/dokku/services/postgres/test_db
       Status:              running
       Version:             postgres:11.6
vagrant@dokku:~$ dokku postgres:expose test_db
-----> Service test_db exposed on port(s) [container->host]: 5432->32403
vagrant@dokku:~$ dokku postgres:destroy test_db
 !     WARNING: Potentially Destructive Action
 !     This command will destroy test_db Postgres service.
 !     To proceed, type "test_db"

> test_db
=====> Deleting test_db
=====> Stopping container
       Container stopped
       Removing container
       Removing data
=====> Postgres container deleted: test_db
vagrant@dokku:~$