search

dokku / dokku-redis

a redis plugin for dokku
MIT License
254 stars 38 forks source link

Stucked on "Stopping container" #97

Closed enricodeleo closed 6 years ago

enricodeleo commented 6 years ago

I use just one redis container on my dokku machine, and resources in general seem okay but suddenly the service that was using redis stopped working. After a few minute I understood that redis was failing so I thought I could restart the service but it keeps on Stopping container. Even unlinking and destroying fails with the service stucked on Stopping container.

Any ideas?

UPDATE I managed to create another redis service and connect the application to that, but the former container is still not responding and stays on

-----> Deleting appcache
       Deleting container data
-----> Stopping container
josegonzalez commented 6 years ago

What is the output of docker ps -a and dokku redis:info SERVICE?

enricodeleo commented 6 years ago 
root@main-paas:~# docker ps -a
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS               NAMES
96ed391896c5        dokku/platform:latest            "/start web"             21 hours ago        Up 21 hours                             platform.web.2
af583148fd3d        dokku/platform:latest            "/start web"             21 hours ago        Up 21 hours                             platform.web.1
2ef4582ed9f5        redis:3.2.8                      "docker-entrypoint.s…"   21 hours ago        Up 21 hours         6379/tcp            dokku.redis.platformcache
1cd27949b5e5        dokku/appunlimited:latest        "/start web"             12 days ago         Up 12 days                              appunlimited:latest.web.1
7ebddb239dec        dokku/blog:latest   "/start web"             12 days ago         Up 12 days                              blog:latest.web.1
63633fd63f6e        dokku/publishing:latest          "/start web"             12 days ago         Up 12 days                              publishing.web.1
c7c4965f11ac        6f3624f0a9aa                     "/start web"             12 days ago         Up 12 days                              platform.web.2.1518722804
677243f1ac46        6f3624f0a9aa                     "/start web"             12 days ago         Up 12 days                              platform.web.1.1518722296
e1d0b758797e        mongo:3.2.9                      "/entrypoint.sh mong…"   12 days ago         Up 12 days          27017/tcp           dokku.mongo.appunlimited
69e9aa45ad8b        mongo:3.2.9                      "/entrypoint.sh mong…"   12 days ago         Up 12 days          27017/tcp           dokku.mongo.publishing
0b43bcafab32        redis:3.2.8                      "docker-entrypoint.s…"   12 days ago         Up 12 days          6379/tcp            dokku.redis.publicache
3eb15bac5d91        redis:3.2.8                      "docker-entrypoint.s…"   12 days ago         Up 12 days          6379/tcp            dokku.redis.defaultcache
27e86e0e8673        mongo:3.2.9                      "/entrypoint.sh mong…"   12 days ago         Up 12 days          27017/tcp           dokku.mongo.blog
fd278168bbd7        d854edd99cfd                     "/start web"             12 days ago         Created                                 serene_bose
a182f8a6b187        6cf48f1219ed                     "/start web"             12 days ago         Created                                 brave_jones
b78fa5639b22        c95935c8d6da                     "/start web"             12 days ago         Created                                 fervent_borg
5c5c3475f89a        7a216d230d3b                     "/start web"             12 days ago         Created                                 priceless_stonebraker
3bb345321225        05501b7ccdda                     "/start web"             12 days ago         Created                                 gracious_borg

and 

root@main-paas:~# dokku redis:info defaultcache
=====> Container Information
       Config dir:          /var/lib/dokku/services/redis/defaultcache/config
       Data dir:            /var/lib/dokku/services/redis/defaultcache/data
       Dsn:                 redis://defaultcache:e5a848220e5095698d8726b5859950da347caacad48c11eaabf169aab0a8a101@dokku-redis-defaultcache:6379
       Exposed ports:       -
       Id:                  3eb15bac5d91f93ed235d6d18ebafae0e6ba09630543550c01b40cc20fbbd09d
       Internal ip:         172.17.0.3
       Links:               -
       Service root:        /var/lib/dokku/services/redis/defaultcache
       Status:              running
       Version:             redis:3.2.8
josegonzalez commented 6 years ago

For the redis that wont be destroyed, can you run:

dokku --trace --force redis:destroy SERVICE

And gist the output?

enricodeleo commented 6 years ago

It still stays on stop

root@main-paas:~# dokku --trace --force redis:destroy defaultcache
+ export DOKKU_DISTRO
++ . /etc/os-release
++ echo ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args redis:destroy defaultcache
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ [[ false == \t\r\u\e ]]
+ case "$arg" in
+ next_index=2
+ for arg in '"$@"'
+ [[ false == \t\r\u\e ]]
+ case "$arg" in
+ next_index=3
+ return 0
+ args=("$@")
+ skip_arg=false
+ [[ redis:destroy =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
+ [[ /dev/pts/0 == \n\o\t\ \a\ \t\t\y ]]
+ return 0
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
+ [[ redis:destroy =~ ^plugin:.* ]]
+ [[ redis:destroy == \s\s\h\-\k\e\y\s\:\a\d\d ]]
+ [[ -n '' ]]
+ dokku_auth redis:destroy defaultcache
+ declare 'desc=calls user-auth plugin trigger'
+ export SSH_USER=root
+ SSH_USER=root
+ export SSH_NAME=default
+ SSH_NAME=default
+ plugn trigger user-auth root default redis:destroy defaultcache
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ redis:destroy =~ (ssh-keys:add|ssh-keys:remove) ]]
+ return 0
+ case "$1" in
+ execute_dokku_cmd redis:destroy defaultcache
+ declare 'desc=executes dokku sub-commands'
+ local PLUGIN_NAME=redis:destroy
+ local PLUGIN_CMD=redis:destroy
+ local implemented=0
+ local script
+ argv=("$@")
+ local argv
+ case "$PLUGIN_NAME" in
++ readlink -f /var/lib/dokku/plugins/enabled/redis
+ [[ /var/lib/dokku/plugins/available/redis == *core-plugins* ]]
+ [[ -x /var/lib/dokku/plugins/enabled/redis:destroy/subcommands/default ]]
+ [[ -x /var/lib/dokku/plugins/enabled/redis:destroy/subcommands/redis:destroy ]]
+ [[ -x /var/lib/dokku/plugins/enabled/redis/subcommands/destroy ]]
+ [[ -n destroy ]]
+ /var/lib/dokku/plugins/enabled/redis/subcommands/destroy redis:destroy defaultcache
+ source /var/lib/dokku/plugins/enabled/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++++ dirname /var/lib/dokku/plugins/enabled/redis/subcommands/destroy
+++ cd /var/lib/dokku/plugins/enabled/redis/subcommands
+++ pwd
++ dirname /var/lib/dokku/plugins/enabled/redis/subcommands
+ source /var/lib/dokku/plugins/enabled/redis/functions
++++ dirname /var/lib/dokku/plugins/enabled/redis/functions
+++ cd /var/lib/dokku/plugins/enabled/redis
+++ pwd
++ source /var/lib/dokku/plugins/enabled/redis/config
+++ export REDIS_IMAGE=redis
+++ REDIS_IMAGE=redis
+++ export REDIS_IMAGE_VERSION=3.2.8
+++ REDIS_IMAGE_VERSION=3.2.8
+++ export REDIS_ROOT=/var/lib/dokku/services/redis
+++ REDIS_ROOT=/var/lib/dokku/services/redis
+++ export PLUGIN_COMMAND_PREFIX=redis
+++ PLUGIN_COMMAND_PREFIX=redis
+++ export PLUGIN_CONFIG_ROOT=/var/lib/dokku/config/redis
+++ PLUGIN_CONFIG_ROOT=/var/lib/dokku/config/redis
+++ export PLUGIN_DATA_ROOT=/var/lib/dokku/services/redis
+++ PLUGIN_DATA_ROOT=/var/lib/dokku/services/redis
+++ PLUGIN_DATASTORE_PORTS=(6379)
+++ export PLUGIN_DATASTORE_PORTS
+++ export PLUGIN_DATASTORE_WAIT_PORT=6379
+++ PLUGIN_DATASTORE_WAIT_PORT=6379
+++ export PLUGIN_DEFAULT_ALIAS=REDIS
+++ PLUGIN_DEFAULT_ALIAS=REDIS
+++ export PLUGIN_ALT_ALIAS=DOKKU_REDIS
+++ PLUGIN_ALT_ALIAS=DOKKU_REDIS
+++ export PLUGIN_IMAGE=redis
+++ PLUGIN_IMAGE=redis
+++ export PLUGIN_IMAGE_VERSION=3.2.8
+++ PLUGIN_IMAGE_VERSION=3.2.8
+++ export PLUGIN_SCHEME=redis
+++ PLUGIN_SCHEME=redis
+++ export PLUGIN_SERVICE=Redis
+++ PLUGIN_SERVICE=Redis
+++ export PLUGIN_VARIABLE=REDIS
+++ PLUGIN_VARIABLE=REDIS
+++ export PLUGIN_BASE_PATH=/var/lib/dokku/plugins
+++ PLUGIN_BASE_PATH=/var/lib/dokku/plugins
+++ [[ -n 1 ]]
+++ export PLUGIN_BASE_PATH=/var/lib/dokku/plugins/enabled
+++ PLUGIN_BASE_PATH=/var/lib/dokku/plugins/enabled
+++ [[ -d /var/lib/dokku/services/redis/* ]]
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++++ dirname /var/lib/dokku/plugins/enabled/redis/functions
+++ cd /var/lib/dokku/plugins/enabled/redis
+++ pwd
++ source /var/lib/dokku/plugins/enabled/redis/common-functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
++ source /var/lib/dokku/plugins/enabled/common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
++ source /var/lib/dokku/plugins/available/config/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+++ source /var/lib/dokku/core-plugins/available/common/functions
++++ set -eo pipefail
++++ [[ -n 1 ]]
++++ set -x
++ [[ -f /var/lib/dokku/plugins/available/docker-options/functions ]]
++ source /var/lib/dokku/plugins/available/docker-options/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+++ source /var/lib/dokku/core-plugins/available/common/functions
++++ set -eo pipefail
++++ [[ -n 1 ]]
++++ set -x
+++ AVAILABLE_PHASES=(build deploy run)
+ service-destroy-cmd redis:destroy defaultcache
+ declare 'desc=delete the Redis service/data/container if there are no links left'
+ argv=("$@")
+ local cmd=redis:destroy argv
+ [[ redis:destroy == \r\e\d\i\s\:\d\e\s\t\r\o\y ]]
+ shift 1
+ declare SERVICE=defaultcache FORCE_FLAG=
+ [[ -z defaultcache ]]
+ verify_service_name defaultcache
+ declare 'desc=Verifies that a service exists'
+ declare SERVICE=defaultcache
+ [[ ! -n defaultcache ]]
+ [[ ! -d /var/lib/dokku/services/redis/defaultcache ]]
+ return 0
+ SERVICE_ROOT=/var/lib/dokku/services/redis/defaultcache
+ LINKS_FILE=/var/lib/dokku/services/redis/defaultcache/LINKS
++ get_service_name defaultcache
++ declare 'desc=Retrieves a docker service label'
++ declare SERVICE=defaultcache
++ echo dokku.redis.defaultcache
+ SERVICE_NAME=dokku.redis.defaultcache
+ [[ -s /var/lib/dokku/services/redis/defaultcache/LINKS ]]
+ [[ '' == \f\o\r\c\e ]]
+ [[ '' == \-\f ]]
+ [[ '' == \-\-\f\o\r\c\e ]]
+ [[ -z 1 ]]
+ dokku_log_info1 'Deleting defaultcache'
+ declare 'desc=log info1 formatter'
+ echo '-----> Deleting defaultcache'
-----> Deleting defaultcache
++ docker ps -aq -f name=dokku.redis.defaultcache
+ [[ -n 3eb15bac5d91 ]]
+ dokku_log_verbose_quiet 'Deleting container data'
+ declare 'desc=log verbose formatter (with quiet option)'
+ [[ -z '' ]]
+ echo '       Deleting container data'
       Deleting container data
+ service_stop defaultcache
+ declare 'desc=Stops a running service'
+ declare SERVICE=defaultcache
+ local SERVICE_ROOT=/var/lib/dokku/services/redis/defaultcache
++ get_service_name defaultcache
++ declare 'desc=Retrieves a docker service label'
++ declare SERVICE=defaultcache
++ echo dokku.redis.defaultcache
+ local SERVICE_NAME=dokku.redis.defaultcache
++ docker ps -f status=running
++ grep -e 'dokku.redis.defaultcache$'
++ awk '{print $1}'
+ local ID=3eb15bac5d91
+ [[ -z 3eb15bac5d91 ]]
+ [[ -n 3eb15bac5d91 ]]
+ dokku_log_info1_quiet 'Stopping container'
+ declare 'desc=log info1 formatter (with quiet option)'
+ [[ -z '' ]]
+ echo '-----> Stopping container'
-----> Stopping container
+ docker stop dokku.redis.defaultcache
josegonzalez commented 6 years ago

What is the output of

docker stop 3eb15bac5d91
enricodeleo commented 6 years ago

This command has no output. Stays on

root@main-paas:~# docker stop 3eb15bac5d91
josegonzalez commented 6 years ago

Right so this sounds like a docker issue.

What happens on docker rm -f 3eb15bac5d91?

enricodeleo commented 6 years ago

Still no output :(

josegonzalez commented 6 years ago

Alright, I would file an issue in docker, not much I can do here about a stuck container. Maybe check docker logs?

Alternatively, you can restart your server. Feel free to log into our slack chat if you have any issues with that.

Sorry I wasn't much help otherwise :(

enricodeleo commented 6 years ago

Thank you anyway. I'd rather prefer not to reboot the machine since the last time it happened (2 weeks ago apparently my host stopped replying to requests) I had to re-start each service manually leading to a service interruption.