[Never finishes] Waiting for container to be ready

[UlTriX]

With latest version of dokku and dokku-rabbitmq the rabbitmq:create action never finishes.

dokku rabbitmq:create xxxx -----> Starting container Waiting for container to be ready

[josegonzalez]

Can you pull the latest version of dokkupaas/wait and try again? I think you may have an older version of that image.

[crm114]

Getting the same error:

Using default tag: latest
latest: Pulling from dokkupaas/wait
Status: Image is up to date for dokkupaas/wait:latest

The service works however.

[UlTriX]

I can confirm I am using the latest image of dokkupaas/wait By forcing a ctrl ^c it finishes the creation and as @crm114 said the service works.

^C=====> RabbitMQ container created: app-rabbitmq

[josegonzalez]

Can you try creating the service after running dokku trace on?

[UlTriX]

Sure.

`+ echo '-----> Starting container' -----> Starting container

• [[ -n RABBITMQ_SSL_CERT_FILE=/var/lib/rabbitmq/ssl_files/cert.pem;RABBITMQ_SSL_KEY_FILE=/var/lib/rabbitmq/ssl_files/key.pem;RABBITMQ_SSL_CA_FILE=/var/lib/rabbitmq/ssl_files/cacert.pem ]]
• echo 'RABBITMQ_SSL_CERT_FILE=/var/lib/rabbitmq/ssl_files/cert.pem;RABBITMQ_SSL_KEY_FILE=/var/lib/rabbitmq/ssl_files/key.pem;RABBITMQ_SSL_CA_FILE=/var/lib/rabbitmq/ssl_files/cacert.pem'
• tr ';' '\n' ++ get_service_name test-rabbitmq ++ local SERVICE=test-rabbitmq ++ echo dokku.rabbitmq.test-rabbitmq
• SERVICE_NAME=dokku.rabbitmq.test-rabbitmq ++ docker run --name dokku.rabbitmq.test-rabbitmq -v /var/lib/dokku/services/rabbitmq/test-rabbitmq/data:/var/lib/rabbitmq --hostname test-rabbitmq -e RABBITMQ_DEFAULT_USER=test-rabbitmq -e RABBITMQ_DEFAULT_PASS=b5980d7592621303fca55d944707ad63 -e RABBITMQ_DEFAULT_VHOST=test-rabbitmq --env-file=/var/lib/dokku/services/rabbitmq/test-rabbitmq/ENV -d --restart always --label dokku=service --label dokku.service=rabbitmq rabbitmq:3.6.0-management
• ID=d6d89f6921561a1bc3d0dcfc7b3ee744a915532d6d4ee332185b8e3764ed3a71
• echo d6d89f6921561a1bc3d0dcfc7b3ee744a915532d6d4ee332185b8e3764ed3a71
• dokku_log_verbose_quiet 'Waiting for container to be ready'
• [[ -z '' ]]
• echo ' Waiting for container to be ready' Waiting for container to be ready
• docker run --rm --link dokku.rabbitmq.test-rabbitmq:rabbitmq dokkupaas/wait`

[gaumire]

TL;DR

The first exposed port is inaccessible. I am not exactly sure why, and because it is inaccessible, netcat couldn't connect to it and dokkupaas/wait waits forever. Though the correct thing to do would be to find out why rabbitmq's ssl listeners are not working, the below tweak however does tend to work.

If you want to try, use this wait image at dockerhub, replacing dokkupaas/wait here with gaumire/wait.

I was facing the same issue ... the container waited indefinitely to be ready. And a ^C finishes the creation. Further diagnosing I found,

$ docker run --rm --link dokku.rabbitmq.testapp-rabbitmq:rabbitmq dokkupaas/wait
waiting for TCP connection to 172.17.0.3:5671.....^C
dokku@ip-10-90-153-190:~$ 

I wondered if that port should be 5672 instead of 5671, testing with nmap on the container's IP address I found that 5671 was closed while 5672 was open. So I dug further and discovered that here, the list of port was being retrieved as below

5671
5672
4369
15671
15672
25672

And as we were taking the first exposed port, and IMO as that port (5671) was inaccessible, netcat never got to connect to that port and thus it was stuck at "Waiting for container to be ready". So I just sorted out the ports in $port and now it seems to work.

[josegonzalez]

What is the difference between the two images?

[josegonzalez]

Note: from reading your comments, sounds like maybe the rabbitmq container was built incorrectly?

[gaumire]

Yes that is one possibility, it does expose the port 5671 but there aren't any services running on it. All my image does is sorts the ports out returning 4369, which is actually serving requests unlike 5671. I see that 3.6.0-management's Dockerfile is the one that's exposing that port.

Also seeing the comment from @UlTriX , there are some certs mentioned, I checked with my installation and those directories don't exist in the container.

[josegonzalez]

Maybe it should be possible to specify the port to listen to instead of sorting. That way we can listen to the proper port. Sorting seems like a hack that could break in the future.

[gaumire]

I agree. In that case, I guess we can use port 5672 as it's the default port. If anyone would ever want to use some non default port, they are better off building their own version of the plugin :-)

[josegonzalez]

Yeah. Can you fork your wait container thing so that it takes an optionally specified port somehow?

[gaumire]

Will do. But coming to think of it, I believe we may actually go on to using the wait container in other plugins too. So was thinking why not iterate over all the ports, if there are more than one, and see if any of the ports can be connected to. If there is just one port exposed, the current behavior applies. I will implement this behavior in my fork and see how this turns out.

Sent from my iPhone

[josegonzalez]

I don't know if I like that as much, but willing to see a pr to our repo.

[gaumire]

@josegonzalez I figured as docker-wait is used in other of dokku plugins too, rather than specifying a specific port, a timeout option would be more convenient.

[rafaferrari]

Getting the same error:

-----> Starting container Waiting for container to be ready

By forcing a "CTRL+C" it finishes the creation and the service works.

^C=====> RabbitMQ container created: rabbitmq-dev DSN: amqp://rabbitmq-dev:cf2fbf07567bd37adabd99414bdd09db@dokku-rabbitmq-rabbitmq-dev:5672/rabbitmq-dev

The service works however.

[matthewmueller]

yah this happened to me too on dokku 0.4.14, same solution though :-)

[bashu]

same problem here

[josegonzalez]

Closed by 314b9f5711d8b0ed6334960573ec193cbf396d72. For existing installs, please pull down the latest dokkupaas/wait image, or at least dokkupaas/wait:0.2.