hilbert / hilbert-docker-images

Application with a dynamic choice of docker containers to run
Apache License 2.0
22 stars 8 forks source link

Shutdown of containers takes too long #61

Open porst17 opened 6 years ago

porst17 commented 6 years ago

I did a bit of research why certain calls to docker-compose stop/kill are so slow. The main reason seems to be related to the SIGTERM it sends to the container's main process (given in the CMD or ENTRYPOINT of the Dockerfile). As soon as this process exits, the container will shut down. If it doesn't exit, a timeout applies (default: 10s per container).

However, often processes do not properly quit on a SIGTERM. python does nothing by default (that's what I read) and there are also problems in bash scripts when you use subprocesses. Not sure what Electron does.

So I assume the problem would go away if all our containers properly handle SIGTERM. The first step would be to check which containers are causing problems and then try to fix it.

(@malex984 I know you are using SIGINT explicitly, but AFAIK it is only suitable for programs running in a terminal)

porst17 commented 6 years ago

I just ran kill against an Electron 0.37 process and it did quit properly. Nevertheless, if it is wrapped into several bash scripts this might still cause problems.

elondaits commented 6 years ago

Some extra info:

https://docs.docker.com/compose/faq/#why-do-my-services-take-10-seconds-to-recreate-or-stop

From the article above:

Make sure you’re using the JSON form of CMD and ENTRYPOINT in your Dockerfile. For example use ["program", "arg1", "arg2"] not "program arg1 arg2". Using the string form causes Docker to run your process using bash which doesn’t handle signals properly. Compose always uses the JSON form, so don’t worry if you override the command or entrypoint in your Compose file.

Also if it's not possible to handle SIGTERM:

Set the stop_signal to a signal which the application knows how to handle:

web:
build: .
stop_signal: SIGINT

Or

If you can’t modify the application, wrap the application in a lightweight init system (like s6) or a signal proxy (like dumb-init or tini). Either of these wrappers take care of handling SIGTERM properly.