Container does not stop

[kroese]

Describe the bug When stopping the container in Docker, it does not stop gracefully. It is killed by Docker using a SIGTERM signal after the timeout (by default 10 seconds). Even you set the timeout to 240 for example, it will not quit untill its hardkilled.

Version 0.42.3

To Reproduce

Steps to reproduce the behavior:

1. Stop the container
2. Wait forever

Expected behavior Container stopping gracefully

[dgtlmoon]

Not enough information

platform? how are you stopping it exactly? what commands? what version docker?

[kroese]

@dgtlmoon

Ubuntu version 23.04, Docker version 24.0.2

Im just using the normal docker stop command.

[kroese]

@dgtlmoon I really dont understand why you closed this issue so quickly. I didnt mention the platform because I expect that this issue can be reproduced on any platform / docker version, so it didnt seem very relevant.

But now I added them I would appreciate if you re-open it.

[dgtlmoon]

@kroese thanks, since I wrote the software I would be perfectly entitled to ask exactly which platform and commands you ran, this is completely normal in any software project

[dgtlmoon]

I ran docker stop myinstance and it stopped after 10seconds or so, it needs time to cleanup and be sure it doesnt corrupt your JSON db. Ubuntu version 22 tho.

[kroese]

@dgtlmoon If you run:

docker stop -t=240 myinstance

to increase the maximum timeout from 10 to 240 seconds, you will see that it does not stop at all until the 240 seconds are over.

So ofcourse it will need some time to shutdown, but that it stops after 10 seconds is not because it stops by itself but because Docker just kills the process because it is taking too long. This can cause all kinds of corruptions on files that are open.

So my guess is that the container does not handle the stop signal that Docker sends to it.

A quick search revealed that in your Python script you must do something like this:

#!/usr/bin/env python3

import signal

def handler(signum, frame):
    print('Exiting')
    exit(0)

if __name__ == '__main__':
    signal.signal(signal.SIGTERM, handler)
    while True:
        pass

But I have very little experience with Python, so I'll leave the exact fix up to you.

[dgtlmoon]

A quick search revealed that in your Python script you must do something like this:

We already do that...

please attach logs of the container (as a text file)

[dgtlmoon]

You need to attach more information including your logs, I am not sitting next to you, so I need as much debug info as possible

furthermore, i cannot reproduce this locally

[kroese]

The problem is that it does not write anything to the log while shutting down?

For example, when the last two lines are:

Saving JSON..
Saving JSON..

And I then stop the container, no new lines are added at all to indicate its shutting down. (It would be handy if it at least printed some debug information that it received the shutdown signal.)

Then when I start the container again:

Watching: 1f3a61aa-f9d2-4b2a-8aab-da25f5e6eb92 https://xxxxx.xx
Watching: 68d424b7-9d5b-402c-8e95-a52c7e446621 https://xxxxx.xx
Saving JSON..
(7) wsgi starting up on http://0.0.0.0:5000
System env MINIMUM_SECONDS_RECHECK_TIME 20

So I have no information in the container logs about the shutdown process. Is there any flag I can add to enable more verbose / debug logging?

Furthermore I noticed when looking at the active processes in the container that the script is active twice instead of once:

root | 419240 | 419220 | 0 | 19:23 | ? | 00:00:01 | python ./changedetection.py -d /datastore
root | 419273 | 419240 | 0 | 19:23 | ? | 00:00:00 | python ./changedetection.py -d /datastore

I dont know if this normal behaviour, just wanted to mention it.

When I look at the source code it has a SIGCHLD handler, but its call is commented out as far as I can tell. And no handler for SIGSTOP that Docker sends.

[kroese]

Any updates about this issue? Because I'm still experiencing it daily. Just a few questions:

• When you stop the container, what lines do you see in the log indicating it received a signal and quits gracefully?

• You said it took about 10 seconds to stop. This is exactly the same duration as when Docker will kill it forcefully. So even if its true that it stops by itself, its still very dangerous to take so long because its right on the border where it will be terminated.

• I run about 75 different containers. None of them has this issue, and I can reproduce it every time. Why do you keep the issue closed and what info do you need to look into it?

[dgtlmoon]

@kroese hello, unless you have a commercial support contract with me, I do not need to be forced to answer your questions for free - please read http://web.archive.org/web/20230602030423/https://joeldenning.com/?p=29

The following boundaries exist to try to help me continue existing in open source without becoming overwhelmed. It is my responsibility to set the boundaries, your responsibility to learn them and try to follow them, and my responsibility to ultimately enforce them if you do not (block).

Me having boundaries does not mean I don’t like you. It also doesn’t mean I would never want to talk to you. I may have linked you to this article without you having violated any boundaries.

You are not entitled to a conversation with me. I do not need to respond to you. You are not entitled to me doing work. I do not need to do the work you think is important. You are not entitled to any time frame. If your production environment is down, it’s not my fault. If I don’t respond to you in a day or a week or a month, I still owe you nothing.

thank you

I am ofcourse open to commercial arrangements

[kroese]

IMHO I am not asking for support at all. I just took the effort to notify you about a serious flaw in the code.

The only reason I started to ask questions is that you immediately closed the issue and said you could not reproduce it, so I was just trying to proof to you that this bug exists. And now you want me to pay you to help you fix problems in your code?

I should not have asked for updates, as you decide your own priorities ofcourse. I just wanted to bring attention to this issue because its status is "closed" so it might fly under the radar. Anyway, I will consider it as "wontfix" from now on, and don't botter you again.

[dgtlmoon]

I'm happy to keep the issue bubbling around in the background, but continuously pushing someone for an answer, when they've provided that software to you for free is not the answer

[dgtlmoon]

https://github.com/dgtlmoon/changedetection.io/pull/2014