docker stop results in SIGTERM being sent

[matus-m]

Hi,

is there something wrong about using docker stop to shutdown the postgres container? After examining the logs I notice the following:

2016-08-10T17:37:48.649652549Z LOG:  received smart shutdown request
2016-08-10T17:37:48.649779343Z LOG:  autovacuum launcher shutting down
2016-08-10T17:39:23.315340438Z LOG:  database system was interrupted; last known up at 2016-08-10 17:34:54 UTC
2016-08-10T17:39:23.619640237Z LOG:  database system was not properly shut down; automatic recovery in progress
2016-08-10T17:39:23.629918073Z LOG:  redo starts at 0/70D0028
2016-08-10T17:39:23.629963085Z LOG:  invalid record length at 0/70D0108
2016-08-10T17:39:23.629969782Z LOG:  redo done at 0/70D00D0
2016-08-10T17:39:23.636000325Z LOG:  MultiXact member wraparound protections are now enabled
2016-08-10T17:39:23.639092551Z LOG:  database system is ready to accept connections
2016-08-10T17:39:23.640314686Z LOG:  autovacuum launcher started

Even if I run docker stop --time=60 postgres, the stop just waits for those 60 seconds, after which it probably sends SIGKILL causing the postres shutdown and subsequent warning during start.

Thanks for any info

[yosifkit]

I've not seen an issue in stopping a postgres container (though mine don't usually have much data or load) and so mine stop almost instantly after receiving SIGTERM. If you need more control, you could use docker kill -s TERM postgres, then you don't get the automatic KILL signal.

[jjmata]

I experience the same all the time, did you find a better solution @yntelectual?

[rkrzewski]

I had actually experienced data corruption because postgress process was SIGKILLed in the middle of flushing logs. Now, I always make sure to do docker exec -u postgres $CID pg_ctl stop instead of docker stop $CID Solving this correctly would require something along EXITPOINT hook described here https://github.com/docker/docker/issues/6982, but sadly that feature seems to be abandoned.

[tianon]

As referenced above, your best bet is to run docker kill -s TERM xyz directly and skip docker stop entirely (thus avoiding SIGKILL):

I've not seen an issue in stopping a postgres container (though mine don't usually have much data or load) and so mine stop almost instantly after receiving SIGTERM. If you need more control, you could use docker kill -s TERM postgres, then you don't get the automatic KILL signal.

Additionally, you might have luck with specifying --stop-timeout on the container itself to give docker stop a larger default, but I think that'll just be a stop-gap solution if it's taking longer than 60 seconds for your database to shut down. It sounds like you might have some architectural issues you'll want to look into to figure out why it's taking so long for your database to shut down.

In the future, these sorts of questions/requests would be more appropriately posted to the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

[navossoc]

Well, as stated in https://www.postgresql.org/docs/10/static/server-shutdown.html:

Smart Shutdown: ... lets existing sessions end their work normally. It shuts down only after all of the sessions terminate.

So if the connection is idle in another process, 60 seconds may never be enough.

About pg_ctl:

Simply, the “smart” mode has been considered the default because it is the least distuptive, particularly it will wait for a backup to finish before shutting down the server. It has been (justly) discussed that it was not enough aggresive, users being sometimes surprised that a shutdown requested can finish with a timeout because a connection has been for example left open, hence the default has been switched to “fast”.

https://paquier.xyz/postgresql-2/postgres-9-5-feature-highlight-pgctl-default-mode/

Since we don't have a fine-grained shutdown control order in docker I'm using the stop_signal to change the behavior and get a clean shutdown with docker-compose.

postgres:
  image: postgres:10.4-alpine
  container_name: postgres
  stop_signal: SIGINT                 # Fast Shutdown mode

I think you can change the Dockerfile and specify the STOPSIGNAL. It should fix too...

[tianon]

Given that SIGTERM is more graceful (and less likely to break folks' databases), I'm wary of explicitly changing that.

[navossoc]

Sure, this is the safest choice. I'm not saying to change the default signal.

What I really want to say is:

As everyone should know their own application and their workload, that may be the tip someone needs to tweak their needs.