Open drewwells opened 1 month ago
A simple docker kill should work
% docker run -d --rm --name postgres-kill -e POSTGRES_PASSWORD=1234 postgres
bffde2d477d889715f4344d90e62b0bad023ac94b6bd1ea6a262814340d86dfd
% docker kill postgres-kill
postgres-kill
or
% docker run -d --rm -e POSTGRES_PASSWORD=1234 postgres
5eb35e90b2cdd5c5245765032351a4ab3c21c40c96cea150d0c4c3178ea6f280
% docker kill 5eb35e90b2cdd5c5245765032351a4ab3c21c40c96cea150d0c4c3178ea6f280
Did you time it? It takes 10 seconds and that error appears in docker daemon. It seems like the image is trapping the signal and not passing it along to postgres
I am not using a signal, just forcing the container to be destroyed
% docker run -d --rm --name postgres-kill -e POSTGRES_PASSWORD=1234 postgres
time docker kill postgres-kill
aee271f8ce85eff0b80bdb64ba4db83a1571c17bd963b664ca6365dc8296cf8f
postgres-kill
docker kill postgres-kill 0.01s user 0.01s system 17% cpu 0.091 total
Something else is going on here, b/c that takes 10 seconds for me and for Jenkins
time docker kill $(docker run -d --rm -e POSTGRES_PASSWORD=1234 postgres)
da2e3db6490ee63454d3f92862c4ef6548467db2adab9ff76ad9e517735a5257
docker kill $(docker run -d --rm -e POSTGRES_PASSWORD=1234 postgres) 0.00s user 0.01s system 0% cpu 10.223 total
My best guess would be the recent AppArmor snafu where Docker itself got denied from sending signals to runc.
10 seconds is the default --stop-timeout, maybe try setting it to a low value? For example:
docker run -d --rm --stop-timeout 1 --name postgres -e POSTGRES_PASSWORD=1234 postgres
The stop-timeout flag is ignored here
/usr/bin/docker run --stop-timeout 1 --stop-signal SIGQUIT -e POSTGRES_PASSWORD=passyBAucNQ -e POSTGRES_USER=userNFX2wAo -e POSTGRES_DB=testdb bitnami/postgresql:latest
time docker kill 7c0a6
7c0a6
docker kill 7c0a6 0.01s user 0.00s system 0% cpu 10.225 total
Daemon logs
Sep 06 06:55:05 amazon dockerd[3878]: time="2024-09-06T06:55:05.568721579-05:00" level=error msg="
Container failed to exit within 10s of kill - trying direct SIGKILL" container=7c0a6b98107f4cc9ff59f5132c584fa9947bdb11ea0efdd82019e7726dca3e12 error="context deadline exceeded"
The stop-timeout flag is ignored here
Unless you have an ancient Docker engine, I doubt that it doesn't support it:
Option | Default | Description |
---|---|---|
--stop-timeout | API 1.25+ Timeout (in seconds) to stop a container |
If that's being ignored, how do we know that the rest, for example --stop-signal
, is not being ignored?
But most importantly, what you seem to be using is the bitnami/postgresql image, where's this is the postgres image repo. Maybe try asking there for help?
Yeah stop timeout has been around for a long time, so I don't think its my daemon being out of date. Running latest ubuntu. Here's the same issue with this postgres container.
I wouldn't be opposed to some daemon flags to fix this, Im just not clear why the kill command is being ignored.
/usr/bin/docker run --stop-timeout 1 --stop-signal SIGQUIT -e POSTGRES_PASSWORD=passyBAucNQ -e POSTGRES_USER=userNFX2wAo -e POSTGRES_DB=testdb postgres:latest
time docker kill b84f8db782bc
b84f8db782bc
docker kill b84f8db782bc 0.01s user 0.00s system 0% cpu 10.264 total
Here's my server version. I've been able to reproduce this on OS X and linux
Server:
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.22.2
Git commit: 24.0.7-0ubuntu4.1
Built: Fri Aug 9 02:33:20 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.12
GitCommit:
runc:
Version: 1.1.12-0ubuntu3.1
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
--stop-timeout
only applies to docker stop
-- docker kill
bypasses that completely, and what you're seeing is something else entirely (again, my best guess is AppArmor blocking sending the signal to runc
, but it's hard to say for sure -- perhaps dmesg
has more useful clues?)
See https://github.com/moby/moby/blob/96898c8be6103ba65f3096782cb9bca65701bf1c/daemon/kill.go#L161 for where that 10s docker kill
timeout is hard-coded into the daemon itself (specifically because this is an operation that shouldn't fail unless something is going horribly wrong, like AppArmor blocking the signal from dockerd
/containerd
to runc
completely).
I think you're right about apparmor, I see a bunch of references to postgres getting a docker-default apparmor policy. Ill try to leave this open until I figure out how to disable apparmor blocking SIGINT,SIGKILL,etc
then close this ticket.
See https://github.com/moby/moby/pull/47749 for links to the specific issue I'm talking about
I'm spinning up postgres containers for unit tests, so I don't care at all if they are corrupted on exit. I have been unable to find any way to kill these in less than 10 seconds.
Here is what I have tried and has not worked.
docker rm -f {pid}
10 secondsdocker kill --signal SIGQUIT {pid}
10 seconds. Actually SIGQUIT will give a permission error heredocker stop -t 1 --sginal SIGQUIT {pid}
10 seconds. Is this timeout even respected?I see these errors in the docker daemon:
How do I terminate these containers immediately?