Open Sh4d1 opened 3 years ago
zq-david-wang
commented
3 years ago I had similar issue, some running container shows up in docker ps has already gone(no pid found). It seems to me that the internal state maintained by dockerd is inconsistent with the real state. And this mostly happened when the system is under memory/IO pressure. And restart docker restore the state.
Maybe you should check oomkill/hang errors in kernel log
akerouanton
commented
3 years ago 
Sh4d1
commented
3 years ago 👋 ah good to know! I'll attach it here if it happens again, thanks!
akerouanton
commented
3 years ago I looked at bit more at your description: given the containerd task is gone but the containerd container and the netns are still there, I believe Docker is stuck somewhere here (daemon.Cleanup() calls containerd to delete its container and removes the netns): https://github.com/moby/moby/blob/4283e93e6431c5ff6d59aed2104f0942ae40c838/daemon/monitor.go#L27-L63
I see you're using fluentd in async mode, do you know if the fluentd server was still running when you tried to stop/kill/rm the container? There's a bug that prevents fluentd logger to stop because it's blocked in an exponential backoff retry loop when there're logs to send but the fluentd server is down. This bug manifests the same symptoms (eg. hanging docker commands, etc...).
Sh4d1
commented
3 years ago Indeed, I suspected flutend at start, but found no clue. IIRC there were some issues with fluentd! I'm going to wait for it to happen again and get the stack trace then! Thanks!
sparrc
commented
2 years ago Hello from AWS ECS, we believe we have also seen this issue, and as the original opener mentioned, it seems rare and hard to reproduce.
We have also noted the relationship to the fluentd log driver, and we have some reason to believe that recent fixes in the fluent-logger-golang library may have fixed it. These fixes were pulled into moby master and backported to docker 20.10.13 here: https://github.com/moby/moby/pull/43147
Has anyone seen this issue using docker 20.10.13+ ?
vnovy
commented
1 year ago I see this or similar issue with following docker logging config
"log-driver": "fluentd", "log-opts": { "mode": "non-blocking", "fluentd-async": "false", "fluentd-address": "tcp://x.x.x.x:24224", "tag": "docker.{{.ID}}", "fluentd-sub-second-precision": "true"
docker ver. 23.0.1
It seems to be 100% reproducible
1/ start stack with fluentd running
2/ stop fluentd
3/ wait a moment
4/ rm stack
5/ containers which tried to log something after fluentd stop are ghosts
"fluentd-async": "true", problem does not appear
Description
Context:
A running container (launched with docker-compose) and a
restart: nopolicy, with a process that exit with a status code of 0. Here is the docker-compose file (docker-compose version 1.25.4, build 8d51620a) (just anonymized some info with***):When seeing this,
restart: neveris not a valid policy yet docker-compose does not mind, so I guess it's thenodefault restart policy that is in use (fixed with later docker-compose release).Issue:
When trying to stop/kill/inspect/rm this container, all the
docker <action> <container_id>hangs.I've found https://github.com/moby/moby/issues/30927 which is kind of old and https://github.com/moby/moby/issues/40817 (see below but I don't have any hung runc processes)
The stuck container ID here is 1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602 and the process linked to this container, is non existant.
What I've seen:
ctr -n moby task ls-> nothingctr -n moby c ls-> I can see the containerctr -n moby containers info 1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602Click to see
find / -name "*1c71de80*"In
/var/run/docker/containerd/1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602/there is two named pipes ->init-stdoutandinit-stderrtree /var/lib/docker/containers/1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb286023 directories, 7 files
file /var/run/docker/netns/06dbfe1d2459Then I tried:
ctr -n moby c kill 1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602-> ok, container is gone withctr c lsbutdocker rmstill hangsrm -Rf /var/run/docker/containerd/1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602/-> ok butdocker rmstill hangsstrace docker rm -f 1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602Then I fixed it with a known fix (had to fix the issue):
systemctl stop docker-> a bit long, in the logs :rm: cannot remove '/var/lib/docker/containers/1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602/mounts/shm': Device or resource busy
Sep 28 15:57:40 dockerd[17375]: time="2021-09-28T15:57:40.517315214Z" level=info msg="Loading containers: start." Sep 28 15:57:40 dockerd[17375]: time="2021-09-28T15:57:40.517551932Z" level=error msg="failed to load container" container=2d36abfab7fa136d8d36359c2cae30314d83ddadc37985d25531a5f0a1529779 error="open /var/lib/docker/containers/2d36abfab7fa136d8d36359c2cae30314d83ddadc37985d25531a5f0a1529779/config.v2.json: no such file or directory" Sep 28 15:57:40 dockerd[17375]: time="2021-09-28T15:57:40.518271443Z" level=error msg="failed to load container" container=1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602 error="open /var/lib/docker/containers/1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602/config.v2.json: no such file or directory" Sep 28 15:57:40 dockerd[17375]: time="2021-09-28T15:57:40.878044379Z" level=info msg="Removing stale sandbox 06dbfe1d245992abf3f075ecd893b1b6a44957519cbee8b77e3acaca579dc625 (1c71de80ad77d3b39579833bd61b80ae51c313ad7d629e1f658f3d6aeeb28602)" Sep 28 15:57:40 dockerd[17375]: time="2021-09-28T15:57:40.896801126Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 8b71e7a01854df19d6bf23ecbf76c1379317f39d34965d0c1992df62b40ed2e7 4395f7f3b45e21fd5b7516a771265d6ce81d9e93b01fd1c0d30767f642e98c6a], retrying...." Sep 28 15:57:41 dockerd[17375]: time="2021-09-28T15:57:41.845363492Z" level=info msg="Loading containers: done." Sep 28 15:57:41 dockerd[17375]: time="2021-09-28T15:57:41.885190387Z" level=info msg="Docker daemon" commit=75249d8 graphdriver(s)=overlay2 version=20.10.8 Sep 28 15:57:41 dockerd[17375]: time="2021-09-28T15:57:41.885304842Z" level=info msg="Daemon has completed initialization" Sep 28 15:57:41 dockerd[17375]: time="2021-09-28T15:57:41.931954248Z" level=info msg="API listen on /var/run/docker.sock" Sep 28 15:57:41 systemd[1]: Started Docker Application Container Engine.
Client: Docker Engine - Community Version: 20.10.8 API version: 1.41 Go version: go1.16.6 Git commit: 3967b7d Built: Fri Jul 30 19:54:27 2021 OS/Arch: linux/amd64 Context: default Experimental: true
Server: Docker Engine - Community Engine: Version: 20.10.8 API version: 1.41 (minimum version 1.12) Go version: go1.16.6 Git commit: 75249d8 Built: Fri Jul 30 19:52:33 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.9 GitCommit: e25210fe30a0a703442421b0f60afac609f950a3 runc: Version: 1.0.1 GitCommit: v1.0.1-0-g4144b63 docker-init: Version: 0.19.0 GitCommit: de40ad0
Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Build with BuildKit (Docker Inc., v0.6.1-docker) scan: Docker Scan (Docker Inc., v0.8.0)
Server: Containers: 8 Running: 5 Paused: 0 Stopped: 3 Images: 9 Server Version: 20.10.8 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: e25210fe30a0a703442421b0f60afac609f950a3 runc version: v1.0.1-0-g4144b63 init version: de40ad0 Security Options: apparmor seccomp Profile: default Kernel Version: 5.4.0-73-generic Operating System: Ubuntu 20.04.3 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 22.81GiB ID: XAWD:7ZZL:Q2TZ:NTTV:ZILV:S335:B3PR:BJRD:76XP:7KVY:CHPS:OGPS Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: :3128 HTTPS Proxy: :3128 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Default Address Pools: Base: 100.64.0.0/15, Size: 24
WARNING: No swap limit support