Docker stop container can't work, When container pid 1 becomes a zombie process

woshihaoren commented 5 years ago

Description Docker stop container can't work, When container pid 1 becomes a zombie process. And kill this zombie's parent process, than container can rm.

Steps to reproduce the issue: 1.sometimes docker stop container ，can not success,than can see pid 1 to be zombie.

unstable reproduction Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally): Here is container pid 1 becomes a zombie process:

Here is the normal process information:

F S UID         PID   PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S root          1      0  0  80   0 -  2923 do_wai Jun24 ?        00:00:00 bash /go/start.sh
0 S root          8      1  0  80   0 - 351053 futex_ Jun24 ?       00:01:15 /go/bin/backstage /go/bin/conf/settings.yaml

Output of docker version:

[xx@docker-110-90 ~]$ sudo docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output of docker info:

[xx@docker-110-90 ~]$ sudo docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false
[xx@docker-110-90 ~]$ sudo docker info
Containers: 48
 Running: 47
 Paused: 0
 Stopped: 1
Images: 60
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.108-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 58.48GiB
Name: docker-110-90.yxzq.com
ID: QMW6:YRIR:FOIP:EWVL:EBDO:JYTR:GHD7:2LOQ:EKSV:5HUX:QIOZ:X57K
Docker Root Dir: /opt/app/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 env=sit
Experimental: false
Insecure Registries:
 10.55.4.5:5000
 10.55.4.9
 test-registry.yxzq.com:5000
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

crosbymichael commented 5 years ago

Have you tried doing docker kill instead of stop to send a SIGKILL to the process? Also try running the container with the docker run --init flag to reap these zombies. There is also a daemon flag to enable the custom pid1 for all containers: dockerd --init

woshihaoren commented 5 years ago

@crosbymichael docker kill and docker run --init are useless docker kill can't stop. docker run --init,I encountered init becoming a zombie process

thaJeztah commented 4 years ago

https://github.com/moby/moby/pull/40749 should probably address this

moby / moby

Docker stop container can't work, When container pid 1 becomes a zombie process #39407