search

moby / moby

The Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
https://mobyproject.org/
Apache License 2.0
68.66k stars 18.65k forks source link

The number of processes in "uninterruptible sleep" ("D") on the host gravely influences dockerd performance #31648

Open piontec opened 7 years ago

piontec commented 7 years ago

Description Our monitoring detected docker performance problems rising slowly - checked with docker ps, docker info and docker run calls. At the same time about 100 processes with "D" state were detected in the system. Increasing number of "D" processes eventually renders docker daemon unresponsive. This was confirmed on multiple occasions (at least 5 times).

Steps to reproduce the issue:

  1. Make some processes running within container stuck in "D" (like on a read from failed remote file system)
  2. Observe docker API performance as the number of "D" processes increases.

Describe the results you received: Docker API response time grow and eventually render the whole daemon unresponsive.

Additional information you deem important (e.g. issue happens only occasionally): Is repeatable, happens every time the number of "D" processes on the host rises.

Output of docker version:

Client:
 Version:      1.12.3-cs4
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   65c6c4c
 Built:        Fri Nov 11 16:23:03 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3-cs4
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   65c6c4c
 Built:        Fri Nov 11 16:23:03 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 464
 Running: 363
 Paused: 0
 Stopped: 101
Images: 761
Server Version: 1.12.3-cs4
Storage Driver: aufs
 Root Dir: /opt/io1/docker/aufs
 Backing Filesystem: extfs
 Dirs: 3837
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-62-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 128
Total Memory: 1.876 TiB
Name: ip-10-69-11-89
ID: UGZS:UFD3:GB4C:W5MX:JU2L:K7PH:6ZWS:4GPM:27Q5:UNNN:X3DC:YDT7
Docker Root Dir: /opt/io1/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 2410
 Goroutines: 2766
 System Time: 2017-02-22T13:10:59.794584753Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

allencloud commented 7 years ago

@piontec Thanks for your feedback. Just to double check that the docker daemon unresponsive means all API calls unresponsive or docker ps unresponsive?

And can you give some hints how to make the processes in container turn to state "D"?