Closed taufek closed 4 years ago
@mmacvicar-splunk if you're hitting "No space left on device" errors after a few hours of high CPU load then my guess is that whatever is causing the CPU load is filling up the disk with logs. Once the disk is full then the docker daemon won't be able to start.
@JofArnold thanks for the update and good advice. I think we should add some of this information to the official docs to help people debug. I'm also considering adding a debug interface to display the events being sent to the VM (a bit like tail -f
of a log). Do you think that would be useful for debugging?
I just run docker with no VM running and it uses 1.25GB of memory directly :|
Solution (for me)
Just found this:
https://docs.docker.com/docker-for-mac/osxfs-caching/#cached
CPU went from 400% to 25% with the following.
Change your volume mounts from this:
../myproject:/usr/src/app
To this:
../myproject:/usr/src/app:cached
When I re-comile something it spikes to 250 but goes back down.
+1
Seeing same issue +1
Client: Version: 17.12.0-ce API version: 1.35 Go version: go1.9.2 Git commit: c97c6d6 Built: Wed Dec 27 20:03:51 2017 OS/Arch: darwin/amd64
Server: Engine: Version: 17.12.0-ce API version: 1.35 (minimum version 1.12) Go version: go1.9.2 Git commit: c97c6d6 Built: Wed Dec 27 20:12:29 2017 OS/Arch: linux/amd64 Experimental: true
Same problem running a django dev server on autoreload mode (setting the --noreload flag reduces cpu usage to normal values)
@gjthompson1 's workaround has helped reducing the cpu utilization
@kingsman142 @BugDiver @diegogub Please stop spamming with "same here" or +1 comments. Use the reaction buttons!.
@ximet Thanks for pointing out. Quite a newbie.
com.docker.hyperkit
was at 60% CPU while performing the download portion of the process and with no containers running. During the decompression process this jumped to 250%.
Command: docker run -d -it -p 2222:22 -p 80:80 -p 443:443 -p 3306:3306 -p 5432:5432 -p 8083:8083 niiknow/vestacp
Client: Version: 18.03.0-ce API version: 1.37 Go version: go1.9.4 Git commit: 0520e24 Built: Wed Mar 21 23:06:22 2018 OS/Arch: darwin/amd64 Experimental: false Orchestrator: swarm
Server: Engine: Version: 18.03.0-ce API version: 1.37 (minimum version 1.12) Go version: go1.9.4 Git commit: 0520e24 Built: Wed Mar 21 23:14:32 2018 OS/Arch: linux/amd64 Experimental: true
If you are using nodemon with polling then you can reduce the load by increasing the polling interval, e.g. --polling-interval 5000
.
I just removed the polling from nodemon and it seems docker is able to handle nodemon without polling. Is this new with v18.0?
/etc/host contains 127.0.0.1 localunixsocket localunixsocket.local # these are for docker
With Memory: 2GB settings the CPU utilisation is 450%. With Memory: 8GB the CPU u. is 25-80%.
Same problem.
Client: Version: 18.03.0-ce API version: 1.37 Go version: go1.9.4 Git commit: 0520e24 Built: Wed Mar 21 23:06:22 2018 OS/Arch: darwin/amd64 Experimental: false Orchestrator: swarm
Server: Engine: Version: 18.03.0-ce API version: 1.37 (minimum version 1.12) Go version: go1.9.4 Git commit: 0520e24 Built: Wed Mar 21 23:14:32 2018 OS/Arch: linux/amd
Just wanted to add that I was able to get CPU utilization down from around 300 to 100% by Removing all data
via the docker app as well as reducing the number of CPUs from 4 to 1.
As with @hedefalk I was seeing com.docker.hyperkit chewing more than 20% CPU and 2GB RAM when no containers were running. Disabled Kubernetes and CPU fell to <1% and ~1.2GB RAM.
macOS High Sierra 10.13.4
Docker CE edge 18.04.0-ce-mac62 (23965)
$ docker --version
Docker version 18.04.0-ce, build 3d479c0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
+1
+1
docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
docker version Client: Version: 18.05.0-ce-rc1 API version: 1.37 Go version: go1.9.5 Git commit: 33f00ce Built: Thu Apr 26 00:58:56 2018 OS/Arch: darwin/amd64 Experimental: true Orchestrator: kubernetes
Server: Engine: Version: 18.05.0-ce-rc1 API version: 1.37 (minimum version 1.12) Go version: go1.10.1 Git commit: 33f00ce Built: Thu Apr 26 01:06:49 2018 OS/Arch: linux/amd64 Experimental: true Kubernetes: Version: v1.9.6 StackAPI: v1beta1
kubectl version Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Reset to factory defaults button did the work over here with ~450% CPU utilization before.
I've noticed that com.docker.osxfs also have high load during these issues. hyperkit currently have 200% CPU usage on my 2014 MacBook pro. I assigned 10GB RAM, 4 cores + 3GB Swap for my docker daemon. Using one container I still have very high load on both osxfs and hyperkit even with 6GB free RAM in the container. CPU load inside the container is very low.
The load inside the container and the daemon doesn't seem to correlate. And running scripts inside the container is painfully slow.
Allowing Docker to use 4 GB of RAM instead of 2 GB and limiting CPU to 1 core changed everything for me (High Sierra, Docker for Mac 18.03.1, cheapest 13" rMBP 2014).
Docker is ~20x faster and the CPU usage is down from over 400% to 3-40%.
Finally, I was seriously considering moving to Ubuntu, because the last few months have been truly awful performance-wise.
woke up to my 2015 macbook (macOS 10.12.6) pro's fan spinning this morning. CPU usage above 200% for com.docker.hyperkit. Running Edge with kube. I had left all my containers running since two days ago, but had let the computer sit idle as it was the weekend. After restarting docker, CPU is down to 37-ish % and the fans are not spinning.
I've currently allocated 4 CPUs, 2GB mem, with 1GB swap. I'll likely up memory, but I feel the key thing here is something "ran away" with the CPU as it was not a high load when I stopped work on Friday.
$ docker version
Client:
Version: 18.05.0-ce-rc1
API version: 1.37
Go version: go1.9.5
Git commit: 33f00ce
Built: Thu Apr 26 00:58:56 2018
OS/Arch: darwin/amd64
Experimental: true
Orchestrator: kubernetes
Server:
Engine:
Version: 18.05.0-ce-rc1
API version: 1.37 (minimum version 1.12)
Go version: go1.10.1
Git commit: 33f00ce
Built: Thu Apr 26 01:06:49 2018
OS/Arch: linux/amd64
Experimental: true
Kubernetes:
Version: v1.9.6
StackAPI: Unknown
My Mac is strong. even though it will die because of this Docker:
My Mac will die, My Mac will die 😤:
I've tried resetting to factory default and using 4 GB of RAM instead of 2 GB and limiting CPU to 1 core (High Sierra, Docker for Mac 18.03.1). But that didn't work for me...
I am limiting the cpu usage for the hyperkit process with cputhrottle to 50% for now.... http://www.willnolan.com/cputhrottle/cputhrottle.html But this isn't an optimal solution...
Found a solution but I don't like it !
same here
Holy shit! Stop it please!
Also getting extremely high hyperkit CPU usage on stable version 17.12.0-ce-mac49 (21995).
I notice this happening every time I canceled the docker update dialog. The problem went away after I performed the actual update. notification leak?
In my case, it fixed by file share settings.
I removed unnecessary directories from sharing settings. (/Volumes, /private, /tmp, etc...)
I shared /Users
directory only.
For those using Laravel and 'npm run watch-poll'; This fixed it for me:
Came in to work this morning and com.docker.hyperkit
was using 650% of my CPU and 9.59gb of memory. Fans are spinning and my Mac sounds like it's trying to take off. Restarting Docker will make it stop, but it comes back every time.
OS X 10.13.4 2.5 GHz Intel Core i7 16gb ram
Docker settings 96gb of my hard drive 6 CPUs 10gb memory 1gb swap
Nope, that doesn’t work for me.
On Jun 13, 2018, at 7:58 AM, Musab Gültekin notifications@github.com wrote:
Solutions to this problem:
Allocate 4gb of memory, instead of 2gb. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/1759#issuecomment-396822738, or mute the thread https://github.com/notifications/unsubscribe-auth/AADELxPiX9APELqqjvBQU5cDYOCOpKB6ks5t8KoXgaJpZM4OAdoX.
Solutions to this problem:
This worked - Thank you. Has anyone from Docker offered a theory as to why both 1CPU and 4gb of ram are required to fix?
@NickJLange I suspect there are multiple different things happening in different situations.
For some workloads, 2 GiB is not enough memory so the VM will spend a lot of time swapping. This is less efficient than usual because all disk reads and writes need to go via hyperkit on the host.
The CPU usage of the VM is all accounted to hyperkit on the host, so reducing to 1 CPU reduces the maximum amount of work the VM is able to do. Perhaps the VM is still running containers which aren't needed any more?
If a large directory is shared (e.g. docker run -v $HOME:/home
) then all file writes anywhere in $HOME
will trigger filesystem events in the VM. This can be very expensive. I'm not completely sure why it's necessary to remove /Users
and /Volumes
entirely from file sharing in the preferences -- this would only help if there was a bug where a file watching registration was leaked. The active registrations can be queried by running /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state
Adding :cached
to volume mounts tells the VM it can cache filesystem data and metadata. If a workload is very read
-heavy, this massively reduces the number of calls from the VM to the host. We're experimenting with write caching for write
-heavy workloads.
I recommend taking a look inside the VM where docker is running with a command like:
docker run -it --privileged --pid=host justincormack/nsenter1
and check memory with free
and CPU with top
to see if there is anything unusual going on.
Just again like to note that I see high CPU even with no containers, no volumes and no images. Haven't tried in a few months, but that was true.
On Tue, 26 Jun 2018, 11:13 David Scott, notifications@github.com wrote:
@NickJLange https://github.com/NickJLange I suspect there are multiple different things happening in different situations.
For some workloads, 2 GiB is not enough memory so the VM will spend a lot of time swapping. This is less efficient than usual because all disk reads and writes need to go via hyperkit on the host.
The CPU usage of the VM is all accounted to hyperkit on the host, so reducing to 1 CPU reduces the maximum amount of work the VM is able to do. Perhaps the VM is still running containers which aren't needed any more?
If a large directory is shared (e.g. docker run -v $HOME:/home) then all file writes anywhere in $HOME will trigger filesystem events in the VM. This can be very expensive. I'm not completely sure why it's necessary to remove /Users and /Volumes entirely from file sharing in the preferences -- this would only help if there was a bug where a file watching registration was leaked. The active registrations can be queried by running /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state
Adding :cached to volume mounts tells the VM it can cache filesystem data and metadata. If a workload is very read-heavy, this massively reduces the number of calls from the VM to the host. We're experimenting with write caching for write-heavy workloads.
I recommend taking a look inside the VM where docker is running with a command like:
docker run -it --privileged --pid=host justincormack/nsenter1
and check memory with free and CPU with top to see if there is anything unusual going on.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/1759#issuecomment-400255980, or mute the thread https://github.com/notifications/unsubscribe-auth/AArljNnu96kzqlHIHFXbpCzcJB9zMyvoks5uAgk-gaJpZM4OAdoX .
In my case I just switched to linux and everything works smoothly since it directly uses host kernel :p who cares about mac os anw...
So no concrete solution to this issue? It's really annoying. Machine heats up, docker operations hangs, ):
I assign Docker 7 of my 8 cores, hyperkit uses all 7.
I assign Docker 50 of my 64gb if RAM, Docker consistently uses 47gb.
I am running an i7 iMac running latest v of High Sierra.
Ive got the same problem on my late 2013 MacBook. Even after reset Docker Edge and simply activating Kubernetes without running anything on Kubernetes or Docker it happens.
I think the key point (which may not be communicated) is that Docker displays this behaviour with no user-containers running (i.e. it's broken out of the box).
Stupid question - is there a way to trace which kube process is chewing up the extra cpu / work back to the source code?
docker ps output...
2458bf7c1bf docker/kube-compose-api-server "/api-server --kubec…" 44 hours ago Up 44 hours k8s_compose_compose-api-75d86595dc-ph9th_docker_9e9f5290-780a-11e8-be7d-025000000001_1
29327001110e docker/kube-compose-controller "/compose-controller…" 44 hours ago Up 44 hours k8s_compose_compose-6f784b699b-zsklj_docker_9e9ea8de-780a-11e8-be7d-025000000001_0
ca4b42368469 6f7f2dc7fab5 "/sidecar --v=2 --lo…" 44 hours ago Up 44 hours k8s_sidecar_kube-dns-86f4d74b45-trqsd_kube-system_9e9e928a-780a-11e8-be7d-025000000001_0
1ddbaa78747d c2ce1ffb51ed "/dnsmasq-nanny -v=2…" 44 hours ago Up 44 hours k8s_dnsmasq_kube-dns-86f4d74b45-trqsd_kube-system_9e9e928a-780a-11e8-be7d-025000000001_0
4b0c97aa7772 80cc5ea4b547 "/kube-dns --domain=…" 44 hours ago Up 44 hours k8s_kubedns_kube-dns-86f4d74b45-trqsd_kube-system_9e9e928a-780a-11e8-be7d-025000000001_0
f7c3471ef4f5 4261d315109d "/usr/local/bin/kube…" 44 hours ago Up 44 hours k8s_kube-proxy_kube-proxy-nq6fk_kube-system_9e720254-780a-11e8-be7d-025000000001_0
992ff72d3e59 k8s.gcr.io/pause-amd64:3.1 "/pause" 44 hours ago Up 44 hours k8s_POD_compose-api-75d86595dc-ph9th_docker_9e9f5290-780a-11e8-be7d-025000000001_0
260fff80a321 k8s.gcr.io/pause-amd64:3.1 "/pause" 44 hours ago Up 44 hours k8s_POD_compose-6f784b699b-zsklj_docker_9e9ea8de-780a-11e8-be7d-025000000001_0
a004bea965fd k8s.gcr.io/pause-amd64:3.1 "/pause" 44 hours ago Up 44 hours k8s_POD_kube-dns-86f4d74b45-trqsd_kube-system_9e9e928a-780a-11e8-be7d-025000000001_0
607f01a8e814 k8s.gcr.io/pause-amd64:3.1 "/pause" 44 hours ago Up 44 hours k8s_POD_kube-proxy-nq6fk_kube-system_9e720254-780a-11e8-be7d-025000000001_0
5469b644c554 e03746fe22c3 "kube-apiserver --ad…" 44 hours ago Up 44 hours k8s_kube-apiserver_kube-apiserver-docker-for-desktop_kube-system_d99c35ecb272d558b6f62e71173c2536_7
ca43b554c749 353b8f1d102e "kube-scheduler --ad…" 44 hours ago Up 44 hours k8s_kube-scheduler_kube-scheduler-docker-for-desktop_kube-system_ea66a171667ec4aaf1b274428a42a7cf_7
62f6f6c8dc30 52920ad46f5b "etcd --advertise-cl…" 44 hours ago Up 44 hours k8s_etcd_etcd-docker-for-desktop_kube-system_498c9fb5ba5bfb877eb777c3a98809c5_12
5911c2c1681a 40c8d10b2d11 "kube-controller-man…" 44 hours ago Up 44 hours k8s_kube-controller-manager_kube-controller-manager-docker-for-desktop_kube-system_f71e4063f9b42e669ab0c25296b9a99e_6
add8fee752ac k8s.gcr.io/pause-amd64:3.1 "/pause" 44 hours ago Up 44 hours k8s_POD_kube-scheduler-docker-for-desktop_kube-system_ea66a171667ec4aaf1b274428a42a7cf_5
1e603aa4b62d k8s.gcr.io/pause-amd64:3.1 "/pause" 44 hours ago Up 44 hours k8s_POD_kube-controller-manager-docker-for-desktop_kube-system_f71e4063f9b42e669ab0c25296b9a99e_5
448b719e7aa0 k8s.gcr.io/pause-amd64:3.1 "/pause" 44 hours ago Up 44 hours k8s_POD_kube-apiserver-docker-for-desktop_kube-system_d99c35ecb272d558b6f62e71173c2536_5
7990462c3d98 k8s.gcr.io/pause-amd64:3.1 "/pause" 44 hours ago Up 44 hours k8s_POD_etcd-docker-for-desktop_kube-system_498c9fb5ba5bfb877eb777c3a98809c5_5
@NickJLange unfortunately it's a known issue that kubernetes uses too much CPU when idle -- see #3023 and the linked Minikube issue . This is a separate cause to some of the others reported here which don't use kubernetes. To investigate I would start with something like
$ docker run -it --pid=host --privileged alpine sh
/ # apk update && apk add strace
...
strace -ff -p <pid>
Since there is some interprocess communication we might need to work backwards -- the process spending the CPU might be doing so on request from another. It should be possible to infer from the trace what the process is trying to do and therefore locate it in the source code.
Just want to note a solution that worked for to reduce the high-CPU load and our case here.
We are using elasticsearch in docker-compose and the following ulimits
config appears to relate directly to the high-CPU load:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.7
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms1g -Xmx1g"
ulimits:
memlock:
soft: -1
hard: -1
Removing the ulimits
section bought CPU load down from > 200%
to < 10%
!
Adding it back brought the CPU load back up again.
So, I've removed that part in local development for now, and the CPU is nice and cool again.
+1
I am experiencing the same issue. Tested it on a
I tested it running docker-compose with multiple containers (Elixir backend, Elasticsearch, Postgresql, React, Angular) and docker-sync on stable 18.03.1-ce
and edge 18.05.0-ce
I tried different configurations and compared CPU load when everything is idle and when I run tests in Elixir. Kubernetes is always deactivated.
Stable:
Edge:
Takeaways:
Interestingly my colleague is running the same setup on an older MacBook Pro. He doesn't have the same issues though. I think that this is because of the HFS filesystem on his machine.
There are some issues on docker-sync that seem related:
https://github.com/EugenMayer/docker-sync/issues/410#issuecomment-403448520
If you use the native delegated volumes, docker-sync isn't needed anymore as far as I'm concerned. I've had solid performance on Magento 2 (TONS of folders/files) with no delay or halts in syncing.
https://github.com/EugenMayer/docker-sync/issues/515#issuecomment-356635435 - Looks like 17.09.0-ce-mac35 stable
doesn't have the issue. Downgrading might work.
the same issue at my Mac:
➜ ~ docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 4
Server Version: 18.05.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.93-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: linuxkit-025000000001
ID: UVEG:RE7Z:C2GY:EHQV:7NKJ:XHQ3:ASDJ:HOMX:SEA6:U7TS:DVDF:HYSH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 46
Goroutines: 78
System Time: 2018-07-15T05:24:10.8858545Z
EventsListeners: 2
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
➜ ~
➜ ~ docker version
Client:
Version: 18.05.0-ce
API version: 1.37
Go version: go1.9.5
Git commit: f150324
Built: Wed May 9 22:12:05 2018
OS/Arch: darwin/amd64
Experimental: true
Orchestrator: kubernetes
Server:
Engine:
Version: 18.05.0-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.10.1
Git commit: f150324
Built: Wed May 9 22:20:16 2018
OS/Arch: linux/amd64
Experimental: true
Kubernetes:
Version: Unknown
StackAPI: Unknown
➜ ~
Same here. High Sierra, encrypted apfs. Both with docker stable and docker edge, hyperkit cpu on the maximum, machine is hot, everything slows down. This usually lasts for ten or fifteen minutes, then things get back to normal for another half an hour.
On my linux machine none of this happens, but primarily we use macs, and the coworker with El Capitan and hpfs seems to be unaffected.
same here. High Sierra, macos 10.13.5 Any solution ?
Did anyone downgrade to Sierra and HPFS? I'm considering that options too :(
Edit: happens on el capitan too :(
My CPU utilization by hyperkit on my mac seems too high. It caused my mac to overheat.
Below is my docker version info:
Below is my diagnostic id when this issue is happening.
1B8B3901-6393-41B7-B7D3-0FC914F1C668
I'm running rails container with
docker-sync
.