High CPU Utilization of Hyperkit in Mac

taufek commented 7 years ago

My CPU utilization by hyperkit on my mac seems too high. It caused my mac to overheat.

Below is my docker version info:

╰─ docker version                                                                                                                                                                                                  13:06:29
Client:
 Version:      17.06.0-ce-rc4
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   29fcd5d
 Built:        Thu Jun 15 17:29:01 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.0-ce-rc4
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   29fcd5dfae
 Built:        Thu Jun 15 18:50:07 2017
 OS/Arch:      linux/amd64
 Experimental: true

Below is my diagnostic id when this issue is happening. 1B8B3901-6393-41B7-B7D3-0FC914F1C668

I'm running rails container with docker-sync.

hlogeon commented 6 years ago

Same issue

davidsielert commented 6 years ago

Same here.. been doing it for awhile.. Causes my macbook pro to continually overheat..

leoanjossantos-zz commented 6 years ago

It is happening here as well. :\

rskuipers commented 6 years ago

Just installed the Edge channel and it seems quiet now, let's see if it stays that way.

manzznam commented 6 years ago

Same problem here with high CPU usage ... even when trying Edge channel.

samlinux commented 6 years ago

Same here also with the edge channel. hyperkit has a CPU usage more than 200% Mac 10.13.2 Version: 17.12.0-ce-rc3

mateuszrusiecki commented 6 years ago

Same here, in my case it happens after npm install is completed. Commands fired in project but not in docker. Synchronisation or sharing host <-> container? Docker Version 17.09.0-ce-mac35 (19611) Mac 10.12.6

zalewskigrzegorz commented 6 years ago

Same here Version 17.12.0-ce-rc3-mac43 (21270) Mac 10.13.2

saleh199 commented 6 years ago

I had the same issue with edge version, and my solution was by using http://docker-sync.io/

efhjones commented 6 years ago

Same here Version 17.09.1-ce-mac42 (21090) Mac 10.12.6

maznag commented 6 years ago

Same here Version 17.09.1-ce-mac42 (21090) Channel: stable 3176a6af01 Mac 10.13.1 (17B1003)

district10 commented 6 years ago

Docker version 17.09.1-ce, build 19e2cf6 Mac 10.13.2 hyperkit cpu 470%.... 😢

rpggio commented 6 years ago

Docker version 17.11.0-ce-mac40 (20561) Mac 10.13.2 hyperkit cpu 300% while running npm install

samlinux commented 6 years ago

In my case I used the -L flag with the CLI command nodemon. In my opinion this was the reason for the high CPU usage, especially in projects with many files to watch.

If I start nodemon without this flag the CPU usage is very low, also in large projects.

vasiliy0s commented 6 years ago

Same here macOS 10.13.2 Docker CE 17.09.1-ce-mac42 (21090) hyperkit use cpu over and over with webpack -hot reload and compilation (too many npm modules shared inside docker container within docker-compose volumes).

stevehogdahl commented 6 years ago

+1 Same here MacOS 10.13.2 (17C88) - High Sierra Docker CE 17.09.1-ce-mac42 (21090) Channel: stable 3176a6af01

Hyperkit jumps to 100%+ immediately and continues to climb above 200% getting to 250% at times

abentele commented 6 years ago

Same here iMac 2017, MacOS 10.13.2 (17C88) Docker 17.12.0-ce-rc4, build 6a2c058 (Edge) Docker settings: 4 CPU, 12GB RAM

Increasing RAM to 12GB in the docker settings seems to help.

lyda commented 6 years ago

Also saw that issue on a machine with no containers.

I made this diagnostic if it helps: 198BB074-5A78-4598-B821-189843C74E08

For now just turning off docker on my laptop.

betsydupuis commented 6 years ago

+1

jeremygaither commented 6 years ago

Issue persists after deleting all data and limiting daemon to 1 CPU thread.

macOS High Sierra 10.13.2 (17C88)

$ docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:22:25 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:28:28 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 9
Server Version: 17.09.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.855GiB
Name: moby
ID: ZOQU:PMLJ:CWGE:VTEO:NOI2:2CPQ:Q44Y:KGQV:ATME:76SW:CY3H:LE6E
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Macmee commented 6 years ago

+1 same problem here High Sierra 10.13.2 Docker 17.09.1-ce-mac42

I thought it might be because of the speculative execution (separate user/kernel pagetables) fix in this version of MacOS but my colleague on 10.12 is experiencing the issue also.

zoka123 commented 6 years ago

+1

alexellis commented 6 years ago

+1 - my battery is really suffering so much so that I only open DfM on when absolutely needed rather than leaving it in the background. Right now the computer is completely idle and hyperkit is using > 60% battery.

djs55 commented 6 years ago

Thanks for highlighting this issue.

There are a number of known causes of high CPU load in hyperkit:

a busy process in a container (although this is obvious, while developing things I've often left a spinning process somewhere by accident)
sharing a busy host directory: for example sharing /Users/foo is very expensive because it includes all the temporary cache files created by the web-browser, causing load spikes while browsing the web. It's best to focus volume sharing on only the directories that matter i.e. the ones containing application code. We're working to make the volume sharing faster and more efficient too.
the kube API server uses CPU when idle -- we're still investigating that.

There may be other reasons too which we're not aware of yet. The next time this happens to you, could you take a look with

docker stats

If that doesn't show anything interesting, try removing the shared volumes to see if the problem is volume-related. If that fails then use

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

to get a shell in the VM where you can run top. If you could create a fresh issue with a diagnostics report, ideally a way to reproduce the result and observations from the above tools then that would be really helpful.

Thanks for your patience!

lyda commented 6 years ago

OK, I start docker and nothing is running. No containers, no volumes, no stopped containers. Load avg hovers around 2.1 on the OS X box. The output of docker stats is empty bar the table headers.

Running the screen command I see this:

Mem: 513568K used, 3528824K free, 168208K shrd, 210100K buff, 174752K cached
CPU:   0% usr   0% sys   0% nic  99% idle   0% io   0% irq   0% sirq
Load average: 0.16 0.06 0.01 1/172 2352
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 1874  1868 root     S     411m  10%   0   0% dockerd --pidfile=/run/docker.pid
 2290  2280 root     R     1532   0%   0   0% top
 1540     1 root     S     351m   9%   0   0% /usr/bin/containerd
 1592  1540 root     S     206m   5%   0   0% containerd-shim rngd /containers/r
 1881  1874 root     S     136m   3%   1   0% docker-containerd -l unix:///var/r
  982     1 root     S    23144   1%   1   0% /usr/sbin/klogd -c 3 -2
 1416     1 root     S    14224   0%   1   0% /sbin/transfused -p /var/run/trans
  947     1 root     S    11284   0%   0   0% /sbin/vsudd -inport 2376:unix:/var
 1326     1 root     S     7968   0%   0   0% /usr/bin/diagnostics-server -vsock
 1737     1 root     S     4940   0%   0   0% /sbin/proxy-vsockd -vsockPort 6237
 1519     1 chrony   S     2960   0%   1   0% /usr/sbin/chronyd -f /etc/chrony/c
 2280     1 root     S     1532   0%   0   0% /bin/sh --
 1005     1 root     S     1528   0%   1   0% /usr/sbin/crond -c /etc/crontabs
 1868     1 root     S     1524   0%   0   0% /bin/sh -c exec dockerd --pidfile=
    1     0 root     S     1520   0%   1   0% /sbin/init
 1875  1868 root     S     1520   0%   1   0% logger -p local0.info
 1466     1 root     S     1520   0%   1   0% /sbin/acpid
 1300     1 root     S     1216   0%   1   0% /sbin/dhcpcd -q
  974     1 root     S      784   0%   1   0% /usr/sbin/syslogd -m 0
 1746  1679 root     S      308   0%   0   0% /usr/sbin/rngd -f

With docker stopped load is around 1.5 (iTerm2, Chrome and Slack were also running through this test).

abentele commented 6 years ago

observations from my system, today:

Docker version 17.12.0-ce, build c97c6d6
hyperkit producing very high CPU load

docker stats output:

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
                    --                  --                  -- / --             --                  --                  --                  --
                    --                  --                  -- / --             --                  --                  --                  --
                    --                  --                  -- / --             --                  --                  --                  --
                    --                  --                  -- / --             --                  --                  --                  --
                    --                  --                  -- / --             --                  --                  --                  --
                    --                  --                  -- / --             --                  --                  --                  --
                    --                  --                  -- / --             --                  --                  --                  --
                    --                  --                  -- / --             --                  --                  --                  --
                    --                  --                  -- / --             --                  --                  --                  --
                    --                  --                  -- / --             --                  --                  --                  --

Used the command to open a shell and run top:

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

top result:

Mem: 8037916K used, 126764K free, 42560K shrd, 1208K buff, 82212K cached
CPU:  43% usr  16% sys   0% nic  39% idle   0% io   0% irq   0% sirq
Load average: 2.90 9.29 8.24 6/1401 4877
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 1029  1003 root     R     988m  12%   0  50% /usr/local/bin/dockerd --config-fi
27151 27085 1        S    5020m  61%   3   3% {java} /usr/lib/jvm/java-8-oracle/
  165     2 root     SW       0   0%   1   2% [kswapd0]
 4620     2 root     SW       0   0%   0   1% [kworker/0:1]
 1028  1026 root     S    12712   0%   1   1% /usr/bin/transfused
 3058     2 root     SW       0   0%   2   1% [kworker/2:3]
30378 30376 992      S     351m   4%   3   0% {prometheus} /opt/gitlab/embedded/
27441 27393 200      S    4979m  61%   2   0% /opt/java/bin/java -server -Dinsta
29881 29857 1        S    4563m  56%   0   0% {java} /usr/lib/jvm/java-8-oracle/
30476 29304 996      S    61156   1%   0   0% postgres: gitlab-psql postgres [lo
30246 30244 998      S     431m   5%   2   0% {gitlab-mon} /opt/gitlab/embedded/
30556 30513 1030     S    5680m  69%   0   0% /docker-java-home/bin/java -Djava.
 2525  1997 1000     S    5562m  68%   3   0% java -Djava.awt.headless=true -jar
29556 29554 998      S     875m  11%   3   0% {bundle} sidekiq 5.0.4 gitlab-rail
 4844 25742 999      S     289m   4%   0   0% postgres: postgres confluence 172.
30472 30416 996      S     229m   3%   0   0% {postgres_export} /opt/gitlab/embe
30149 30147 992      S     170m   2%   1   0% {node_exporter} /opt/gitlab/embedd
   39     2 root     SWN      0   0%   3   0% [khugepaged]
30593 30453 998      S    2329m  28%   2   0% ruby /opt/gitlab/embedded/service/
 3788  1029 root     S     296m   4%   1   0% docker-containerd --config /var/ru

dockerd is always around 50% CPU.

Macmee commented 6 years ago

@djs55 I don't think the problem is that the containers we are running are just poorly optimised. In previous versions of docker and Mac OS I did not observe this problem. Also when running docker inside an ec2 instance with Ubuntu I don't experience the same issue.

I realise all this information is relatively unhelpful without diagnostic information so I'll try and follow up with more when I'm near a computer.

ghost commented 6 years ago

I was having the same problem (CPU constantly over 200% and overheating), the fix for me was twofold:

On Docker for Mac's "File sharing" menu, remove all entries and replace only with the paths I'm actually sharing between my host and containers (It had /Users, /tmp, and other stuff from previous configs)
For development of my own stuff I'm using "webpack watch", I increased the "watch" frequency from 300 milliseconds to two seconds per https://webpack.js.org/configuration/watch/

DfM is now regularly at ~20% CPU usage and no overheating!

My only concern is that if you fix this problem, you will be breaking workflows!:

workflow

efhjones commented 6 years ago

FWIW, not sure if this fix will help for everyone, but I was having all these symptoms as well as a lot of too many open files errors eventually, which didn't make sense as I had a limit somewhere over a million. It seems like elasticsearch was having some big problems, maybe some crufty data messing things up. Turns out when I removed the local data for elasticsearch it fixed the issue. In my project directory I ran rm -rf .local/es-data-2.

benetis commented 6 years ago

I am experience this issue when running sbt and polling every 500ms (watching resources) for compilation. Colleague with Sierra (I am on High Sierra) does not experience same issue

lukasborawski commented 6 years ago

So, this one solved my problem with CPU: https://medium.com/@sbr464/limit-dropbox-and-others-on-macos-from-taking-100-cpu-877266df104d

dfee commented 6 years ago

@lukasborawski that's not solving the problem, it's making the situation manageable – like cough suppressant medication. Worse, you're now throwing endless cycles at nothing(?) when you intend to actually be using docker for something productive – like an application that should be drawing 100% (of the 60%) of the CPU.

carmelocatalfamo commented 6 years ago

Same issue here, it is small project with react and node.

monitoraggio

Diagnostic ID: 43CE30EB-344C-4940-8E1E-353A23297FEF

Docker for Mac: version: 17.09.0-ce-mac35 (69202b202f497d4b6e627c3370781b9e4b51ec78)
macOS: version 10.13.2 (build: 17C205)
logs: /tmp/43CE30EB-344C-4940-8E1E-353A23297FEF/20180118-130324.tar.gz
[OK]     db.git
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

Docker version: Version 17.09.0-ce-mac35 (19611) Channel: stable a98b7c1b7c

OS: macOS High Sierra 10.13.2 MacBook Pro (15-inch, 2016) 2,6 GHz Intel Core i7 16 GB 2133 MHz LPDDR3

Docker stats

tayfunyasar commented 6 years ago

+1

lukasborawski commented 6 years ago

@dfee of course you have right, but for now, to prevent waking up my daughter at the middle of the night from horribly noisy windmill that is trying to cold up my macbook on 220% CPU it's OK :)

and still waiting for something better

+1

tayfunyasar commented 6 years ago

I found a workaround.

1- Update your docker to latest version (Version 17.12.0-ce-mac49 (21995)), 2- Preferences > Reset > Reset to factory defaults

thats it. It was about %440-420 in my mac, not it is about %1.2-1.6

jamesjryan commented 6 years ago

Purely anecdotally, recent Docker Edge versions (Version 18.01.0-ce-mac48 (22004)) don't seem to be exhibiting this for me as badly.

Using macOS, I removed /Users and /Volumes from file sharing and hyperkit hovers around ~20% instead of 80-90%.

djs55 commented 6 years ago

@jamesjryan that's an interesting observation about /Users and /Volumes. A known problem with the file sharing is if a very busy directory is shared (e.g. $HOME) then a lot of CPU is wasted sending pointless filesystem events to the VM. Perhaps some of the volume shares are leaking and remaining active even after containers shutdown? Next time this happens, try running

$ /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state
Exported directories:
 - /Users to /Users (nodes_/Users table size: 236735)
 - /Volumes to /Volumes (nodes_/Volumes table size: 0)
 - /private to /private (nodes_/private table size: 0)
 - /tmp to /tmp (nodes_/private/tmp table size: 0)
Container-mounted directories:
 - /Users/.../ember/myapp into 1b199c0670948dfb8ad58ea4f29abad967698da00dc637d22196b03dae2cdf36 (state=default)
 - /Users/.../ember/myapp into 365ab3b07dfd1f48a6f4c9f4c0b378da3a195e323dfd6e69bac5f06b01e53ba7 (state=default)
 - /Users/.../ember/myapp into 8bc2ea3936422012d07e098d577b0b1a4172ca79c69d0a8c5a51f0dbb4d738d8 (state=default)

and see if the container-mounted directories looks right. On an idle system it should be an empty list. (In the mean time I'll add this to the diagnostics)

dfee commented 6 years ago

Wow. Pure speculation here. But I wonder if this is related to iCloud storage sync? They both began appearing for me around a year ago, and removing the Home directory from Docket for Mac apparently solves the problem.

jamesjryan commented 6 years ago

With 3 containers running and 2 stopped:

/Applications/Docker.app/Contents/MacOS/com.docker.osxfs state
Exported directories:
 - /private to /private (nodes_/private table size: 0)
 - /tmp to /tmp (nodes_/private/tmp table size: 5)
Container-mounted directories:

After stopping and removing all containers:

➜  git:(master) docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
➜  git:(master) /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state
Exported directories:
 - /private to /private (nodes_/private table size: 0)
 - /tmp to /tmp (nodes_/private/tmp table size: 5)
Container-mounted directories:

@djs55 Hope this helps

Further info w.r.t iCloud speculation - not signed into iCloud on this machine, or the other I use regularly, afaik.

jeremygaither commented 6 years ago

I'm not seeing this after updating to the latest version of Docker for Mac:

$ docker --version
Docker version 17.12.0-ce, build c97c6d6
$ /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state
Exported directories:
 - /Users to /Users (nodes_/Users table size: 27688)
 - /Volumes to /Volumes (nodes_/Volumes table size: 0)
 - /tmp to /tmp (nodes_/private/tmp table size: 0)
 - /private to /private (nodes_/private table size: 0)
Container-mounted directories:
 - /Users/.../Projects/qwerty/wasdfg into 64dc2fde055a17243f56d4b12b87313d85dd8c317cf9c0edfe21e4abea4292e4 (state=cached)

zone-live commented 6 years ago

+1 👍 Docker version 17.12.0-ce, build c97c6d6

nicolam1 commented 6 years ago

+1

RobertDeRose commented 6 years ago

This is actually the first version of Docker I am seeing the issue with: Docker version 17.12.0-ce, build c97c6d6

XCiber commented 6 years ago

+1

Docker Version 18.02.0-ce-rc1-mac50 (22256) (Edge)

Server: Engine: Version: 18.02.0-ce-rc1 API version: 1.36 (minimum version 1.12) Go version: go1.9.3 Git commit: 5e1d90a Built: Thu Jan 25 00:40:43 2018 OS/Arch: linux/amd64 Experimental: true

docker restart helps…

saumesh commented 6 years ago

+1

Hyperkit consumes more than 240% CPU MacOS: Sierra (10.12.6) Docker: 17.12.0-ce-mac46

jlyon commented 6 years ago

+1

Docker version 18.02.0-ce-rc1, build 5e1d90a

System Version: macOS 10.13.4 (17E139j) Kernel Version: Darwin 17.5.0

Axel-Jacobsen commented 6 years ago

+1

Docker version 17.12.0-ce, build c97c6d6

MacOS: 10.13.3 High Sierra

majelbstoat commented 6 years ago

I don't know if yet more info is helpful, but:

Docker: Version 18.02.0-ce-rc2-mac51 (22446) Edge OSX: 10.13.3 (17D47)

Allowed resources: 4 CPUs, 3 GB memory

Kubernetes activated hosting a single deployment of three Ambassador pods doing no real work (no polling etc).

hyperkit CPU frequently above 100%, never any less than 60%. When webpack --watch is running, frequently 250+%.

Result of top after screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty:

File sharing:

 ~$  /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state
Exported directories:
 - /Users to /Users (nodes_/Users table size: 0)
Container-mounted directories:

Restarting docker/kubernetes does not solve it.

jason-linthwaite commented 6 years ago

I also have the same problem. Running a docker container that's got node.js running in it. CPU at 160%..... Docker on mac

Version 17.12.0-ce-mac49 (21995) Channel: stable f50f37646d

MacOs Version 10.13.3 (17D47)

docker / for-mac

High CPU Utilization of Hyperkit in Mac #1759