Memory Leakage com.docker.hyperkit

Sebastianbrg commented 5 years ago

NOTES TO FUTURE READERS OF THIS THREAD. PLEASE READ THE WHOLE THREAD BEFORE REPLYING. IN PARTICULAR, PLEASE UNDERSTAND: 1) There is a bug in MacOS that it reports twice the amount of memory actually allocated. 2) It is normal that all the memory has to be reserved up front before any containers are running, because of the way that virtualisation works in MacOS Hypervisor.framework.

See https://docs.google.com/document/d/17ZiQC1Tp9iH320K-uqVLyiJmk4DHJ3c4zgQetJiKYQM/edit?usp=sharing for a detailed investigation on this.

-- Stephen Turner

[ x ] I have tried with the latest version of my channel (Stable or Edge)
[ x ] I have uploaded Diagnostics
Diagnostics ID: C19AE3AD-3E52-4F8C-A16A-D06D797B36A4/20180925100823

Expected behavior

No memory leakage.

Actual behavior

Memory leakage.

Information

Is it reproducible? Yes. Just start Docker on Mac OS Mojave
Is the problem new? Yes, when I upgraded to Mac OS Mojave
Did the problem appear with an update?
A reproducible case if this is a bug, Dockerfiles FTW.
macOS Version: 10.14 (18A391)

Diagnostic logs


Docker for Mac: Version 18.06.1-ce-mac73 (26764)

lou-lan commented 5 years ago

I also get a memory leak

sickness29 commented 5 years ago

Me too

EduardRakov commented 5 years ago

and me too :( also com.docker.hyperkit loads CPU very much

Zabudska-Yuliia commented 5 years ago

Me too

iMerica commented 5 years ago

See #178.

vfontes commented 5 years ago

Also happening for me. Running on macOS High Sierra 10.13.6.

Docker version 18.06.1-ce, build e68fc7a

gvbkr commented 5 years ago

Yes this is happening on macOS High Sierra 10.13.6.

Docker version Version 18.06.1-ce-mac73 (26764)

This started happening after I updated the Docker version.

ylor commented 5 years ago

Seeing the same thing.

2.83GB being used without anything running.

Docker version 18.06.1-ce, build e68fc7a

AnielloFalcone commented 5 years ago

4.77Gb with no images and no containers on: macOS: 10.14.1 Beta (18B45d), Docker: 2.0.0.0-beta1-mac75 (27117), Engine: 18.09.0-ce-beta1

mkohlmyr commented 5 years ago

Mac OS High Sierra 10.13.4 Docker 18.03.1-ce

Regularly consumes ~1.5GB while completely idle after startup.

ghost commented 5 years ago

[ x ] I have tried with the latest version of my channel (Stable or Edge) [ x ] I have uploaded Diagnostics Diagnostics ID: 01B9140E-D975-46AE-AEA6-442DC281FAE0/20181001181235 Expected behavior: Low memory use Actual behavior: Doing nothing and consuming 5Gb of memory, it is also reported as the highest energy consumer on my system.

Information Is it reproducible? Yes. Just start Docker on Mac OS Mojave Is the problem new? Don't know, never used it on a different version Did the problem appear with an update? Don't know, never used it on a different version.

MacOS 14 (18A391) Docker Community Edition 18.06.1-ce-mac73 (26764)

Hope this helps.

gauravthadani commented 5 years ago

Likewise, noticing this after the os upgrade to mojave. does the update fix the issue? I couldn't figure out from the commits

asburymr commented 5 years ago

I have run into the same issue. Below are the specifics of my environment. This is with no containers running and a fresh restart.

asburymr commented 5 years ago

An interesting data point may be that increasing the SWAP seems to dramatically affect the Memory consumed by the hyperkit.

512MB SWAP -> 2GB of memory

1GB SWAP -> 3 GB Memory

2 GB SWAP -> 5 GB Memory

alvarobem commented 5 years ago

I have the same issue

wskinner-eml commented 5 years ago

Same here. With a container running, com.docker.hyperkit used over 9GB of memory. After a restart, with no containers running, it immediately jumped up to 5.23 GB. Docker version: 18.06.1-ce-mac73 (26764) Docker memory limit: 12GB Docker Swap limit: 4.0GB

I was also able to reproduce the effect @asburymr noticed. When I reduced the swap limit to 512MB, hyperkit went down to under 2GB RAM.

andypan-twn commented 5 years ago

Same issue.

imuller commented 5 years ago

Same issue here with version 18.06.1-ce-mac73, using almost 3 GB memory after startup of my mac with Mojave (10.14)

TedSpinks commented 5 years ago

I have this issue with High Sierra 10.13.6 and Docker version 18.06.1-ce-mac73 (26764). com.docker.hyperkit consumes 1.6GB with no containers and 1GB swap. 800MB with no containers and 512MB swap.

Running a container causes com.docker.hyperkit to consume 50-60MB while it is running, but it releases the 50-60MB as soon as the container stops.

rurkss commented 5 years ago

2018-10-05 13 26 48 same here, on mac

bradrich commented 5 years ago

Upgrading to the Edge release seems to alleviate the problem. The below screenshot is with a set of containers running.

screenshot 2018-10-05 10 49 43

TedSpinks commented 5 years ago

@bradrich what does your memory look like after you stop those containers?

bradrich commented 5 years ago

@TedSpinks If there are no containers running, it is a little better, but not perfect.

screenshot 2018-10-05 10 27 04

ylor commented 5 years ago

I had to give up and create a Linux VM inside of Parallels to be my Docker host. It uses less energy and memory than Docker. As a bonus it allows for --net=host.

It's an unfortunate state of affairs, but until Docker for Mac improves it's honestly the best way to go.

marekpizner commented 5 years ago

Me too OS: macOS Mojave (10.14) Docker 18.06.1-ce-mac73 (26764) EEAD09C8-BC82-4E6E-993E-F4BBB220B93B/20181104225806

florentin commented 5 years ago

Same issue here, Docker Version 18.06.1-ce-mac73 (26764) on Mohave 10.14 (18A391) com.docker.hyperkit takes 3.69GiB without any container running Docker settings: swap 512MiB, CPUS: 4, Memory: 6GiB Increasing the swap value automatically increases the hyperkit's memory For a swap setting of 1.0 GiB the hyperkit takes 4.69 GiB of system's memory.

For the Edge version (Version 2.0.0.0-beta1-mac75 (27117) hyperkit: 2.91 GiB for the Docker settings: 4 cpus, 6GiB ram and 1GiB swap

renielsalvador commented 5 years ago

Me too OS: macOS Mojave Docker version 18.06.1-ce, build e68fc7a

chaseWillden commented 5 years ago

Same issue

mglowala commented 5 years ago

OS: Mojave Engine: 18.09.0-ce-beta1 Version 2.0.0.0-beta1-mac75 (27117) Cpu: 5 Memory: 9 Swap: 2 Hyperkit: 9,39

appeltonen commented 5 years ago

I have this issue with High Sierra 10.13.6 and Docker version 18.06.1-ce-mac73 (26764). com.docker.hyperkit consumes 1.6GB with no containers and 1GB swap. 800MB with no containers and 512MB swap.

Running a container causes com.docker.hyperkit to consume 50-60MB while it is running, but it releases the 50-60MB as soon as the container stops.

I have the exact same issue with these versions.

aihkas commented 5 years ago

Same problem on Mojave with the following docker version, a whopping 3.6 plus GB after boot up!

screenshot 2018-10-09 at 20 41 09

A resources clean up with docker system prune -a then a restart has reclaimed 10.6 GB of disk space and just 650 MB of memory.

kalexmills commented 5 years ago

Just left this Diagnostic ID on the moby repository (moby/hyperkit/#231), and realized that is an OSS project. The Docker devs will probably find it more useful. :grin:

In my case, memory usage grows to 11GB after <30 minutes of usage, with some containers running. Let me know if you need any more information. I am happy to help.

3CBF0FFC-A3ED-4F24-944A-650842923449/20181010135026

fgeck commented 5 years ago

Same for me. Even after a docker system prune -a 7.49GB and 20 Threads are used by com.docker.hyperkit.

But docker info tells me that only 3,854 GB are used (I allocated 4 GB in the preferences):

Docker version 18.06.1-ce, build e68fc7a Architecture: x86_64 CPUs: 6 Total Memory: 3.854GiB

I am running Mac OS Mojave 10.14.

rerowep commented 5 years ago

I have also problems with memory. I have set the memory preference of docker to 8Gb but the com.docker.hyperkit is growing and growing. I have to restart docker to free the memory. I running on Mac Os Majave 10.14 and I tried with the latest and edge versions of docker. My coworkers still on Mac Os High Sierra 10.13 don't have this problem.

uvesten commented 5 years ago

Same here: After boot, no containers running, docker is consuming 3.16 GB of memory.

macOS 10.14 (18A391) Docker: Version 18.06.1-ce-mac73 (26764)

Docker settings: 8 CPUs 8.0 GiB Memory 1.0 GiB swap

Decreasing the abovementioned settings to

8 CPUs 4.0 GiB Memory 512 MiB swap

decreased Docker's memory usage to 2.01 GB of memory. Still with absolutely nothing at all running...

jhaagmans commented 5 years ago

Most people in these topics seem to be referencing virtual memory instead of resident memory and I've seen this causing some confusion before. @ijc has addressed this in #178. Granted, this is confusing, but I agree that this is not an issue with specifically docker-for-mac.

However, my issue is actually with rss growing too large. I run docker on 3 types of infrastructure (Linux, Windows and MacOS). The first 2 rarely grow over 10 MB in an idle state, while the latter (running docker-for-mac) always has a very high resident memory usage.

With no running containers on MacOS Mojave on a brand new MBP:

sudo ps -awxm -o %mem,rss,comm | sort -nr | grep docker
14.2 1187336 com.docker.hyperkit
 0.2  19076 com.docker.driver.amd64-linux
 0.1  11904 /Applications/Docker.app/Contents/MacOS/com.docker.supervisor
 0.1   7384 com.docker.vpnkit
 0.1   6880 com.docker.osxfs
 0.0   4168 /Library/PrivilegedHelperTools/com.docker.vmnetd

docker --version
Docker version 18.06.1-ce, build e68fc7a

This unfortunately also seems to lead to random freezes that don't happen when not running docker. I do feel this needs more attention. Please let me know what diagnostic info I can share to enable investigation.

avxkim commented 5 years ago

Any news?

kalexmills commented 5 years ago

There might be more traction if folks would follow @jhaagmans lead and edit their comments to include RSS rather than raw memory as reported by top. I plan to edit my own comment in this fashion later today once I am in front of my machine.

asburymr commented 5 years ago

With no containers running on a clean boot showing the various available memory metrics.

kalexmills commented 5 years ago

@asburymr Please edit your response to include a diagnostic ID as this will help Docker devs more.

iMerica commented 5 years ago

@jhaagmans

Your point about as resident memory vs virtual memory is a distinction without a difference. My operating system sees it as the highest consumer of memory, and also the highest consumer of battery power.

(Completely idle. 0 running Containers) $ ps -awxm -o %mem,rss,comm | sort -nr | head -5

22.3 3734748 com.docker.hyperkit
 7.5 1261564 ~/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/182.4505.26/PyCharm.app/Contents/MacOS/pycharm
 3.1 516608 /Applications/iTerm.app/Contents/MacOS/iTerm2
 1.9 319180 /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
 1.3 220628 /Applications/Google Chrome.app/Contents/Versions/69.0.3497.100/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper

kalexmills commented 5 years ago

Below is the output from the command @jhaagmans posted.

$  sudo ps -awxm -o %mem,rss,comm | sort -nr | grep docker
 0.2  36120 com.docker.vpnkit
 0.2  28236 com.docker.hyperkit
 0.1  17540 com.docker.osxfs
 0.1  16648 com.docker.driver.amd64-linux
 0.0   5516 /Applications/Docker.app/Contents/MacOS/com.docker.supervisor
 0.0      4 /Library/PrivilegedHelperTools/com.docker.vmnetd

Below is the output from top.

PID    COMMAND      %CPU  TIME     #TH   #WQ  #PORT MEM    PURG   CMPRS  PGRP  PPID  STATE    BOOSTS         %CPU_ME %CPU_OTHRS UID       FAULTS   COW
13399  com.docker.h 94.9  09:23:38 18/1  1    43    18G    0B     7778M  13393 13396 running  *0[1]          0.00000 0.00000    473958687 9743528+ 327

And here is a diagnostic ID from Docker output taken at roughly the same time: 3CBF0FFC-A3ED-4F24-944A-650842923449/20181017132639

This is all on a running Docker instance with a quiescent Kubernetes cluster which is running several containers all at once. No intensive processing is being performed by the cluster, (which was not the case when I started seeing performance issues).

@iMerica the output from the above commands seem like a big difference on my box. top is reporting 18G of memory usage, but ps is reporting only 36 MB or so of RSS (if I am reading it right). Could you post the output from top and ps taken within a few minutes of one another? I can't tell from your post what other values you are comparing the output from ps to.

I think the lesson here is that memory reporting varies greatly among different linux tools depending on what they are measuring. This SO question seems to have more to say also.

The diagnostic reports sent to the Docker team will (hopefully) be more useful to them than whatever we can determine on our own from our local machines... so please send them wherever possible

lookfirst commented 5 years ago

95FE31E3-5295-4781-AF08-69B23B71BD7D/20181019060130

bradrich commented 5 years ago

Another thing that Apple is not reporting in the Activity monitor that is causing issues is Spotlight indexing. I came to this realization due to the fact that my computer has 16GB of memory, and even though hyperkit is taking 4GB of it, that is still not a lot.

Go to System Preferences > Spotlight > Privacy and add your hard drives, etc. This will effectively remove Spotlight indexing. I've been running fine since then.

Flexicon commented 5 years ago

@bradrich what does this actually have to do with hyperkit and Docker for Mac though?

bradrich commented 5 years ago

@Flexicon Absolutely nothing. But, if you want to get your computer back to a functioning state, you might use the instructions that I provided, thank me, and then patiently wait for a fix for the main issue...

avxkim commented 5 years ago

@bradrich lol, you suggest to turn off spotlight indexing, so spotlight / Alfred won't work.

bradrich commented 5 years ago

@heihachi88 I would rather those systems not work, temporarily, and still be able to use my computer past 2 minutes after boot, than having to power cycle until my index finger goes numb.

Please understand that I offered a means to simply use your computer while these updates are being provided, not to fix the issue.

pkdetlefsen commented 5 years ago

I'm having the same issue after upgrading to Mojave 10.14. Docker would normally allocate 2 GB of memory on a fresh boot but after the upgrade it started using 3.5 GB.

Version: 18.06.1-ce-mac73 (26764) Engine: 18.06.1-ce Memory: 2 GB Swap: 1 GB

Edit:

With these settings the memory usage dropped to 2 GB:

Memory: 1 GB Swap: 0.5 GB

Diagnostics ID: B68D0BDE-9F20-46F8-92EA-E7A34BB85C31/20181102070743

emhagman commented 5 years ago

Same issue.

OSX: Mojave 10.14 Version: 18.06.1-ce-mac73 (26764) Engine: 18.06.1-ce Memory Setting: 6GB Swap Setting: 1GB

Used Memory in Activity Monitor With All Containers Running: 12GB

Virtual Memory: 5.76GB Real Memory: 2.69GB

Screenshot of Docker for Mac running without any running containers: capture d ecran 2018-10-23 a 12 58 14

docker / for-mac