CPU Usage Still High on Docker 3.0.0

[pjjanak]

Seems like the issue described in #5044 is persisting even after the latest upgrade. I have removed unneeded shares (as mentioned in #3499) but that has not fixed the problem.

• [x ] I have tried with the latest version of my channel (Stable or Edge)
• [x ] I have uploaded Diagnostics
• Diagnostics ID: 5F7920E2-CFFC-44A8-9509-AE9C0DDF6CE2/20201210183007

Expected behavior

com.docker.hyperkit takes up a normal amount of CPU

Actual behavior

com.docker.hyperkit is constantly running at > 100% CPU

Information

Please, help us understand the problem. For instance:

• Is it reproducible? - Yes
• Is the problem new? - No (probably not given the amount of times this has been reported)
• Did the problem appear with an update? - Yes after the 3.0.0 update the problem resurfaced
• macOS Version: 10.15.7

Steps to reproduce the behavior

1. Start docker

[ricristian]

Now it's worst

OS: Catalina 10.15.7 Diagnostic ID: 152E874F-49F1-48A0-9A4E-30D9B88D56EA/20201210203925 File sharing is not working anymore if gRPC is enabled.

Settings: Experimental: all disabled

docker-compose.yml

version: "2"

services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    volumes:
      - "./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro"
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.2.0
    volumes:
      - "./kibana.yml:/usr/share/kibana/config/kibana.yml:ro"
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch
  filebeat:
    container_name: filebeat
    hostname: filebeat
    user: root
    image: "docker.elastic.co/beats/filebeat:7.2.0"
    volumes:
      #Mount the filebeat configuration so users can make edit
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
      #Mount the prospectors directory. Users can in turn add propspectors to this directory and they will be dynamically loaded
      - "../logs/:/usr/share/logs/"
    networks:
      - elk
    command: filebeat -e
    restart: on-failure

networks:
  elk:
    driver: bridge

EDIT: after 10-15 minutes of continuously "burning" CPU usage it seems that com.docker.hyperkit is eating between 8-30% CPU with 3 containers running ( kibana, Elasticsearch, filebeat ) That spike of 10-15 minutes of high CPU usage is triggered by docker-compose which creates 3 containers.

[stephen-turner]

@pjjanak @ricristian Can you clarify whether you're also seeing high CPU load when no containers and no Kubernetes are running, or only when there are running containers? Those would likely have different causes. Thanks.

[ricristian]

@stephen-turner I can confirm that by default when all containers are stopped the load is small under 10% When I'm starting containers with docker-compose ( in my case ) for 10-15 minutes the load increases to between 100-500% and after those 10-15 minutes the load decreases back to 10% which is really great. So the high load appears only when I'm trying to start containers and after that load decreases to more than decent

EDIT:

I've made now more tests and I've discovered following: UseCase 1. When I try to start some containers with images that are not already downloaded locally the entire time ( download + build + run ) (about 10-15 minutes) cpu load increases

After those 10-15 minutes everything is downloaded + started cpu decreases to 10%

UseCase 2. When I try to start same containers that are already downloaded cpu load increases for few seconds ( under 1 minute ) probably until containers finish loading some settings

After those few seconds cpu load decreases again to 10% and stays in 10-30%

All the tests were realised using docker compose mention earlier ( 3 containers - es - kibana - filbeat )

Maybe I was to hurry to report this but if we ignore the start-up. time of containers which generates high cpu load this release improves overall cpu usage.

[stephen-turner]

@ricristian Thanks, that's very interesting. I'm still surprised the images take quite so long to download: are they enormous?

[ricristian]

@stephen-turner it's highly probable that it is caused by my network ... sometimes bandwidth is not that high ...

[pjjanak]

@stephen-turner - I noticed this right after docker restarted after update, so no containers were running. I also don't use kubernetes on this machine, so don't have any running on here ever. I will say that things have quieted down a little since reporting, but my usage is still hovering around 70-80% with no containers running.

[osi]

I'm seeing the same observation as https://github.com/docker/for-mac/issues/5070#issuecomment-730235895 w/3.0.1 on Big Sur, 11.0.1 (20B50)

[pposca]

I have the same issue, but it only happens when there is a volume bound:

• Container with volume bounded: ~20% CPU usage
• Same container without volume bounded: ~4% CPU usage

[dawi]

After update to 3.0.0 the cpu usage is around 80 % without running any containers. With running containers it's constantly over 200 %.

[alexcouret]

Having the same issue, upgraded to v3 and CPU got back to >200% usage. Diagnostic ID: 74807466-79B8-4824-8035-608D77708609/20201214144155

Version 2.3 is the latest version that doesn't have this issue for me.

[Patrick-Remy]

HyperKit takes about 200% CPU (no K8s) on 3.0.1 without any containers running – exactly the same as at 2.5.0.

[Ky6uk]

Yep, it's back, but not all the time as it was on v2.5. Sometimes it's eating all attached CPUs, sometimes not. I have no idea about the condition, it looks random at the moment.

C1E3D580-6312-45F9-8E7F-B7D86211FCF9/20201215122501

[YotillaMarco]

I habe the same problem. ~170% CPU usage without any container running. Version 3.0.1 on macOS 10.15.7

[cyberatz]

Similar issue here. I stopped all container but one - watchtower - which is set to run at 15min intervals.

CPU usage of docker:

CPU usage of the container:

These are the hyper kit process stats:

My gut feeling currently is that there's something running aggressively inside the docker created VM (~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw) that is eating up cycles - but this is just a hypothesis at the moment and I've no idea how to test that theory just yet.

Diagnostic ID: 55ADF900-D456-4893-BC2B-812E6F6622DD/20201218084210

EDIT: Upon further troubleshooting (and after killing all Kubernetes instances as well. The thing that seems to be using the more CPU in the docker-desktop vm is etcd (etcd --advertise-client-urls=https://192.168.65.3:2379 --cert-file=/run/config/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://192.168.65.3:2380 --initial-cluster=docker-desktop=https://192.168.65.3:2380 --key-file=/run/config/pki/etcd/server.key --listen-cli)

so not sure if something can be tweaked here to see if it's the culprit (assuming my hypothesis of the VM being the issue). That said, it could also be something with HyperKit which could be causing the issues here.

EDIT: Screw that. I blew away the existing docker-desktop VM and rebuilt it. With only one container running (watchtower) I'm down to ~3% CPU utilization at idle which I think is totally acceptable. Perhaps just rebuilding the VM could help folks here too. (stopped Docker, deleted Docker.raw, restarted Docker). Keep in mind that this could be super destructive - so I guess - only try this at home.

[n00bvn]

I've just updated to the latest version which is 3.0.2 but hyperkit cpu usage still at more than 200%. I'm using Big Sur 11.1.

[NaeemSohrabi]

Hi, i have the same problem of high cpu usage abut 320 to 350 % OS : Big Sur 11.1. Docker: 3.0.2

docker-compose.yml

version: "3.2"

services:
  node:
    build:
      dockerfile: node_watch.Dockerfile
      context: ./logger
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    volumes:
      - ./logger/public/:/var/www/logger/public/
      - ./logger/resources/:/var/www/logger/resources/
      # - ./logger/node_modules:/var/www/logger/node_modules
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    container_name: node
    networks:
      - mavara-network

  laravel_echo_server:
    build:
      dockerfile: Dockerfile
      context: ./echo
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    restart: unless-stopped
    container_name: laravel_echo_server
    volumes:
      - ./echo/laravel-echo-server.json:/opt/echo/laravel-echo-server.json
      - ./echo/certs:/opt/echo/certs/
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro

    ports:
      - 6001:6001
    depends_on:
      - php
      - redis
    networks:
      - mavara-network

  mysql:
    image: ${REGISTRY_URI}/mysql:5.7
    container_name: mysql
    restart: unless-stopped
    volumes:
      - ./mysql/init.sql:/docker-entrypoint-initdb.d/init.sql
      - ./mysql/data:/var/lib/mysql
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    env_file: ./mysql/.env
    networks:
      - mavara-network

  nginx:
    image: ${REGISTRY_URI}/nginx:1.19.2
    container_name: nginx
    restart: unless-stopped
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf
      - ./logger/:/var/www/logger
      - ./nginx/cert:/etc/nginx/certs/
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    ports:
      - 80:80
      - 443:443
    depends_on:
      - php
    networks:
      - mavara-network

  redis:
    image: ${REGISTRY_URI}/redis:6.0.8
    container_name: redis
    restart: unless-stopped
    networks:
      - mavara-network
    volumes:
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro

  php:
    restart: unless-stopped
    build:
      dockerfile: php_dev.Dockerfile
      context: ./logger
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    container_name: php
    hostname: php
    environment:
      SOURCE_DIR: ${SOURCE_DIR}
      HOST_USER: ${HOST_USER}
      HOST_ADDR_NAME: ${HOST_ADDR_NAME}
    volumes:
      - ./logger/local.ini:/usr/local/etc/php/conf.d/local.ini
      - ./logger/:/var/www/logger/
      - ~/.ssh/:/root/.ssh/
      - /etc/sudoers:/var/www/logger/sudoers
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    env_file:
      - ./logger/.env.dev
    depends_on:
      - mysql
      - redis
    networks:
      - mavara-network

  snmp-server:
    build:
      dockerfile: Dockerfile
      context: ./snmp
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    restart: unless-stopped
    container_name: snmp-server
    volumes:
      - /proc:/proc
      - ./snmp/snmpd.conf:/etc/snmp/snmpd.conf
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    networks:
      - mavara-network
  reporter:
    restart: unless-stopped
    build:
      context: ./pdf-reporter
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    container_name: reporter
    networks:
      - mavara-network
# Docker Networks
networks:
  mavara-network:
    driver: bridge

[ishmayeck]

Seeing the same with 3.0.3 on macOS 11.1. Just did a complete clean install of Docker Desktop, no containers running at all, and com.docker.hyperkit is hovering around 200% CPU usage.

Diagnostic ID: 0789f91c-d606-44fe-b1b9-c4363bbe7825/20201221191457

[fatroom]

Same problem. OSX 11.1 + docker 3.0.3 eats 130% CPU. Diag ID: 22478378-9FD7-4163-ACB5-21C629CD6CCC/20201228181903

[casprwang]

I suspect this is related to https://github.com/docker/roadmap/issues/12

Also, rolling back to 2.3.0.5 worked for my Big Sur(20C5048l).

[richyen]

@pjjanak @ricristian Can you clarify whether you're also seeing high CPU load when no containers and no Kubernetes are running, or only when there are running containers? Those would likely have different causes. Thanks.

I recently upgraded to 3.0.1 on Mac OS Catalina, and I am seeing hyperkit consume 86-100% of cpu even while there are no containers running (docker ps -a is blank)

[jasonlancaster]

Similar issues here. I upgraded from Catalina/3.0.1 -> Big Sur 11.1/3.0.3 and am seeing hyperkit CPU in constant load again ~30%. Disabling turbo boost seems to be the only method to control this.

[jeitnier]

Same here, Docker version 3.0.4 (51218) on Big Sur 11.2 Beta and hyperkit consistently ~206% CPU.

[fuxingloh]

TLDR: The same issue for me but figured external Docker daemon connection caused the high CPU usage.

Experienced the same issue when I upgraded to docker 3.0.4 from 3.0.3. Using macOS Catalina, com.docker.hyperkit was using 150%+ CPU. Stopped everything, restarted, pruned and downgraded to 2.5.1 all didn't help and CPU usage was still around 150%+. I was using IntelliJ with Docker plugin at that time with IntelliJ connected to Docker daemon (was connected during the upgrade as well) I assumed external connection caused the high CPU. Restarted IntelliJ and no issue after that.

You can easily reproduce this by running Docker through JetBrains products and have it attached during the restart or upgrade. The CPU usage will sit at around ~50-175%.

[richyen]

@pjjanak @ricristian Can you clarify whether you're also seeing high CPU load when no containers and no Kubernetes are running, or only when there are running containers? Those would likely have different causes. Thanks.

I recently upgraded to 3.0.1 on Mac OS Catalina, and I am seeing hyperkit consume 86-100% of cpu even while there are no containers running (docker ps -a is blank)

The worst part of this is that if I forget to quit Docker for Mac, and I walk away from my laptop, I'll come back to a laptop with 7% battery, even when it's plugged in the whole time.

[sidler]

TLDR: Same issue for me but figured external Docker daemon connection caused the high CPU usage.

Experienced the same issue when I upgraded to docker 3.0.4 from 3.0.3. Using macOS Catalina, hyperkit was using 150%+ CPU. Stopped everything, restarted, pruned and downgraded to 2.5.1 all didn't help and CPU usage was still around 150%+. I was using IntelliJ with Docker plugin at that time with IntelliJ connected to Docker daemon (was connected during the upgrade as well) I assumed external connection caused the high CPU. Restarted IntelliJ and no issue after that.

This. When restarting Docker for Mac (e.g. due to an update) a previously opened Docker-Service connection from IntelliJ (here: PHPStorm) tried to reconnect instantly, causing high cpu loads on hyperkit. Stopping the Docker-Service Plugin in PHPStorm stopped the cpu spike and everything went back to normal cpu load.

[ishmayeck]

Indeed, this was my issue as well! Disconnecting and reconnecting the Docker plugins in WebStorm does the trick, and brings they hyperkit process to a much more reasonable <10%. Good find!

[westonruter]

Disconnecting and reconnecting the Docker plugins in WebStorm does the trick

By disconnecting and reconnecting, does this refer to disabling and enabling the Docker plugin?

[ishmayeck]

Nah, just disconnecting it in the Services pane is enough:

[wodka]

Disabled the IntelliJ Plugins but this did not change the CPU usage :(

[mike-potter]

Just updated to Big Sur 11.1 (from Catalina) and to Docker Desktop 3.0.4 (from the latest 2.4 version) and am running into this also. Except in my case the Activity Monitor is showing com.docker.backend as using 100% CPU. com.docker.hyperkit is fine (4%). All with no containers running.

I had been working with containers when I ran into this, but not doing anything with xdebug etc. Not using IntelliJ (but was using VSCode, just not debugging). Shut down VSCode and then shut down all containers. The 100% CPU for com.docker.backend persists.

This is the 2nd day in a row this has happened (upgraded to Big Sur 2 days ago). Diagnostics uploaded to 1747679A-0414-4CE2-A8B0-CB28CFAA217C/20210113163627

[SHornung1]

@mike-potter I had a similar issue and solved it following: https://github.com/docker/for-mac/issues/5200

Maybe this also solves your issue.

[mike-potter]

I'll try disabling gRPC Fuse and also updating to v3.1.0 to see if that helps.

[mike-potter]

Unfortunately turning off gRPC Fuse caused some of my docker scripts to hang. Specifically I have a bash script that does:

docker run -it --rm -v ${HOME}:/root -v ${PROJECT_ROOT}:/home/git -w ${workdir} outrigger/lab mr create ...

and that would hang. Turned gRPC Fuse back on and then it worked again. So I won't be able to determine if this helps with the performance issues.

But as soon as I turned gRPC back on and restarted Docker Desktop and then ran my script I started seeing com.docker.backend running 100% again. So updating to v3.1.0 didn't help.

[mike-potter]

Oh, well, DOH! I just noticed that docker run command is trying to mount ${HOME}! That is likely the cause of my performance issue. That really shouldn't be necessary.

Edited: Confirmed that removing this and just mounting the specific folder(s) needed fix my specific performance issue. Still not sure why this wasn't a performance problem in Docker Desktop v2.5, so might still be something to look into. I can see use-cases where people might want to share their entire HOME folder into a docker container.

[benlpvt]

I had just installed/tried out Docker today on my Macbook for the first time. I had the same sorts of issues, especially with com.docker.backend spiking the CPU. It started as soon as I started up a container, and I had used /Users/username/: as a mount point for /root/shared. I am grateful to have seen this thread before too long. I just changed the mount point to a more specific directory inside my ${HOME} directory and the com.docker.backend issues are gone. Thanks for posting above, and hope this holds!

[jeitnier]

I mounted only the directory I needed and still hover around ~150%, occasionally dipping down to 68% but right back up again, even at idle. This does seem to offer slight improvement as I was previously holding steady at ~200%+ with default directories mounted, but this still seems like a huge issue.

[kraftartberlin-micha]

same problem here, hyperkit has more then 170% usage, still after restart with no containers. after quit docker for mac, hyperkit still burning cpu. i like the comment mac is like a airplane, here same. again restart no containers again high cpu :( my last changes:

• docker-update
• phpstorm-integration
• mount changed in compose from standard to cached, ro and delegated

[yreifschneider]

I encountered the same issue wirh 170% CPU usage with no containers running. Looking at Console.app I saw many requests to the Docker API. The CPU usage went down after disabling IntelliJ IDEA Docker integration. It looks like IntelliJ has some sort of polling loop which went crazy during the Docker Desktop update.

[kraftartberlin-micha]

yes, thats it. i removed docker from phpstorm-settings and it works! i startet docker again and runned some unittests, under 30%. create cache by first run symfony app(browserrequest) put the usage up to 170 and more, but fall down quick after hes finished i also can change any option, cpu doesnt rise up, so it looks good i dont need the intellj integration thx yreif

[westonruter]

@yreifschneider @kraftartberlin-micha Did you disable the Docker plugin as I mentioned in https://github.com/docker/for-mac/issues/5116#issuecomment-758122403 or did you disconnect Docker from the Services pane as in https://github.com/docker/for-mac/issues/5116#issuecomment-758223937?

[jeitnier]

I never had Docker service integrated with my PhpStorm and still have extremely high CPU. Tried to backport to 2.5.1 and 3.0 and same issues consistently while containers are running at idle.

[kraftartberlin-micha]

i removed the deamon integration here , nothing more.

im happy :)

now max 8% cpu usage when running in background with 4 containers

hyperkit get closed very fast now

[yreifschneider]

@westonruter disconnecting the IntelliJ docker service was enough for me to lower the CPU usage to a normal limit for me. I did not uninstall the plugin completely.

[soltmar]

I can confirm problem with PhpStorm docker plugin. When removed connection, CPU usage came back to normal 🥳

[collinalexbell]

This issue is legendary. The previous thread had 600+ comments, #3499

[kraftartberlin-micha]

yep, seems that no one tried out to remove docker from phpstorm ^^ (maybe both logviewers or something else is blocking)

[stephen-turner]

FYI We are in discussion with JetBrains on this.

[yole]

FYI the tracking issue on the JetBrains side is https://youtrack.jetbrains.com/issue/IDEA-215742. Please vote for it if you experience high CPU consumption of dockerd when connecting to it from the Docker plugin in a JetBrains IDE.

[stephen-turner]

JetBrains have replied on their ticket:

For 2021.1 we are completely rewriting the integration of the docker plugin with the Services view, which will start to rely on docker events subscription instead of permanent polling of the engine state. This should dramatically improve the performance of the integration and should remove the pressure put on the Docker engine by polling.

Also, we have recently backported to 2020.3.x stream the fix for IDEA-237744, which should prevent the potential burst of the plugin activity in some edge cases related to the communication problems between IDE and engine.

https://youtrack.jetbrains.com/issue/IDEA-215742#focus=Comments-27-4653800.0-0

[deitch]

Ugh. I was experiencing this, found this thread, disabled gRPC for Fuse, and I thought it went away. But it came back. I love watching docker eat up 100% of 1 CPU with zero containers running.