search

docker / for-mac

Bug reports for Docker Desktop for Mac
https://www.docker.com/products/docker#/mac
2.43k stars 118 forks source link

Very slow network performance (host to container) #3497

Open AlexC opened 5 years ago

AlexC commented 5 years ago

Expected behavior

HTTP calls to docker container should respond quickly

Actual behavior

HTTP calls are very slow to respond. Compared to my Ubuntu 16.04 host whereby I get the ab command to complete in ~1 second, the Mac OSX host is taking ~17 seconds to complete.

Information

Diagnostic logs

Diagnose succeeded

Steps to reproduce the behavior

$ cat foo.php
<?php echo 'bar'; ?>
$ cat Dockerfile
FROM php:7.2-apache
COPY foo.php /var/www/html/
$ docker build --file Dockerfile -t slow-container .
$ docker run --rm -d -p 80:80 slow-container
$ ab -n5000 -c5 http://127.0.0.1/foo.php
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests

Server Software:        Apache/2.4.25
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /foo.php
Document Length:        3 bytes

Concurrency Level:      5
Time taken for tests:   17.705 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      975000 bytes
HTML transferred:       15000 bytes
Requests per second:    282.41 [#/sec] (mean)
Time per request:       17.705 [ms] (mean)
Time per request:       3.541 [ms] (mean, across all concurrent requests)
Transfer rate:          53.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    5   3.5      4      46
Processing:     2   10   4.5     10      69
Waiting:        1    8   4.1      7      58
Total:          4   15   5.1     15      71

Percentage of the requests served within a certain time (ms)
  50%     15
  66%     16
  75%     17
  80%     17
  90%     19
  95%     21
  98%     26
  99%     43
 100%     71 (longest request)
kazak1377 commented 5 years ago

Same here Docker desktop Version 2.0.0.3 (31259) Engine: 18.09.2 Compose: 1.23.2 Machine: 0.16.1

AlexC commented 5 years ago

Does anybody know of any further information that I can get to help debug this?

dave-redfern commented 5 years ago

Same versions as @kazak1377 ; I thought it could be because I run LittleSnitch, but it doesn't seem to have any impact and I did a full restart this morning. Memory usage is normal. I have seen that when the docker network starts slowing down, typically com.docker.vpnkit is consuming large amounts of RAM, but this is not the case (docker has only been running for an hour).

There is not very much in the system logs. The only thing I have found so far is a .diag report on power usage: com.docker.hyperkit_2019-03-06-075653_<host>.wakeups_resource.diag. I'm including it, but it doesn't seem particularly useful.

Command:         com.docker.hyperkit
Path:            /Applications/Docker.app/Contents/Resources/bin/com.docker.hyperkit
  5  vcpu_thread + 1213 (com.docker.hyperkit + 1871990) [0x103b1e076]
  4  xh_vm_run + 343 (com.docker.hyperkit + 1759971) [0x103b02ae3]
  4  vmx_run + 1718 (com.docker.hyperkit + 1722321) [0x103af97d1]
Powerstats for:  com.docker.hyperkit [1309]
Path:            /Applications/Docker.app/Contents/Resources/bin/com.docker.hyperkit
        5  vcpu_thread + 1213 (com.docker.hyperkit + 1871990) [0x103b1e076]
          4  xh_vm_run + 343 (com.docker.hyperkit + 1759971) [0x103b02ae3]
            4  vmx_run + 1718 (com.docker.hyperkit + 1722321) [0x103af97d1]
          1  xh_vm_run + 1335 (com.docker.hyperkit + 1760963) [0x103b02ec3]
        1  callout_thread_func + 217 (com.docker.hyperkit + 1767299) [0x103b04783]
          1  vlapic_callout_handler + 92 (com.docker.hyperkit + 1744908) [0x103aff00c]
            1  vcpu_notify_event + 104 (com.docker.hyperkit + 1756334) [0x103b01cae]
           0x103955000 -        0x103b97fff  com.docker.hyperkit (0)             <71246D3E-B4F4-34BA-9BE4-201218570ACE>  /Applications/Docker.app/Contents/Resources/bin/com.docker.hyperkit
AlexC commented 5 years ago

We've been able to try on a few more Macbooks all with similar result on most. The only thing that connects them is that they are all a corporate Macbook that has been tweaked slightly, though some of them work OK and the test completes in around 3.5 seconds.

One thing we have noticed is that if you run the tests shorty after each other then performance gets slower and slower. e.g. run the test and it takes 3.5 seconds, wait 5 seconds before trying and it will take 6-7 seconds, wait 5 seconds again then it'll take 12 seconds. If you then give it some time to rest, it'll be back at 3.5 seconds

docker-robott commented 5 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

randvoorhies commented 5 years ago

/remove-lifecycle stale

blwsh commented 5 years ago

Docker version 19.03.0-rc2, build f97efcc docker-compose version 1.24.0, build 0aa59064

I ran this command inside an ubuntu container:

curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -

From within the container i got about 120mbps

When I run on the host i get about 500mbps. Why is the container so much slower? I am using an external network but didn't think it would have that much of a hit.

docker-robott commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

Geod24 commented 4 years ago

/remove-lifecycle stale

docker-robott commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

Geod24 commented 4 years ago

/lifecycle frozen

Geod24 commented 4 years ago

/remove-lifecycle stale

fourgates commented 4 years ago

I have experienced the same issue.

Diagnosis ID: 54A0E2A5-DFEA-4380-8AAC-AABF6C10E66B/20200124232052

I use Docker on my work PC with no issues. I recently started developing on an iMac at home and it takes significantly longer to serve up the same main.js file. No issues when working on the PC. Same code, same images/containers, and I think my iMac actually has a much more powerful processor and more RAM.

blwsh commented 4 years ago

I have experienced the same issue.

Diagnosis ID: 54A0E2A5-DFEA-4380-8AAC-AABF6C10E66B/20200124232052

I use Docker on my work PC with no issues. I recently started developing on an iMac at home and it takes significantly longer to serve up the same main.js file. No issues when working on the PC. Same code, same images/containers, and I think my iMac actually has a much more powerful processor and more RAM.

Very much a known separate issue.

wader commented 4 years ago

I can reproduce this with docker for mac 2.2.0.0 (42247)

main.go

package main

import (
    "io"
    "net/http"
)

func main() {
    http.ListenAndServe(":8080", http.HandlerFunc(func (w http.ResponseWriter, r *http.Request) {
        io.Copy(w, r.Body)
    }))
}
$ docker run --rm -ti -v $PWD:$PWD -w $PWD -p 8080:8080 golang:1.13 go run main.go

From host

$ dd if=/dev/zero bs=1000000 count=100 | curl --data-binary @- 0:8080 | dd bs=1000000 of=/dev/null
100+0 records in
100+0 records out
100000000 bytes transferred in 0.072270 secs (1383701612 bytes/sec)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  190M    0 95.3M  100 95.3M  15.0M  15.0M  0:00:06  0:00:06 --:--:-- 15.0M
0+5975 records in
0+5975 records out
100000000 bytes transferred in 6.401760 secs (15620704 bytes/sec)

From inside container:

root@1b0f3c04b6f8:/go# dd if=/dev/zero bs=1000000 count=100 | curl --data-binary @- 0:8080 | dd bs=1000000 of=/dev/null
100+0 records in
100+0 records out
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100000000 bytes (100 MB, 95 MiB) copied, 0.330937 s, 302 MB/s
100  190M    0 95.3M  100 95.3M   262M   262M --:--:-- --:--:-- --:--:--  525M
0+1726 records in
0+1726 records out
100000000 bytes (100 MB, 95 MiB) copied, 0.71619 s, 140 MB/s
Cospel commented 4 years ago

The same issue here, I need to install the older 2.1.0.1 version docker for-mac.

cr-lgl commented 4 years ago

I'm having the same problem on a Docker for Mac.

Why is it slow when approaching the host?

davisengeler commented 4 years ago

I'm having this issue. Services running inside Docker containers on macOS have very slow network speeds. This happens when running on two different MBPs.

Original Edit: Scratch that [maybe?]... my problem was with something else

Long edit: I played with it more and it seems to be caused by an AT&T Fiber router (Pace something). When I posted, I hadn’t noticed the same container had fast network speed when I wasn’t using it via a port forwarded by the Pace. I’m now mostly bypassing the Pace router and forwarding the ports with my own router and my particular problem is gone.

New Edit: The comment by slowr below makes me think my problem was still related to this overall issue. Not totally sure why a different router would avoid the bug, but maybe that's a clue? lol

cr-lgl commented 4 years ago

I removed the volume from my docker-compose, and tried Mutagen.
Then my slow network improved.
I've read that Docker volume is slow on a Mac, but I don't know what it has to do with the network.

However, this approach requires writing separate docker-compose files and Mutagen scripts in the development environment.
I look forward to coding more happily on the Mac...

soccer99 commented 4 years ago

All the requests to my container have about 14 seconds of latency added just in the network request.

slowr commented 4 years ago

I can reproduce this with docker for mac 2.2.0.0 (42247)

main.go

package main

import (
  "io"
  "net/http"
)

func main() {
  http.ListenAndServe(":8080", http.HandlerFunc(func (w http.ResponseWriter, r *http.Request) {
      io.Copy(w, r.Body)
  }))
}
$ docker run --rm -ti -v $PWD:$PWD -w $PWD -p 8080:8080 golang:1.13 go run main.go

From host

$ dd if=/dev/zero bs=1000000 count=100 | curl --data-binary @- 0:8080 | dd bs=1000000 of=/dev/null
100+0 records in
100+0 records out
100000000 bytes transferred in 0.072270 secs (1383701612 bytes/sec)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  190M    0 95.3M  100 95.3M  15.0M  15.0M  0:00:06  0:00:06 --:--:-- 15.0M
0+5975 records in
0+5975 records out
100000000 bytes transferred in 6.401760 secs (15620704 bytes/sec)

From inside container:

root@1b0f3c04b6f8:/go# dd if=/dev/zero bs=1000000 count=100 | curl --data-binary @- 0:8080 | dd bs=1000000 of=/dev/null
100+0 records in
100+0 records out
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100000000 bytes (100 MB, 95 MiB) copied, 0.330937 s, 302 MB/s
100  190M    0 95.3M  100 95.3M   262M   262M --:--:-- --:--:-- --:--:--  525M
0+1726 records in
0+1726 records out
100000000 bytes (100 MB, 95 MiB) copied, 0.71619 s, 140 MB/s

Still reproducible on 2.2.0.5 release. It seems that every connection going to the container through the port forward makes it 10-20 times slower.

davisengeler commented 4 years ago

I ran into another issue with Docker for Mac and I'm curious if it may be related to what we're seeing here. If not, sorry to muddy up the conversation.

The open issue https://github.com/docker/for-mac/issues/180 made me realize that Docker for Mac in particular has some pretty weird behavior related to networking (I think due to hyperkit). Docker's documentation about this is pretty lacking, but there are some other writeups that may be worth checking into. To me, this all kinda seems like a bug related to the virtualization happening on the macOS version.

But this is probably not news to any of you hahah.

paales commented 4 years ago

We've been analyzing network latency for our docker development box we've been trying to create:

Benchmarks with Docker latest: https://github.com/ho-nl/docker-development-box/issues/10#issuecomment-638677595

That means:

Benchmarks with Docker 2.1.0.1: https://github.com/ho-nl/docker-development-box/issues/10#issuecomment-639384629

If we subtract the local TCP (0.281) from the 0.4 we get about 0,1ms of latency, which seems acceptable. (We still need to figure out why mysql is responding so much slower compared to a dedicated server, but that's another discussion.)

It seems there is a huge network latency overhead when running with the latest docker version which wasn't there when running the older docker version.

dylan-chong commented 4 years ago

Have done some debugging, and in our set up using Elixir/Phoenix API, the network delay of roughly 15 seconds isn't caused by Docker network itself. It is caused by Phoenix checking for files changed ( Presumably source files and dependencies) when it receives a network request plug(Phoenix.CodeReloader). disabling this avoids the problem - indicating that it is the problem with Docker file system.

pdefreitas commented 4 years ago

I can confirm that downgrading to Docker 2.1.0.1 increases substancially the performance in my use case (Wordpress). I've noticied the decrease of performance after upgrading my Docker setup to the latest release even through I'm using docker-sync due to the poor performance of volume mapping in Docker for Mac.

eak24 commented 4 years ago

I can confirm that downgrading to Docker 2.1.0.1 increases substancially the performance in my use case (Wordpress). I've noticied the decrease of performance after upgrading my Docker setup to the latest release even through I'm using docker-sync due to the poor performance of volume mapping in Docker for Mac.

This did not change performance for me at all. I am still seeing extremely slow performance on 2.1.0.1

reduardo7 commented 4 years ago

???????

4np commented 3 years ago

I'm seeing the docker container being able to use just about 1/5th of the total available network throughput using the SABnzbd+ docker container.

kevinhq commented 3 years ago

Run through this problem too. Killing/restaring docker desktop works. But, it's too much hurdle.

jbarop commented 3 years ago

Downloading Debian packages with modem speeds... Pure nostalgia

pwais commented 3 years ago

I am seeing a similar issue but for local disk access. Writes to the in-container filesystem can achieve well over 100MBytes / sec while writes to the same physical disk but using the volume-mounted host_mnt directory can only get about 50MBytes / sec. This is on a new 2020 iMac with Apple NVMe SSD. I see a similar cap using a Thunderbolt external drive that is also volume-mounted into the container.

While this Github Issue focuses on networking, I hypothesize that the hyperkit hypervisor is interfering with any I/O.

shaunhp commented 3 years ago

Same issue here. Host write performance to my NAS is around 770Mb/s. Write performance to same target from docker on that host is 230Mb/s. Not fun when you're pushing 70GB files. (Mac Mini/Catalina)

joshspicer commented 3 years ago

+1 to this, although i'm seeing it from container -> internet.

Consistently spinning up a container in a docker-compose scenario with no custom networking config

root@6cc6a923ef59:/# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=4 ttl=37 time=1920 ms <-----
64 bytes from 1.1.1.1: icmp_seq=5 ttl=37 time=880 ms
64 bytes from 1.1.1.1: icmp_seq=6 ttl=37 time=11.3 ms
64 bytes from 1.1.1.1: icmp_seq=7 ttl=37 time=13.5 ms

apt update rarely succeeds as it times out

gonzofish commented 3 years ago

While this Github Issue focuses on networking, I hypothesize that the hyperkit hypervisor is interfering with any I/O.

Me too @pwais. I have a MongoDB container that, on >= v3.2.0 complains with serverStatus was very slow. If I downgrade to 3.1.0, there's no error, though.

artheus commented 3 years ago

This is a problem for me, as well. Though, I am running the Docker VM through docker-machine and virtualbox. So I do not think that this is a problem tied only to docker/for-mac.

dylan-chong commented 3 years ago

If you are sharing the folder between host and VM then FS performance will probably not be fast. See if things improve if you don’t use shared folders

On 22/05/2021, at 1:13 AM, Morten Hekkvang @.***> wrote:

 This is a problem for me, as well. Though, I am running the Docker VM through docker-machine and virtualbox. So I do not think that this is a problem tied only to docker/for-mac.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

blwsh commented 3 years ago

The problem seems to have been resolved (for me at least)

From the host:

Retrieving speedtest.net configuration...
Testing download speed................................................................................
Download: 274.02 Mbit/s
Testing upload speed................................................................................................
Upload: 36.29 Mbit/s

and from the container (My internet fluctuates):

Retrieving speedtest.net configuration...
Testing download speed................................................................................
Download: 318.35 Mbit/s
Testing upload speed......................................................................................................
Upload: 39.14 Mbit/s

Commands used:

curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python - # On the host
docker run --rm -it python sh -c "curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -" # On the container
tomav commented 2 years ago

Using @blwsh commands, from the host:

Testing download speed................................................................................
Download: 888.99 Mbit/s
Testing upload speed................................................................................................
Upload: 366.19 Mbit/s

From Docker for Mac 3.6.0:

Testing download speed................................................................................
Download: 548.20 Mbit/s
Testing upload speed......................................................................................................
Upload: 156.35 Mbit/s

From Docker for Mac 4.1.0:

Testing download speed................................................................................
Download: 521.60 Mbit/s
Testing upload speed......................................................................................................
Upload: 170.52 Mbit/s

Still have a gap here compared to host. Both Docker versions have similar results.

artheus commented 2 years ago

The problem seems to have been resolved (for me at least)

From the host: ...


docker run --rm -it python sh -c "curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -" # On the container```

I am unable to confirm this as I am running on an enterprise environment, where all www traffic has to go through a proxy server (w/o support for ICMP)

zfil commented 2 years ago

From the host Download: 912.78 Mbit/s Upload: 391.63 Mbit/s

From docker for mac 4.5.0 Download: 475.84 Mbit/s Upload: 355.90 Mbit/s

tomav commented 2 years ago

From the host (OSX)

Testing download speed................................................................................
Download: 739.55 Mbit/s
Testing upload speed................................................................................................
Upload: 292.14 Mbit/s

Docker Desktop 4.6.0 (75818)

Testing download speed................................................................................
Download: 523.70 Mbit/s
Testing upload speed......................................................................................................
Upload: 302.98 Mbit/s

Still a significant difference on download (tried multiple times).

hyzyla commented 2 years ago

Apple Macbook M1 2022:

From the host:

Hosted by Techgroup Jakub Osuch (Warszawa) [2.12 km]: 7.73 ms
Testing download speed................................................................................
Download: 186.38 Mbit/s
Testing upload speed......................................................................................................
Upload: 16.23 Mbit/s

Docker Desktop 4.6.1 (76265)

Hosted by Techgroup Jakub Osuch (Warszawa) [2.12 km]: 2512.785 ms
Testing download speed................................................................................
Download: 2.35 Mbit/s
Testing upload speed......................................................................................................
Upload: 2.80 Mbit/s
Giuseppetm commented 1 year ago

This is still an issue ..

nothing2obvi commented 1 year ago

On Mac Mini with M1 chip, Ventura 13.2.1, Docker Desktop 4.17.0, using Virtualization Framework:

From the host:

Testing download speed................................................................................
Download: 442.37 Mbit/s
Testing upload speed.....................................................................................................
Upload: 32.35 Mbit/s

From Docker container:

Testing download speed................................................................................
Download: 55.03 Mbit/s
Testing upload speed......................................................................................................
Upload: 15.97 Mbit/s
nothing2obvi commented 1 year ago

Can confirm that completely quitting Docker Desktop and re-opening it does bring some improvement, but still not perfect. I tried turning off Virtualization Framework, too, and it was much worse on my Mac Mini running Ventura. Interestingly, on my older Mac Mini running Monterey instead of Ventura, the host and container results are comparable, also with no difference when using Virtualization Framework.

Mac Mini with Apple M1 chip (2020), Ventura 13.2.1, Docker Desktop 4.17.0

With Virtualization Framework

From the host:

Testing download speed................................................................................
Download: 609.29 Mbit/s
Testing upload speed......................................................................................................
Upload: 41.24 Mbit/s

From Docker container:

Testing download speed................................................................................
Download: 158.45 Mbit/s
Testing upload speed......................................................................................................
Upload: 42.42 Mbit/s

Without Virtualization Framework

From Docker container:

Testing download speed................................................................................
Download: 32.08 Mbit/s
Testing upload speed......................................................................................................
Upload: 32.95 Mbit/s

Mac Mini (2018), Monterey 12.6, Docker Deskto 4.17.0

Note that this Mac is further away from my router.

With Virtualization Framework

From the host:

Testing download speed................................................................................
Download: 82.91 Mbit/s
Testing upload speed......................................................................................................
Upload: 39.28 Mbit/s

From Docker container:

Testing download speed...............................................................................
Download: 79.45 Mbit/s
Testing upload speed......................................................................................................
Upload: 45.88 Mbit/s

Without Virtualization Framework

From Docker container:

Testing download speed................................................................................
Download: 86.66 Mbit/s
Testing upload speed......................................................................................................
Upload: 40.99 Mbit/s

Is it Ventura? Not sure what's going on here.

nothing2obvi commented 1 year ago

Upgraded to 4.18.0 and my speeds have increased dramatically. I'm happy with this.

From the host:

Testing download speed................................................................................
Download: 653.12 Mbit/s
Testing upload speed......................................................................................................
Upload: 29.14 Mbit/s

From Docker container:

Testing download speed................................................................................
Download: 561.10 Mbit/s
Testing upload speed......................................................................................................
Upload: 40.34 Mbit/s
nothing2obvi commented 1 year ago

Upgraded to 4.19.0 a few weeks ago and this issue has pretty much resolved for me.