search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
756 stars 85 forks source link

Slow networking performance with port mapping #169

Open nerdoftech opened 6 years ago

nerdoftech commented 6 years ago

Expected behavior

Apps in docker container should run at normal performance levels.

Actual behavior

It appears that the port mapping is creating a significant performance hit.

Steps to reproduce the behavior

I was running nginx in a container with port mapping and it was painfully slow, sometimes taking 5-10 seconds for webpages to load.

However, I have my containers set to use a linux bridge on the host and that bridge is bridged to a second interface on an internal lab network. That lab network has it's own NATing router, so I port forwarded from the router VIP to the container's direct IP address in the lab network. With that it works at a performance level that should be expected.

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:23 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:49 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.09.0-ce
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-693.5.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51GiB
Name: localhost
ID: RWCC:B6FU:UWKQ:RQE5:S7Y7:PPFH:Z5PN:CTKL:RPUF:XG2L:JNOJ:ZIZG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

[root@host nginx]# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core)

[root@host nginx]# docker inspect nginx_nginx_1
[
    {
        "Id": "4a254ab57792d53e15683cc7eb56148de9f1d8254dda8b480b769f761bcca243",
        "Created": "2017-11-23T04:42:42.839493122Z",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "nginx -t && echo \"Starting nginx.\" && nginx"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2629,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-11-23T04:42:43.974967996Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:2361625a064c8f5cfbeee410dd8d7978845ce25dc49596badd1f3943e53e9623",
        "ResolvConfPath": "/var/lib/docker/containers/4a254ab57792d53e15683cc7eb56148de9f1d8254dda8b480b769f761bcca243/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/4a254ab57792d53e15683cc7eb56148de9f1d8254dda8b480b769f761bcca243/hostname",
        "HostsPath": "/var/lib/docker/containers/4a254ab57792d53e15683cc7eb56148de9f1d8254dda8b480b769f761bcca243/hosts",
        "LogPath": "/var/lib/docker/containers/4a254ab57792d53e15683cc7eb56148de9f1d8254dda8b480b769f761bcca243/4a254ab57792d53e15683cc7eb56148de9f1d8254dda8b480b769f761bcca243-json.log",
        "Name": "/nginx_nginx_1",
        "RestartCount": 0,
        "Driver": "overlay",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/docker/nginx/pki:/etc/pki:ro",
                "/docker/nginx/nginx:/etc/nginx:ro",
                "/www/html:/www/html:ro",
                "/var/log/nginx:/var/log/nginx:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "labbr0",
            "PortBindings": {
                "443/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "443"
                    }
                ],
                "80/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "80"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "on-failure",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay/d775696efa8fe56fdfd920f198758998dc7b8fc9efb5637d31a298e09a81c229/root",
                "MergedDir": "/var/lib/docker/overlay/da45e6aa904556717fd80ccbae3aad703cdd8fd6dec173e47e511fc55d46db13/merged",
                "UpperDir": "/var/lib/docker/overlay/da45e6aa904556717fd80ccbae3aad703cdd8fd6dec173e47e511fc55d46db13/upper",
                "WorkDir": "/var/lib/docker/overlay/da45e6aa904556717fd80ccbae3aad703cdd8fd6dec173e47e511fc55d46db13/work"
            },
            "Name": "overlay"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/docker/nginx/nginx",
                "Destination": "/etc/nginx",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/www/html",
                "Destination": "/www/html",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/log/nginx",
                "Destination": "/var/log/nginx",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/docker/nginx/pki",
                "Destination": "/etc/pki",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "4a254ab57792",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "nginx -t && echo \"Starting nginx.\" && nginx"
            ],
            "ArgsEscaped": true,
            "Image": "ericflores/alpine-nginx",
            "Volumes": {
                "/etc/nginx": {},
                "/etc/pki": {},
                "/var/log/nginx": {},
                "/www/html": {}
            },
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "b531d1ffc806c1e09d858f3257f01d8fd93a17c44fc2829f00956f496b648ce7",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "nginx",
                "com.docker.compose.service": "nginx",
                "com.docker.compose.version": "1.17.1",
                "maintainer": "Eric Flores <ericflorescode@gmail.com>",
                "version": "3.6"
            },
            "StopSignal": "SIGTERM"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "a1b11c44c5b11c7e1e33d877abd8d9f477028774b3affa4a02e5ff172df62c40",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "443/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "443"
                    }
                ],
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "80"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/a1b11c44c5b1",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "labbr0": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "nginx",
                        "4a254ab57792"
                    ],
                    "NetworkID": "8ca9f763f789c7ebb2bfcbbbd76c15273faafb6a6be477e6f5fb007ea4bd2e7e",
                    "EndpointID": "358a49613ac29158c9d16a1939b6e331d2502e0950270a9a60b0a9b9fbcd9507",
                    "Gateway": "10.0.0.1",
                    "IPAddress": "10.0.0.128",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:00:80",
                    "DriverOpts": null
                }
            }
        }
    }
]

[root@host nginx]# docker network inspect labbr0
[
    {
        "Name": "labbr0",
        "Id": "8ca9f763f789c7ebb2bfcbbbd76c15273faafb6a6be477e6f5fb007ea4bd2e7e",
        "Created": "2017-11-22T20:41:16.734504575-08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "IPRange": "10.0.0.128/25",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "4a254ab57792d53e15683cc7eb56148de9f1d8254dda8b480b769f761bcca243": {
                "Name": "nginx_nginx_1",
                "EndpointID": "358a49613ac29158c9d16a1939b6e331d2502e0950270a9a60b0a9b9fbcd9507",
                "MacAddress": "02:42:0a:00:00:80",
                "IPv4Address": "10.0.0.128/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.name": "labbr0"
        },
        "Labels": {}
    }
]

[root@host nginx]# ip addr show
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master labbr0 state UP qlen 1000
    link/ether xxxx brd ff:ff:ff:ff:ff:ff
4: labbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether xxxx brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 scope global labbr0
       valid_lft forever preferred_lft forever
cpuguy83 commented 6 years ago

Some things to keep in mind:

  1. Port forwarding incurs an additional cost of NATing for external IP's.
  2. For traffic hitting the forwarded port originating from the bridge network, the traffic is routed through a usperspace proxy which will have significant performance impact.