search

google / gvisor

Application Kernel for Containers
https://gvisor.dev
Apache License 2.0
15.81k stars 1.3k forks source link

DNS not working with gVisor and netstack in Docker container #334

Closed jielingit closed 4 years ago

jielingit commented 5 years ago

DNS lookup fails in Docker container with gVisor and netstack. Network passthrough works. But we would prefer using netstack for network isolation.

docker run --rm -dit --runtime runsc --name ubuntu_runs jie/ubuntu bash
a73e6e85de4cb6337c214f076f454fd5a5f415dc3ef7bd25951709a7a639b370

docker attach a73e6e85de4cb6337c214f076f454fd5a5f415dc3ef7bd25951709a7a639b370
root@a73e6e85de4c:/# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.21.0.2
search us-west-2.compute.internal
root@a73e6e85de4c:/# 
root@a73e6e85de4c:/# apt update
Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease                  
  Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu bionic-security InRelease        
  Temporary failure resolving 'security.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done        
Building dependency tree       
Reading state information... Done
5 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@a73e6e85de4c:/# 
root@a73e6e85de4c:/# nc -vz google.com
google.com: forward host lookup failed: Host name lookup failure : Resource temporarily unavailable
docker version
Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:35:27 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       481bc77
  Built:            Sat May  4 01:59:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false
uname -a
Linux ip-10-21-30-51 4.4.0-1083-aws #93-Ubuntu SMP Wed May 8 16:08:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
jielingit commented 5 years ago

runsc_log.tar.gz

iangudger commented 5 years ago

This is the pattern I am seeing in the logs:

I0607 19:49:25.041483   12962 x:0] [  10] http E socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP)
I0607 19:49:25.041522   12962 x:0] [  10] http X socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 0x3 (19.12µs)
I0607 19:49:25.041553   12962 x:0] [  10] http E connect(0x3 socket:[3], 0x7f5b489efbd4 {Family: AF_INET, Addr: 10.21.0.2, Port: 53}, 0x10)
I0607 19:49:25.041666   12962 x:0] [  10] http X connect(0x3 socket:[3], 0x7f5b489efbd4 {Family: AF_INET, Addr: 10.21.0.2, Port: 53}, 0x10) = 0x0 (7.542µs)
I0607 19:49:25.041779   12962 x:0] [  10] http E poll(0x7fb7c0f15668 [{FD: 0x3 socket:[3], Events: POLLOUT, REvents: ...}], 0x1, 0x0)
I0607 19:49:25.041799   12962 x:0] [  10] http X poll(0x7fb7c0f15668 [{FD: 0x3 socket:[3], Events: POLLOUT|POLLERR|POLLHUP, REvents: POLLOUT}], 0x1, 0x0) = 0x1 (3.337µs)
I0607 19:49:25.041827   12962 x:0] [  10] http E sendto(0x3 socket:[3], 0x7fb7c0f157b0, 0x25, 0x4000, null, 0x0)
I0607 19:49:25.041846   12962 x:0] [  10] http X sendto(0x3 socket:[3], 0x7fb7c0f157b0, 0x25, 0x4000, null, 0x0) = 0x0 errno=112 (host is down) (3.821µs)
I0607 19:49:25.041872   12962 x:0] [  10] http E close(0x3 socket:[3])
I0607 19:49:25.041889   12962 x:0] [  10] http X close(0x3 socket:[3]) = 0x0 (5.452µs)

"host is down" is syserr.ErrHostDown, which the only netstack translation is tcpip.ErrNoLinkAddress. This generally means that ARP for this address is failing.

fvoznika commented 5 years ago

I'm not able to repro the problem. Can you verify that the same command on the same machine works fine with runc? If yes, then please also attach:

  1. docker inspect <container ID>
  2. tcpdump trace showing arp message would be helpfull too: sudo tcpdump -i <veth name> arp or udp

Thanks!

jielingit commented 5 years ago

Yes, same command on the same machine works fine with runc. 

docker run --rm -dit --name ubuntu_runc jie/ubuntu bash
c29719b3786599623b5147b57e3ee1bce5e3b709cdc5c997e9d215899fdd2026
docker attach c29719b3786599623b5147b57e3ee1bce5e3b709cdc5c997e9d215899fdd2026
root@c29719b37865:/# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.21.0.2
search us-west-2.compute.internal
root@c29719b37865:/# apt update
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]         
Get:4 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [4169 B]
Hit:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease                               
Get:6 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [535 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [837 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [717 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1220 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [7239 B]
Fetched 3498 kB in 2s (1736 kB/s)                         
Reading package lists... Done
Building dependency tree       
Reading state information... Done
14 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@c29719b37865:/# nc -vz google.com 80
DNS fwd/rev mismatch: google.com != sea15s12-in-f14.1e100.net
google.com [172.217.3.206] 80 (?) open
docker inspect c29719b37865
[
    {
        "Id": "c29719b3786599623b5147b57e3ee1bce5e3b709cdc5c997e9d215899fdd2026",
        "Created": "2019-06-13T17:14:48.358747872Z",
        "Path": "bash",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 27244,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-06-13T17:14:48.786218777Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:5bf19b59bc56353624acffe386b2551451d064e50fe36817faabe6f5de924cef",
        "ResolvConfPath": "/var/lib/docker/containers/c29719b3786599623b5147b57e3ee1bce5e3b709cdc5c997e9d215899fdd2026/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/c29719b3786599623b5147b57e3ee1bce5e3b709cdc5c997e9d215899fdd2026/hostname",
        "HostsPath": "/var/lib/docker/containers/c29719b3786599623b5147b57e3ee1bce5e3b709cdc5c997e9d215899fdd2026/hosts",
        "LogPath": "/var/lib/docker/containers/c29719b3786599623b5147b57e3ee1bce5e3b709cdc5c997e9d215899fdd2026/c29719b3786599623b5147b57e3ee1bce5e3b709cdc5c997e9d215899fdd2026-json.log",
        "Name": "/ubuntu_runc",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": true,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/35a5738bf22b8af4a22d70b6aff0149dc524bbd45b336f1c81d4df5085ee76ff-init/diff:/var/lib/docker/overlay2/a734a4a6acd892cd5d8f00d8c06411e84c35efc299849886c57b23423628ac15/diff:/var/lib/docker/overlay2/aae8689cb6d62deef4a716181abe2c2f7f15b00d300bd66bd12ab679cbd2b3f9/diff:/var/lib/docker/overlay2/d6403429a20aec8e526508fac557dfa5e984727db10f01e94bc2eae1712088b5/diff:/var/lib/docker/overlay2/d3cc58dfdb63b367150f14b87b403afa13afe85fd7f43b41df60c7575699f81e/diff",
                "MergedDir": "/var/lib/docker/overlay2/35a5738bf22b8af4a22d70b6aff0149dc524bbd45b336f1c81d4df5085ee76ff/merged",
                "UpperDir": "/var/lib/docker/overlay2/35a5738bf22b8af4a22d70b6aff0149dc524bbd45b336f1c81d4df5085ee76ff/diff",
                "WorkDir": "/var/lib/docker/overlay2/35a5738bf22b8af4a22d70b6aff0149dc524bbd45b336f1c81d4df5085ee76ff/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "c29719b37865",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "bash"
            ],
            "Image": "jie/ubuntu",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "5b7c32cfa5f274202b90b0bb94f3495b0568713a111a2a0c539a7a5726ebfe53",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/5b7c32cfa5f2",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "cc0108cd57b62e34f9702785aba65448875be439b5bc0a67249fe555ad5e2f0c",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "3a872ebb7f59a8185035bf25ef61e833551d5ed10eb2dcd7175da1b16783de20",
                    "EndpointID": "cc0108cd57b62e34f9702785aba65448875be439b5bc0a67249fe555ad5e2f0c",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]
tcpdump -v -i docker0 arp
tcpdump: listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:32:09.091372 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has ip-172-17-0-2.us-west-2.compute.internal tell ip-172-17-0-1.us-west-2.compute.internal, length 28
17:32:09.091406 ARP, Ethernet (len 6), IPv4 (len 4), Reply ip-172-17-0-2.us-west-2.compute.internal is-at 02:42:ac:11:00:02 (oui Unknown), length 28
fvoznika commented 5 years ago

ARP response looks fine. Can you also collect UDP packets?

sudo tcpdump -i <veth name> arp or udp

And also use the veth name of the container's namespace. The name looks something like this: veth13cc193@if978. To find it, you run ip link before starting the container, and then run it again after the container has started. The new device to show up in the list is the [external side of the] container's veth.

jielingit commented 5 years ago

Failed to run tcpdump. Got ioctl failed: No such device. See below.

root:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 06:77:d0:ce:1b:58 brd ff:ff:ff:ff:ff:ff
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:6c:f5:86:e8 brd ff:ff:ff:ff:ff:ff
550: br-7d512d2a0116: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:90:4c:50:8f brd ff:ff:ff:ff:ff:ff
root:~# 
root:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 06:77:d0:ce:1b:58 brd ff:ff:ff:ff:ff:ff
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:6c:f5:86:e8 brd ff:ff:ff:ff:ff:ff
550: br-7d512d2a0116: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:90:4c:50:8f brd ff:ff:ff:ff:ff:ff
570: veth721974a@if569: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 06:b3:5b:6b:f9:b0 brd ff:ff:ff:ff:ff:ff link-netnsid 0

root:~# sudo tcpdump -i veth721974a@if569 udp
tcpdump: veth721974a@if569: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device

root:~# sudo tcpdump -i veth721974a@if569 arp
tcpdump: veth721974a@if569: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device
ghost commented 5 years ago

the same issue, gvisor seems not working right ( the network not accessable)

fvoznika commented 5 years ago

@jielingit for the interface name, you need to drop the part after @ (yeah, it's confusing):

sudo tcpdump -i veth721974a arg or udp

@netroby can you provide more details. What docker run parameters did you use? What is the error? Please capture logs with debug --strace --logpackets, instructions here.

fvoznika commented 4 years ago

Closing due to no activity. Please re-open if still an issue.