Can't lookup index.docker.io with gVisor in Kubernetes

[saphoooo]

Actual behavior I'm trying to build an image in Kubernetes with runsc, but it fails with error message:

error building image: getting stage builder for stage 0: Get https://index.docker.io/v2/: dial tcp: lookup index.docker.io on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: no route to host 

Expected behavior Without sandboxing in gVisor, the same build works well and push the image to the registry.

To Reproduce Steps to reproduce the behavior:

1. Configure kubeadm to work with cri-o
2. Add gVisor for insecure Pods
3. Run the yaml provided

Additional Information

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
  annotations:
    io.kubernetes.cri-o.TrustedSandbox: "false"
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args: ["--dockerfile=/data/Dockerfile",
            "--context=/data",
            "--destination=docker.io/jblaskowich/hellogvisor:latest",
            "--force",]
    volumeMounts:
      - name: datadir
        mountPath: /data
      - name: docker-cfg
        mountPath: /root
  restartPolicy: Never
  initContainers:
  - name: pull-container
    image: docker.io/alpine/git:latest
    command: ['git', 'clone', 'https://github.com/jblaskowich/hellogvisor.git', '/data']
    volumeMounts:
      - name: datadir
        mountPath: /data
  volumes:
    - name: datadir
      emptyDir: {}
    - name: docker-cfg
      projected:
        sources:
        - secret:
            name: regcred
            items:
              - key: .dockerconfigjson
                path: .docker/config.json

Error Message

I1128 22:00:50.668673   31172 x:0] ***************************
I1128 22:00:50.668728   31172 x:0] Args: [/usr/local/bin/runsc create --bundle /var/run/containers/storage/overlay-containers/dc6b3fd7b9a092e0176c35354a8eaf4edf931dfd97db2c15d1914c0df1c49ed7/userdata --pid-file /var/run/containers/storage/overlay-containers/dc6b3fd7b9a092e0176c35354a8eaf4edf931dfd97db2c15d1914c0df1c49ed7/userdata/pidfile dc6b3fd7b9a092e0176c35354a8eaf4edf931dfd97db2c15d1914c0df1c49ed7]
I1128 22:00:50.668742   31172 x:0] Git Revision: 1918563525662d6645ec921e61aa7e6da92af0dd
I1128 22:00:50.668748   31172 x:0] PID: 31172
I1128 22:00:50.668757   31172 x:0] UID: 0, GID: 0
I1128 22:00:50.668761   31172 x:0] Configuration:
I1128 22:00:50.668766   31172 x:0]      RootDir: /var/run/runsc
I1128 22:00:50.668770   31172 x:0]      Platform: ptrace
I1128 22:00:50.668777   31172 x:0]      FileAccess: exclusive, overlay: false
I1128 22:00:50.668793   31172 x:0]      Network: sandbox, logging: false
I1128 22:00:50.668801   31172 x:0]      Strace: false, max size: 1024, syscalls: []
I1128 22:00:50.668817   31172 x:0] ***************************
I1128 22:00:50.669894   31172 x:0] Mounting src: "/var/run/containers/storage/overlay-containers/b73f1e665b32b0e280fb95ef415b22696caf766b794a1d13deb8c88e1f1e5768/userdata/resolv.conf", dst: "/var/lib/containers/storage/overlay/67ac876a2296f8496a51165e85367de25db9285dfd2467c3bc4e7751abe5b272/merged/etc/resolv.conf", flags: 0x100e
I1128 22:00:50.678997   31172 x:0] Mounting src: "/var/run/containers/storage/overlay-containers/b73f1e665b32b0e280fb95ef415b22696caf766b794a1d13deb8c88e1f1e5768/userdata/hostname", dst: "/var/lib/containers/storage/overlay/67ac876a2296f8496a51165e85367de25db9285dfd2467c3bc4e7751abe5b272/merged/etc/hostname", flags: 0x1001
I1128 22:00:50.683394   31172 x:0] Mounting src: "/var/lib/kubelet/pods/a044ea25-f350-11e8-b146-c26268402bf0/volumes/kubernetes.io~empty-dir/datadir", dst: "/var/lib/containers/storage/overlay/67ac876a2296f8496a51165e85367de25db9285dfd2467c3bc4e7751abe5b272/merged/data", flags: 0x45001
I1128 22:00:50.687979   31172 x:0] Mounting src: "/var/lib/kubelet/pods/a044ea25-f350-11e8-b146-c26268402bf0/volumes/kubernetes.io~projected/docker-cfg", dst: "/var/lib/containers/storage/overlay/67ac876a2296f8496a51165e85367de25db9285dfd2467c3bc4e7751abe5b272/merged/root", flags: 0x45001
I1128 22:00:50.693970   31172 x:0] Mounting src: "/var/lib/kubelet/pods/a044ea25-f350-11e8-b146-c26268402bf0/etc-hosts", dst: "/var/lib/containers/storage/overlay/67ac876a2296f8496a51165e85367de25db9285dfd2467c3bc4e7751abe5b272/merged/etc/hosts", flags: 0x45001
I1128 22:00:50.700498   31172 x:0] Mounting src: "/var/lib/kubelet/pods/a044ea25-f350-11e8-b146-c26268402bf0/containers/kaniko/c97145d5", dst: "/var/lib/containers/storage/overlay/67ac876a2296f8496a51165e85367de25db9285dfd2467c3bc4e7751abe5b272/merged/dev/termination-log", flags: 0x45001
I1128 22:00:50.706061   31172 x:0] Mounting src: "/var/lib/kubelet/pods/a044ea25-f350-11e8-b146-c26268402bf0/volumes/kubernetes.io~secret/default-token-zgcvt", dst: "/var/lib/containers/storage/overlay/67ac876a2296f8496a51165e85367de25db9285dfd2467c3bc4e7751abe5b272/merged/var/run/secrets/kubernetes.io/serviceaccount", flags: 0x45001
I1128 22:00:50.724225   31172 x:0] Gofer started, PID: 31209
I1128 22:00:50.732512   31172 x:0] Creating sandbox process with addr: runsc-sandbox.dc6b3fd7b9a092e0176c35354a8eaf4edf931dfd97db2c15d1914c0df1c49ed7
I1128 22:00:50.732597   31172 x:0] Sandbox will be started in new mount, IPC and UTS namespaces
I1128 22:00:50.732605   31172 x:0] Sandbox will be started in the current PID namespace
I1128 22:00:50.732614   31172 x:0] Sandbox will be started in the container's network namespace: {Type:network Path:/proc/30610/ns/net}
I1128 22:00:50.732648   31172 x:0] Sandbox will be started in new user namespace
I1128 22:00:50.733484   31172 x:0] Sandbox will be started in minimal chroot
I1128 22:00:50.733622   31172 x:0] Setting up sandbox chroot in "/tmp/runsc-sandbox-chroot-218416254"
I1128 22:00:50.733679   31172 x:0] Mounting "proc" at "/tmp/runsc-sandbox-chroot-218416254/proc"
I1128 22:00:50.739743   31172 x:0] Mounting "/usr/local/bin/runsc" at "/tmp/runsc-sandbox-chroot-218416254/runsc"
I1128 22:00:50.754493   31172 x:0] applying namespace network at path "/proc/30610/ns/net"
I1128 22:00:50.756163   31172 x:0] restoring namespace network
I1128 22:00:50.756233   31172 x:0] Sandbox started, PID: 31217
I1128 22:00:50.809098   31172 x:0] Exiting with status: 0
WARN[0000] kaniko is being run outside of a container. This can have dangerous effects on your system
INFO[0000] Downloading base image golang:alpine3.8
error building image: getting stage builder for stage 0: Get https://index.docker.io/v2/: dial tcp: lookup index.docker.io on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: no route to host

[saphoooo]

Nobody experienced the same issue? Any idea to help me investigate further?

[priyawadhwa]

Hi @saphoooo I tried to build the image using gvisor in minikube and I was able to build successfully. I'm not very familiar with gvisor in general though; you could try opening an issue in the gvisor repo to see if they are familiar with this bug?

[saphoooo]

Ok, I've opened an issue #122. Did you use cri-o for building your image in minikube or cri-o as I did?

[priyawadhwa]

minikube requires containerd as the container runtime to enable gvisor

[tequilarista]

Unfortunately cri-o is not a supported container runtime for this project, closing out as won't fix for now. We can certainly have a larger discussion around cri-o in the future if there's sufficient community demand.

[tequilarista]

okay I apparently lied -- crio has been added, will recheck the steps to reproduce...

[tequilarista]

Priya, if you could do a quick eyeball with the latest minikube that would be glorious

[tequilarista]

okay, gviso + crio still an issue -- moving this to blocked