kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.42k stars 4.88k forks source link

enable minikube mount for kic drivers (docker, podman) #6628

Closed medyagh closed 4 years ago

medyagh commented 4 years ago

on linux this works

docker@p2:~$ /sbin/ip route|awk '/default/ { print $3 }'

also

docker inspect --format '{{(index .IPAM.Config 0).Gateway}}' $(docker network ls --filter name=bridge --format {{.ID}})

medyagh commented 4 years ago

dig +short host.docker.internal

medyagh commented 4 years ago

update for mount on Mac I get the IP using ( 'dig +short host.docker.internal')

it keeps telling me "special device 192.168.65.2 does not exist."

to debug and confirm the problem is not connectivity I started a UFS server manually with this code

    p, err := getPort()
    if err != nil {
        fmt.Printf("Failed to get port: %v", err)
    }
    fmt.Println(p)
    ufs.StartServer(net.JoinHostPort("0.0.0.0", strconv.Itoa(p)), 8, "/Users/medmac/minikube/tmp13")

then inside container I tried the command that minikube is running which is

root@minikube:/# mount -t 9p -o dfltgid=$(grep ^docker: /etc/group | cut -d: -f3),dfltuid=$(id -u docker),msize=262144,port=56172,version=9p2000.L 192.168.65.2 /m13
mount: /m13: special device 192.168.65.2 does not exist.

(I get the IP using 'dig +short host.docker.internal') I ran the same command and I change the port to the one I have :

and exact same error, however I am able to curl the ip from inside the container and on the server I see the response :

root@minikube:/# curl 192.168.65.2:56172
curl: (52) Empty reply from server
root@minikube:/# curl 192.168.65.2:56172
curl: (52) Empty reply from server
root@minikube:/# curl 192.168.65.2:56172
curl: (52) Empty reply from server
root@minikube:/# curl 192.168.65.2:56172
curl: (52) Empty reply from server
root@minikube:/# curl 192.168.65.2:56172
curl: (52) Empty reply from server
root@minikube:/# curl 192.168.65.2:56172
curl: (52) Empty reply from server
root@minikube:/# curl 192.168.65.2:56172
curl: (52) Empty reply from server

same as dns

root@minikube:/# curl host.docker.internal:56172
curl: (52) Empty reply from server
root@minikube:/# curl host.docker.internal:56172
curl: (52) Empty reply from server
root@minikube:/# curl host.docker.internal:56172
curl: (52) Empty reply from server
root@minikube:/# curl host.docker.internal:56172
curl: (52) Empty reply from server
root@minikube:/#
root@minikube:/# curl host.docker.internal:56172
curl: (52) Empty reply from server

however the mount command insider the cntainer special device doesn't exists and on the server side I see no evidance that it receives the request at all

root@minikube:/# mount -t 9p -o dfltgid=$(grep ^docker: /etc/group | cut -d: -f3),dfltuid=$(id -u docker),msize=262144,port=56172,version=9p2000.L 192.168.65.2 /m13
mount: /m13: special device 192.168.65.2 does not exist.
root@minikube:/#
root@minikube:/# mount -t 9p -o dfltgid=$(grep ^docker: /etc/group | cut -d: -f3),dfltuid=$(id -u docker),msize=262144,port=56172,version=9p2000.L host.docker.internal /m13
mount: /m13: special device host.docker.internal does not exist.

any idea @afbjorklund @tstromberg @josedonizetti

medyagh commented 4 years ago

one difference I could think of with VM is, the mount version is different by one patch version on the container:

mount --version
mount from util-linux 2.34 (libmount 2.34.0: selinux, smack, btrfs, namespaces, assert, debug)
afbjorklund commented 4 years ago

Seem to be missing a colon : after the IP, in the mount command ?

medyagh commented 4 years ago

@afbjorklund

Seem to be missing a colon : after the IP, in the mount command ?

the command is same command is used in VM. this PR only changed what IP would return.... where would the : be ?

tstromberg commented 4 years ago

@medyagh - To me, the error message indicates a parse error. Try running the embedded shell script commands to see which is failing:

tstromberg commented 4 years ago

My suggestion would be to use $SUDO_UID and $SUDO_GID instead of those shell lookups.

tstromberg commented 4 years ago

Looks like a red herring. Those subcommands work just fine, but for whatever reason, the mount command is still failing. Is it a privilege thing, or a quirk of the Docker VM? I don't know.

That said, I can confirm that the Docker container can exchange packets to the 9p filesystem just fine:

# echo test | nc -vv 192.168.65.2 59542
192.168.65.2: inverse host lookup failed: Unknown host
(UNKNOWN) [192.168.65.2] 59542 (?) open
 sent 5, rcvd 0

The error message seems to be unrelated to connectivity. straces shows:

access("/run/mount", R_OK|W_OK)         = 0
mount("192.168.65.2", "/mount-9p", "9p", 0, "dfltgid=110,dfltuid=1000,msize=2"...) = -1 ENOENT (No such file or directory)
lstat("/mount-9p", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mount-9p", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("192.168.65.2", 0x7fffa0f4e9a0)    = -1 ENOENT (No such file or directory)
write(2, "mount: ", 7mount: )                  = 7
write(2, "/mount-9p: special device 192.16"..., 54/mount-9p: special device 192.168.65.2 does not exist.) = 54
write(2, "\n", 1
)                     

It's still unclear to me why it's looking for the argument to contain local device rather than passing it to the 9p driver as an IP.

tstromberg commented 4 years ago

Seen in dmesg:

[ 631.443997] 9pnet_virtio: no channels available for device 192.168.65.2

Woops, mount -9p on the Docker VM defaults to VirtIO instead of TCP for it's transmission method. Easy fix.