search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.41k stars 4.88k forks source link

Files missing within minikube mounted volume #1515

Closed georgecrawford closed 6 years ago

georgecrawford commented 7 years ago

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Minikube version v0.19.0

Environment:

What happened: On creating a fresh VM (minikube start --vm-driver=xhyve), I can see my /Users directory mounted at the same location on the VM (minikube ssh). However, I've encountered a directory where the file count is wrong:

ls -l /Users/george/[MYDIR]/ | wc && minikube ssh -- ls -l /Users/george/[MYDIR]/ | wc
      123     123    5620
      121     121    5547

I don't know how to explore this further. If there are fewer than 70 files in the directory, the numbers are the same. However, at some point over 70, the numbers diverge, as if the mount inside the minikube VM has simply not seen some of the files. They all have identical permissions.

What you expected to happen: The directory count in minikube VM and my host Mac should always be the same

How to reproduce it (as minimally and precisely as possible): I'm not sure. I've tried to reproduce it by creating empty files in a directory, but I can't recreate the bug. I've only seen it in one particular directory, but it doesn't seem to matter which of the files are in the directory - as soon as there are more than ~70, I see incorrect counts.

Anything else do we need to know: I've cross-posted this here, as I wasn't sure if it was a problem with xhyve or minikube. Very happy to close one of them as a appropriate.

If it's important, the directory is mounted into a pod using a hostPath volume.

georgecrawford commented 7 years ago

NB: cross-posted here: https://github.com/zchee/docker-machine-driver-xhyve/issues/176

georgecrawford commented 7 years ago

I'm still suffering from this same problem with minikube ISO v0.20.0 and docker-machine-driver-xhyve v0.3.3. Is there any other information I can provide to get to the bottom of the issue?

r2d4 commented 7 years ago

The default /Users mount is coming from xhyve. Can you try to see if the same problem exists with minikube mount? The 9p filesystem implementations we use are different, so its possible this bug doesn't exist in the implementation we use.

georgecrawford commented 7 years ago

mount doesn't seem to work at all for me

# OS X Host:
$ ls -l test | grep -v total | wc -l
126

# Minikube VM:
$ ls -l /Users/[path to...]/test/ | grep -v total | wc -l
124

# OS X Host
$ minikube mount $(pwd)/test:/test
Mounting /Users/[path to...]/test into /test on the minikube VM
This daemon process needs to stay alive for the mount to still be accessible...
ufs starting

# Minikube VM
$ ls -l /test | grep -v total | wc -l
ls: reading directory '/test': Unknown error 526
0
georgecrawford commented 7 years ago

Could be related to https://github.com/kubernetes/minikube/issues/1563 and https://github.com/kubernetes/minikube/issues/1562.

$ minikube mount $(pwd)/test:/test --v 10
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:52322
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .DriverName
Mounting /Users/[SNIP]/test into /test on the minikube VM
This daemon process needs to stay alive for the mount to still be accessible...
ufs starting
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:52327
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/george/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.35 -o IdentitiesOnly=yes -i /Users/george/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo umount /test;
SSH cmd err, output: exit status 32: umount: /test: mountpoint not found

(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/george/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.35 -o IdentitiesOnly=yes -i /Users/george/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:

sudo mkdir -p /test || true;
sudo mount -t 9p -o trans=tcp -o port=52324 -o uid=1001 -o gid=1001 192.168.64.1 /test;
sudo chmod 775 /test;
2017/07/10 10:43:23 connected
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Tversion tag 65535 msize 8192 version '9P2000.L'
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rversion tag 65535 msize 8192 version '9P2000'
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Tattach tag 1 fid 0 afid 4294967295 uname 'nobody' nuname 0 aname ''
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rattach tag 1 aqid (1af9adb 1e4819e0 'd')
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Tstat tag 1 fid 0
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rstat tag 1 st ('test' 'george' '20' '' q (1af9adb 1e4819e0 'd') m d775 at 0 mt 1499451628 l 4352 t 0 d 0 ext )
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Tstat tag 1 fid 0
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rstat tag 1 st ('test' 'george' '20' '' q (1af9adb 1e4819e0 'd') m d775 at 0 mt 1499451628 l 4352 t 0 d 0 ext )
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Twstat tag 1 fid 0 st ('' '' '' '' q (ffffffffffffffff ffffffff 'daAltL') m d775 at 4294967295 mt 4294967295 l 18446744073709551615 t 65535 d 4294967295 ext )
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rwstat tag 1
SSH cmd err, output: <nil>:
antonmarin commented 7 years ago

check free RAM on VPS. Helped for me

flowstate commented 6 years ago

So this issue, and the two linked above are all open. Is there a workaround? I understand if it's a tough solve, but without a workaround, this completely breaks our ability to set up local test environments. Any help would be appreciated.

fejta-bot commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 6 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten /remove-lifecycle stale

fejta-bot commented 6 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close