search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.41k stars 4.88k forks source link

9p mount fails with >650 files in a directory: 'too small read size for dir entry' ecode 22 #1753

Open hanks opened 7 years ago

hanks commented 7 years ago

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Minikube version (use minikube version): v0.21.0 Environment:

What happened: I use the new flag minikube mount --msize 104857600, and I tried some values from the default value 262144(256KB) to 104857600(100MB), still can not mount the alembic versions directory, that contains about total 644 (same filename with different ext, .py and .pyc) files, and filename length is average 30 character.

But when I remove all the .pyc files to reduce the total file number to 322, it can be mounted.

Debug log is below:

sudo mount -t 9p -o trans=tcp,port=55116,dfltuid=1001,dfltgid=1001,version=9p2000.u,msize=104857600 192.168.99.1 /xxxx/alembic/versions;

2017/07/26 12:44:06 <<< 192.168.99.106:48318 Rstat tag 1 st ('versions' 'none' 'none' 'none' q (7cf86b 7ce03230 'd') m d755 at 0 mt 1501038654 l 21964 t 0 d 0 ext )
2017/07/26 12:44:06 >>> 192.168.99.106:48318 Tread tag 1 fid 5 offset 0 count 65512
2017/07/26 12:44:06 <<< 192.168.99.106:48318 Rread tag 1 count 65428
2017/07/26 12:44:06 >>> 192.168.99.106:48318 Tread tag 1 fid 5 offset 65428 count 84
2017/07/26 12:44:06 <<< 192.168.99.106:48318 Rerror tag 1 ename 'too small read size for dir entry' ecode 22
2017/07/26 12:44:06 >>> 192.168.99.106:48318 Tclunk tag 1 fid 5

What you expected to happen: Directory should be mounted

How to reproduce it (as minimally and precisely as possible):

Anything else do we need to know:

davefinster commented 7 years ago

I'm encountering something similar with a node_modules folder containing 912 individual folders. For me, I just get an empty directory. When running minikube mount with -v 10 I get the following entries:

2017/09/26 14:05:24 >>> 192.168.99.100:59698 Tstat tag 1 fid 3
2017/09/26 14:05:24 <<< 192.168.99.100:59698 Rstat tag 1 st ('node_modules' 'none' 'none' 'none' q (2006205e9 bb3fb888 'd') m d755 at 0 mt 1506380069 l 29280 t 0 d 0 ext )
2017/09/26 14:05:24 >>> 192.168.99.100:59698 Twalk tag 1 fid 3 newfid 4
2017/09/26 14:05:24 <<< 192.168.99.100:59698 Rwalk tag 1
2017/09/26 14:05:24 >>> 192.168.99.100:59698 Topen tag 1 fid 4 mode 0
2017/09/26 14:05:24 <<< 192.168.99.100:59698 Ropen tag 1 qid (2006205e9 bb3fb888 'd') iounit 0
2017/09/26 14:05:24 >>> 192.168.99.100:59698 Tread tag 1 fid 4 offset 0 count 65512
2017/09/26 14:05:24 <<< 192.168.99.100:59698 Rread tag 1 count 65508
2017/09/26 14:05:24 >>> 192.168.99.100:59698 Tread tag 1 fid 4 offset 65508 count 4
2017/09/26 14:05:24 <<< 192.168.99.100:59698 Rerror tag 1 ename 'too small read size for dir entry' ecode 22
2017/09/26 14:05:24 >>> 192.168.99.100:59698 Tclunk tag 1 fid 4
2017/09/26 14:05:24 <<< 192.168.99.100:59698 Rclunk tag 1

Oddly it appears that only directory listing is affected. If I attempt to go directly to a particular path, it succeeds. This is with the following:

minikube version: v0.22.2
OS: macOS High Sierra Gold Master - 10.13 (17A362a)
Driver: Virtualbox
ISO: minikube-v0.23.4.iso

I have gone as high as using this:

minikube mount --msize=1004857600 -v 10 $(pwd):/mnt/Projects
amnk commented 6 years ago

I'm facing the same behavior with node_modules folder (797 directories in it). Tries setting msize to the biggest possible value, but still get this.

minikube mount --msize=1048576000 --v 10 node_modules/:/node_modules
fejta-bot commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 6 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten /remove-lifecycle stale

fejta-bot commented 6 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

Diggsey commented 6 years ago

/reopen /remove-lifecycle rotten

AFAICT, this makes minikube completely unusable with npm??

k8s-ci-robot commented 6 years ago

@Diggsey: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to [this](https://github.com/kubernetes/minikube/issues/1753#issuecomment-403842066): >/reopen >/remove-lifecycle rotten > >AFAICT, this makes minikube completely unusable with npm?? Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
fejta-bot commented 5 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 5 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

tstromberg commented 5 years ago

Confirmed with v1.1.0. Reproduction instructions:

Host:

mkdir /tmp/test
cd /tmp/test
for i in $(seq 1 1500); do dest=$(printf %30.30d $i); touch $dest; done
minikube mount /tmp/test:/tmp/test --msize 262144000 --alsologtostderr  -v=8

In VM:

$ ls /tmp/test
ls: reading directory '/tmp/7779': Unknown error 526

Logs:

2019/05/22 13:06:34 >>> 192.168.39.191:43630 Tread tag 1 fid 1 offset 65500
count 12
2019/05/22 13:06:34 <<< 192.168.39.191:43630 Rerror tag 1 ename 'too small read size for dir entry' ecode 0
2019/05/22 13:06:34 >>> 192.168.39.191:43630 Tclunk tag 1 fid 1            
2019/05/22 13:06:34 <<< 192.168.39.191:43630 Rclunk tag 1

Every file adds 70 + (length of filename) to the offset, so at 30 characters per filename, this breaks at around 660 files in a single directory. I suspect the Go 9p server we use was not designed to handle so many files in a directory.

As an alternative, one can setup NFS or use file sync. I suspect we'll need a better 9p server.

tstromberg commented 4 years ago

This issue will likely remain open until we drop support for 9p.

Alternatively, if someone wants to attack the go9p source code to solve the issue at the root, help wanted!

ChumaA commented 4 years ago

I have the same issue

MPLew-is commented 3 years ago

@tstromberg has anyone tried to integrate this still-active go9p implementation? It looks like the one currently in the repository has not been maintained for a long time.

If nobody has tried it yet I can give it a stab, though I'm pretty much brand new to Go so it'll take me a while to muddle through it.

tstromberg commented 3 years ago

@MPLew-is - no, but I would be exceedingly happy to see that transition happen! Let me (or the #minikube Slack channel) know if you need any help.