search

openscopeproject / ZipROFS

FUSE file system with transparent access to zip files as if they were folders.
MIT License
12 stars 5 forks source link

Project tries to open EVERY file in directory when listing #2

Closed JuniorJPDJ closed 4 years ago

JuniorJPDJ commented 4 years ago

I'm trying to use this project on remote filesystem, which is pretty slow. My worst case directory has over 4k files, where like 2k are ZIPs. Problem is that ZipROFS tries opening EVERY file from that directory on EVERY listing of directory. Probably problem is in this line: https://github.com/openscopeproject/ZipROFS/blob/dev/ziprofs.py#L75 This call is not cached and it's reading file on every listing with this line https://github.com/openscopeproject/ZipROFS/blob/dev/ziprofs.py#L134 This makes my first dir listing call take +-5 minutes, which is not acceptable ;/

JuniorJPDJ commented 4 years ago

Screenshot_20200829_030706 I may be wrong with exact lines, but there's an example of this running. Only thing I did is mount ZipROFS (up-right corner), feedback is visible in debug view of remote fs (up-left corner) and bmon showing network usage. When it will finish I'll show example on ls of directory.

JuniorJPDJ commented 4 years ago

Even better - it's even on list of directory below this one ;o EDIT: Nevermind, just ZSH called list on this directory anyway.

JuniorJPDJ commented 4 years ago

There's video showing my issue: https://pixeldrain.com/u/PxBckYGJ

qu1ck commented 4 years ago

This reads happen because shell calls getattr on every node in the listing, not because of the listing itself.

I added a shortcut to not check files that don't have .zip extension. I'm reluctant to completely remove the iz_zipfile() but I could probably make it an option.

JuniorJPDJ commented 4 years ago

The only solution which could make it better that comes to my mind is also caching is_zipfile() return values.

JuniorJPDJ commented 4 years ago

Fixed in #4