search

v9fs / linux

Linux kernel source tree
Other
2 stars 2 forks source link

Directory listing caching #37

Open ericvh opened 5 months ago

ericvh commented 5 months ago

We currently do not cache directory listings, add a notion of caching directory listings to the baseline.

rickynils commented 3 weeks ago

@ericvh May I ask if any progress has been made on the directory caching in v9fs? My use case is sharing read-only, immutable stores over 9P, where directory caching would really improve performance without any real downsides.

ericvh commented 3 weeks ago

I had started on an implementation but got derailed by some of the rework I did to prep for doing better caching across the board and ran out of the time I had allocated to this. Better directory caching remains one of the biggest gaps in our cache strategy. That being said -- if you are in a read-only, immutable store you should be able to use loose mode which should get you most of the benefit since it shouldn't be reissuing getattrs for the directory contents even though it will still request the file list which doesn't currently get cached.

    -eric

On Thu, Oct 3, 2024 at 7:55 AM Rickard Nilsson @.***> wrote:

@ericvh https://github.com/ericvh May I ask if any progress has been made on the directory caching in v9fs? My use case is sharing read-only, immutable stores over 9P, where directory caching would really improve performance without any real downsides.

— Reply to this email directly, view it on GitHub https://github.com/v9fs/linux/issues/37#issuecomment-2391350709, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQ7WELKWGBZXR23CVE5FTZZU5CTAVCNFSM6AAAAABHAJILUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJRGM2TANZQHE . You are receiving this because you were mentioned.Message ID: @.***>

rickynils commented 2 weeks ago

That being said -- if you are in a read-only, immutable store you should be able to use loose mode which should get you most of the benefit since it shouldn't be reissuing getattrs for the directory contents even though it will still request the file list which doesn't currently get cached.

Yeah, stat calls etc are nicely cached, and with cache=fscache file contents are also cached. But if you have any kind of latency at all (even low latency like a 9P server running on localhost), the uncached directory listing kills many use cases, like grep --recursive. I will try to do some tracing to back up my claim that it is actually the directory listing that is the problem, but I'm pretty certain from simple timings.

ericvh commented 2 weeks ago

I can believe it still hurts for sure -- there may be a shortcut to try to loose cache directory contents (I was working towards proper cache). I'll see if I can get some time to look at it but I've been pretty busy elsewhere these days -- but it might be nice if we could come up with something for the next merge window.

On Thu, Oct 3, 2024 at 9:35 AM Rickard Nilsson @.***> wrote:

That being said -- if you are in a read-only, immutable store you should be able to use loose mode which should get you most of the benefit since it shouldn't be reissuing getattrs for the directory contents even though it will still request the file list which doesn't currently get cached.

Yeah, stat calls etc are nicely cached, and with cache=fscache file contents are also cached. But if you have any kind of latency at all (even low latency like a 9P server running on localhost), the uncached directory listing kills many use cases, like grep --recursive. I will try to do some tracing to back up my claim that it is actually the directory listing that is the problem, but I'm pretty certain from simple timings.

— Reply to this email directly, view it on GitHub https://github.com/v9fs/linux/issues/37#issuecomment-2391586413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQ7WFOTGQ4MODASC7TIRDZZVI3RAVCNFSM6AAAAABHAJILUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJRGU4DMNBRGM . You are receiving this because you were mentioned.Message ID: @.***>

rickynils commented 2 weeks ago

@ericvh Loose caching of directory contents sounds promising. Should you find time to work on that, I'd be happy to try out any patches.