Closed ncw closed 8 months ago
I wouldn't mind some performance test on this, but not for me to complain.
Were you thinking of a Go benchmark?
Here are some with rclone - unpacking my favourite tensorflow 3GB squashfs with rclone
no caching | 256 MB cache | 128MB cache | 64 MB cache | 32 MB cache |
---|---|---|---|---|
206s | 16.7s | 17.5s | 23.4s | 54.s |
Whereas unsquashfs
takes 12.0s
Do you want to add a comment somewhere (likely in squashfs.go, or in doc.go) about why we have a cache for this, capture some of your thoughts from the PR and issue?
I've added some notes to the Filesystem.Read method which is where the cache gets initialized - this will appear in the Go doc and be hopefully obvious to the users.
Interestingly the squashfs kernel module uses the same design choice (which I hadn't noticed before) about having a fragment and metadata cache.
Note that this PR will conflict with #201 I think, which I'll fix up.
Note that this PR will conflict with https://github.com/diskfs/go-diskfs/pull/201 I think, which I'll fix up.
Oooh, you merged that while I was writing the above! It is now fixed.
I think this is solid.
Do you want to throw a benchmarking doc in there? You captured much of it here, but this tends to get lost. Specifically either in the doc.go
, or in the lru.go
? Just showing how you created the file, how you used rclone, so we have some basis for understanding it in the future?
I added some info about benchmarks here
What do you think - look ok?
Actually I forgot to say how I made the file, let me add that!
This adds a configurable block cache as discussed in #205
This is done in 3 commits for ease of review - it compiles and the tests pass at each commit so they can be merged as 3 commits if desired.
With this cache this now performs within a factor of 2 of unsquashfs