search

diskfs / go-diskfs

MIT License
494 stars 112 forks source link

squashfs: add configurable block cache #206

Closed ncw closed 8 months ago

ncw commented 8 months ago

This adds a configurable block cache as discussed in #205

This is done in 3 commits for ease of review - it compiles and the tests pass at each commit so they can be merged as 3 commits if desired.

With this cache this now performs within a factor of 2 of unsquashfs

ncw commented 8 months ago

I wouldn't mind some performance test on this, but not for me to complain.

Were you thinking of a Go benchmark?

Here are some with rclone - unpacking my favourite tensorflow 3GB squashfs with rclone

no caching 256 MB cache 128MB cache 64 MB cache 32 MB cache
206s 16.7s 17.5s 23.4s 54.s

Whereas unsquashfs takes 12.0s

Do you want to add a comment somewhere (likely in squashfs.go, or in doc.go) about why we have a cache for this, capture some of your thoughts from the PR and issue?

I've added some notes to the Filesystem.Read method which is where the cache gets initialized - this will appear in the Go doc and be hopefully obvious to the users.

Interestingly the squashfs kernel module uses the same design choice (which I hadn't noticed before) about having a fragment and metadata cache.

Note that this PR will conflict with #201 I think, which I'll fix up.

ncw commented 8 months ago

Note that this PR will conflict with https://github.com/diskfs/go-diskfs/pull/201 I think, which I'll fix up.

Oooh, you merged that while I was writing the above! It is now fixed.

deitch commented 8 months ago

I think this is solid.

Do you want to throw a benchmarking doc in there? You captured much of it here, but this tends to get lost. Specifically either in the doc.go, or in the lru.go? Just showing how you created the file, how you used rclone, so we have some basis for understanding it in the future?

ncw commented 8 months ago

I added some info about benchmarks here

https://github.com/ncw/go-diskfs/blob/a7c64791e8ccc64f3bed0e0618e712224a1338b5/filesystem/squashfs/squashfs.go#L126-L141

What do you think - look ok?

ncw commented 8 months ago

Actually I forgot to say how I made the file, let me add that!

ncw commented 8 months ago

Done!

https://github.com/ncw/go-diskfs/blob/84de30b32142fa0fec9e29ab378e0a83076a3fc3/filesystem/squashfs/squashfs.go#L126-L147