search

project-machine / puzzlefs

Apache License 2.0
381 stars 18 forks source link

do some experiments around finding inodes #19

Closed tych0 closed 1 year ago

tych0 commented 3 years ago

right now we do an on-disk binary search for finding each inode. the fs cache might help us here, but it may be worth either 1. maybe mmaping the inode metadata region? or 2. reading some/all of the inodes into memory in order to speed this up.

This probably depends on the number of inodes in a particular layer.

ariel-miculas commented 1 year ago

Comparison with squashfuse:

Prerequisites

Installation (Arch Linux):

❯ sudo pacman -S squashfuse
❯ sudo pacman -S squashfs-tools

Setup

Squashfs

❯ mksquashfs real_rootfs/barehost/rootfs barehost.sqhs
Parallel mksquashfs: Using 8 processors
Creating 4.0 filesystem on barehost.sqhs, block size 131072.
[=========================================================================|] 20365/20365 100%
❯ squashfuse -h
squashfuse 0.1.104 (c) 2012 Dave Vasilevsky

Puzzlefs (commit e97f53c3d55973674a0495adaa9db3e284f9415f)

❯ ../puzzlefs/target/debug/puzzlefs build real_rootfs/barehost/rootfs /tmp/oci-barehost test

Single-threaded squashfuse vs puzzlefs

This is the fairest comparison since puzzlefs doesn't support parallel operation

First measurement after mounting

❯ squashfuse -f -s barehost.sqhs /tmp/squash
tree /tmp/squash  0.06s user 0.15s system 7% cpu 2.856 total
❯ ../puzzlefs/target/debug/puzzlefs mount -f /tmp/oci-barehost test /tmp/puzzle
tree /tmp/puzzle  0.13s user 0.40s system 0% cpu 1:24.17 total
❯ target/release/puzzlefs mount -f /tmp/oci-barehost test /tmp/puzzle
tree /tmp/puzzle  0.08s user 0.36s system 0% cpu 52.685 total

Second measurement after mounting

tree /tmp/squash  0.06s user 0.15s system 7% cpu 2.870 total
# debug
tree /tmp/puzzle  0.06s user 0.09s system 5% cpu 3.105 total
# release
tree /tmp/puzzle  0.06s user 0.08s system 7% cpu 1.857 total

Conclusion

Traversing a puzzlefs mounted filesystem the first time after it is mounted is significantly slower than traversing a squashfuse filesystem, but subsequent runs are close to the performance of squashfuse. Also note that the output of tree is not the same for squashfuse and puzzlefs, that's because puzzlefs has some limitations (e.g. it cannot read symbolic links).

Multi-threaded squashfuse

First measurement after mounting

❯ squashfuse -f barehost.sqhs /tmp/squash
tree /tmp/squash  0.08s user 0.15s system 8% cpu 2.902 total

Second measurement after mounting

tree /tmp/squash  0.06s user 0.16s system 7% cpu 2.868 total

Conclusion

Multi-threaded operation of squashfuse doesn't help performance in the case of traversing the filesystem with tree.

ariel-miculas commented 1 year ago

Setup:

puzzlefs:

$ target/release/puzzlefs mount -f /tmp/oci-barehost test /tmp/puzzle

squashfuse:

squashfuse -f barehost.sqhs /tmp/squash

Result:

squashfuse

$ time tree /tmp/squash
2325 directories, 19265 files
tree /tmp/squash  0.08s user 0.17s system 8% cpu 2.893 total

puzzlefs without memory mapping

$ time tree /tmp/puzzle
2325 directories, 19265 files
tree /tmp/puzzle  0.10s user 0.37s system 1% cpu 44.427 total

puzzlefs with memory mapping of the metadata file(s)

2325 directories, 19265 files
tree /tmp/puzzle  0.07s user 0.17s system 13% cpu 1.764 total