nmanchinn commented 2 years ago

Issue Type: Bug Version: 1.14.7 OS: Linux Distribution: Debian stretch

We are trying to use bindfs and observed the bindfs performance overhead.

My observations are detailed below:

( observations taken from a simple filesystem (SSD), users are local to the machine and a decent hardware allocation - 4cpu, 16GB RAM )

Create a 1K file for copying

$ dd if=/dev/zero of=/tmp/file1.txt count=1000 bs=1k 1024000 bytes (1.0 MB, 1000 KiB) copied, 0.015898 s, 64.4 MB/s

create test folders

$ mkdir /tmp/test1 /tmp/test1a /tmp/test1b

copy to a local folder

$ cd /tmp/test1; rm -rf foo; mkdir foo; time for i in {1..500}; do cp /tmp/file1.txt foo/file$i.txt; done real 0m1.150s

do a bindfs mount

$ /usr/bin/bindfs -o '' '/tmp/test1' '/tmp/test1a'

copy to a bindfs mount

$ cd /tmp/test1a; rm -rf foo; mkdir foo; time for i in {1..500}; do cp /tmp/file1.txt foo/file$i.txt; done real 0m8.412s

copy to a bindfs mount, additional flags

$ /usr/bin/bindfs -o '' '/tmp/test1' '/tmp/test1b' --read-rate=1T --write-rate=1T $ cd /tmp/test1b; rm -rf foo; mkdir foo; time for i in {1..500}; do cp /tmp/file1.txt foo/file$i.txt; done real 0m17.849s

Note: It's even slow when we share the bindfs from a different fuse mount

Please let me know if you need more details. Thanks in advance!

mpartel commented 2 years ago

It is surprising that the rate limiting flags have such a large impact. That appears to be a bug.

Other than that, it doesn't surprise me that bindfs is by default quite a bit slower than direct filesystem access. Some overhead will always be unavoidable, but I think some of it is due to various caches being disabled, because the source directory's content can in general change independently. You can experiment with the flags documented in man mount.fuse.

Also, Debian Stretch is quite old and has only FUSE 2, but FUSE 3 comes with new perormance-related options, such as writeback_cache.

If you're interested, you can also compare bindfs performance to the example passthrough filesystems that come with FUSE (called fusexmp_fh in the FUSE 2 branch). I'd expect them to be similar, given the same caching flags.

nmanchinn commented 2 years ago

Thank you for the quick response. I will experiment with few fuse flags to see if there are any differences. I will share my findings soon.

nmanchinn commented 1 year ago

Hi Martin,

Thank you for the suggestion and apologies for getting back to you late. Please find our findings below:

Upgrading to FUSE 3 resulted in a huge performance boost.
The rate limit flags resulted in little to no impact and the amount of time it took to write the 500kb was only about 4.1 seconds compared to a local folder which was about 2.2 seconds.
Local Folder real 0m2.248s user 0m0.623s sys 0m0.528s
Bindfs 1.17 real 0m3.820s user 0m0.628s sys 0m0.553s
Bindfs 1.17 with --read-rate=1T --write-rate=1T real 0m4.098s user 0m0.679s sys 0m0.551s

These results are not averages and might have some margin of error. However, they are enough to say FUSE 3 is yielding better performance than the default binary available on Debian 11, so thank you for the suggestion!

Also, I'm wondering if bindfs can be compiled and statically linked with libfuse3 and released to apt-get?

mpartel commented 1 year ago

It's interesting that FUSE 3 has such a large impact. Thanks for following up!