bindfs bottlenecks copy speed

mpartel / bindfs

Mount a directory elsewhere with changed permissions.

https://bindfs.org/

GNU General Public License v2.0

448 stars 64 forks source link

bindfs bottlenecks copy speed #156

Closed jvllmr closed 1 month ago

jvllmr commented 1 month ago

Backstory: I'm currently working on a project where large video files have to be processed as fast as possible. However, we run into issues with the correct permissions set on Linux, macOS and Windows clients. To throw the problem out of the window completely we decided to give bindfs a try wth perms=777 and create-with-perms=777.

Issue: When I copy a large video file inside a folder with rsync on the SAN client filesystem I reach around 500MB/s, but with my bindfs layer only 100MB/s. I could improve the copy time to 200MB/s by building 1.17.7 locally and using the direct-io option, but that's still not enough.

Are there tricks to make bindfs perform close to the original filesystem?

jvllmr commented 1 month ago

Actually investigating into this further it seems that fuse is the root reason for the overhead.

I link some sources here if anyone is interested: https://www.usenix.org/system/files/conference/fast17/fast17-vangoor.pdf

https://dl.acm.org/doi/pdf/10.1145/3678015.3680493

mpartel commented 1 month ago

Yes, FUSE unfortunately adds quite a bit of overhead.

Some quick ideas:

Maybe you can find a way to tell FUSE to transfer more data per read/write call. max_read here might be relevant, and perhaps there are some kernel settings somewhere too.
You can try modifying the FUSE settings set in bindfs_init. Maybe enable some caches and try parallel_direct_writes together with multithreaded mode. Just be aware that these things can open the door for temporary inconsistencies seen by simultaneous viewers of the FS, but with the flags you're using, that probably won't be an issue.

jvllmr commented 3 weeks ago

Thanks for the hints. I will take a look.