Performance flags for use over sshfs (Mac)

[knaccc]

I'm mounting a cipher directory over a 1GbE wired LAN via sshfs, and then mounting that via gocryptfs:

sudo mkdir /Volumes/cipher
sudo chown me:staff /Volumes/cipher
sshfs remoteuser@remote:/cipher /Volumes/cipher -o \
reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,allow_other,noapplexattr,noappledouble,\
defer_permissions,volname=displayName,IdentityFile=/Users/me/.ssh/id_ed25519
sudo mkdir /Volumes/plain
sudo chown me:staff /Volumes/plain
gocryptfs /Volumes/cipher /Volumes/plain

If I copy a 10GB file to the /Volumes/cipher directory (i.e. not via gocryptfs), it copies at 76.4 MB/s. If I copy a 10GB file to the /Volumes/plain directory (i.e. via gocryptfs), it copies at 33.8 MB/s.

gocryptfs -speed reports AES-GCM-256-Go 3963.06 MB/s (using gocryptfs v2.2.1 without_openssl)

Can anyone recommend any sshfs or other flags that might help increase performance please?

[knaccc]

After lots of searching, I've found that I have been able to increase r/w performance on the raw sshfs connection to 110 MB/s each way, and r/w performance via gocryptfs to 72MB/s write, 110MB/s read.

This was achieved by modifying and compiling sshfs 2.10 using instructions here: https://github.com/osxfuse/sshfs/issues/64

There is still the mystery of why gocryptfs is limited to 72MB/s write despite the sshfs connection being capable of 110MB/s.

Perhaps there is scope for gocryptfs in the future to use larger write buffers or other means to achieve higher throughput over sshfs?

[rfjakob]

The numbers you get look pretty good, but yes, gocryptfs performance over sshfs is limited for two reasons:

• FUSE request size is limited to 128kiB. On latest Linux kernels, this can be increased to 1MB ( https://github.com/hanwen/go-fuse/issues/309 ). But gocryptfs does not do that yet, and I don't know if this is possible on MacOS at all. So all writes and reads are limited to 128kiB.
• Writes run serially, one after the other (reads are parallelized, at least on Linux). This means the added latency from SSHFS hurts a lot. 1ms of latency limits the theoretical write throughput to 128 MiB/s already (1/1ms * 128kiB)

[knaccc]

@rfjakob Thanks for the insight.

FYI the performance tests above were from a Mac to a Linux server, with an average ping of 0.70ms.

I repeated the experiment to a remote Linux server with an average ping of 87ms, and I achieved an upload speed of 0.36 MB/s and a download speed of 2.20 MB/s via gocryptfs (over SSHFS).

Via SSHFS only, I get an upload speed of 1.5MB/s and a download speed of 13.0 MB/s.

[ctroo]

I'm using gocryptfs over sshfs between two linux cloud servers.

via only SSHFS: 83.7 MB/s

via gocryptfs v1.8: 14.5 MB/s

via gocryptfs v2.3: 5.4 MB/s

@rfjakob thanks for this awesome piece of software! I suspect my speeds are below expectation even considering the above explanation. Any reason my speed deteriorates with the newer version? Is there anything I can try to improve throughput?

gocryptfs --speed
gocryptfs 1.8.0; go-fuse 2.0.3; 2021-07-18 go1.15.9 linux/amd64
AES-GCM-256-OpenSSL      670.42 MB/s    
AES-GCM-256-Go          1314.56 MB/s    (selected in auto mode)
AES-SIV-512-Go           170.01 MB/s    
XChaCha20-Poly1305-Go    814.12 MB/s

gocryptfs --speed
gocryptfs v2.3 without_openssl; go-fuse v2.1.1-0.20211219085202-934a183ed914; 2022-08-28 go1.19 linux/amd64
cpu: AMD EPYC 7282 16-Core Processor; with AES acceleration
AES-GCM-256-OpenSSL             N/A
AES-GCM-256-Go              2738.07 MB/s    (selected in auto mode)
AES-SIV-512-Go               223.17 MB/s
XChaCha20-Poly1305-OpenSSL      N/A
XChaCha20-Poly1305-Go       1630.10 MB/s    (selected in auto mode)

sshfs --version
SSHFS version 3.7.1
FUSE library version 3.10.3
using FUSE kernel interface version 7.31
fusermount3 version: 3.10.3