merging: Support request size negotiation and increased throughput on high-latency connections

[h4sh5]

attempting to merge PR https://github.com/libfuse/sshfs/pull/267

[h4sh5]

@ccope all the tests pass, it works well, it's going ahead. Thanks for the code!

[rozhuk-im]

I have mounts with ~70ms RTT, see no performance increase. Here my patches https://github.com/rozhuk-im/sshfs to increase file load speed, it works for me.

Original sshfs/latest commit from this repo gives to me up to 1,5mb/sec. With -o max_read=261120 got ~ 2mb/sec, that try with 67108864 got hang on dir listing. With my patches I got up to 9 mb/sec.

sshfs -o reconnect -o remember=30 -o auto_cache -o cache=yes -o kernel_cache -o compression=yes -o max_write=261120 -o max_read=261120 -o dir_cache=yes -o noatime

Also looks like: https://github.com/deadbeefsociety/sshfs/issues/8 is regress after this PR.

PS: test done on FreeBSD<->FreeBSD, may be kernel does not handle read_ahead.

[rozhuk-im]

I also play with max read/write size and got: https://github.com/libfuse/sshfs/commit/de03645540f72671a19022efdef6947ff512d931

It depend on MAX_REPLY_LEN - "some data from packet header".

MAX_REPLY_LEN (1 << 17) -> DEFAULT_MAX_SFTP_MSG_LIMIT 65536 a bit strange replace.

Idea with limits is good but some questions to implementation )

PS: there is 3 separate changes:

1. limits
2. max_readahead
3. uids

[ccope]

Before the request size negotiation feature was implemented, the protocol limited requests to 64kb, so the client just hardcoded the limit. When OpenSSH merged size negotiation, they bumped up the default limit to 256kb. It's been a while since I've experimented with this, but (if I remember correctly) I either recompiled the server binary myself with a custom limit, or determined that there was a way to run the server without any limits of its own, in which case the client limit is used.

[ccope]

Also, for reference, I am dealing with ~200ms latency connections.

[ccope]

The trickiest part of this PR was that there was a bunch of hand-rolled synchronous requests being made during the creation of the SFTP session. As I needed to add yet another synchronous call for limit negotiation, I converted the asynchronous request flow to a stripped down synchronous version and ported all of the other synchronous calls to use it as well. That's why you see changes related to UID and check root.

[ccope]

Ok, I did some checking. The openssh sftp server changed SFTP_MAX_MSG_LENGTH in sftp-common.h from 64kb to 256kb when they added request size negotiation. For my tests I patched the limit to 4MB. I haven't gotten around to making a PR to the openssh project to make that argument a command line flag/config file setting.

[rozhuk-im]

Most common issue with low speed on big RTT is not segment size, but algo how it work. Client send read request, wait reply and send next request. In that case speed = read_seg_size * segs_per_sec. segs_per_sec depend on RTT.

IMHO it is possible to implement read_ahead feature inside sshfs and not depend on kernel. If read_ahead will send 3+ read requests (ahead) while first segments loading than even small segment size will not downgrade throughput.

In my case some how FreeBSD send to fuse or fuse to sshfs small read blocks, even if I play with dd ... bs=1m - sshfs got 128k (not remember exactly, but small) read request.

Also TCP CC = hybla (may be bbr) for linux and rack+htcp for FreeBSD allow speed up segment transmit speed.

[ccope]

There is readahead logic in libfuse, but it's disabled if it detects the kernel will do readahead. When I first looked into this Linux kernel readahead had a hard coded cap of, iirc, 128kb. This might have changed in the last few years, I'll have more time to investigate later this week.