search

chaos / diod

Distributed I/O Daemon - a 9P file server
GNU General Public License v2.0
349 stars 56 forks source link

[Question] Performance bottleneck for large files #59

Open lfourquaux opened 4 years ago

lfourquaux commented 4 years ago

Where is the performance bottleneck for reading large files through diod?

Here is a micro-benchmark: Linux v9fs: 300Mb/s diodcat (default msize): 400Mb/s direct tcp (netcat): 900Mb/s This is for reading a 1GB file through a 1Gb/s Ethernet connection (latency 0.4ms, mtu 9000).

Why is diod that much slower at transferring data than netcat? Would it be possible to increase the max msize on the server and would it help?

garlick commented 4 years ago

I'm not sure. As I recall when I was really working on performance, I used a 64K msize. That was a long time ago.

lfourquaux commented 4 years ago

I tried to change the hardcoded msize=64K in diod/ops.c (to msize=1M). Hopefully there is not hidden assumption elsewhere that msize is not too large. (Maybe it could be made configurable from the command line?) Using diodcat (also msize=1M), I get close to the raw tcp throughput. On the other hand, v9fs limits msize to 64K for fd transport (see MAX_SOCK_BUF in net/9p/trans_fd.c) so it's not very useful. Maybe should suggest changing MAX_SOCK_BUF (to 2M, maybe)?

garlick commented 4 years ago

I vaguely remember that v9fs or the 9p transport requires the allocation of msize+overhead slab buffers, and so the current max msize may be the largest that can be handled practically. Maybe that is no longer an issue though - the kernel has changed a lot since I last studied this (plus my memory is not great).

nkichukov commented 3 years ago

This will be fixed by: https://github.com/chaos/diod/pull/67 and running a 5.15 kernel (I am not sure if the kernel change is a hard prerequisite but my tests were only done with a patched kernel, thus MAX_SOCK_BUF set to 1MB).

I have backported the kernel changes mentioned in the pull request to a 5.14.10 and also backported the patch to diod version 1.0.24 and I have confirmed the setup works stable and the throughput is much faster when transferring big files instead of the previous maximum msize of 64KB.