Closed chucklever closed 2 months ago
One suggestion for managing this situation is that NFSD should respect the client's advertised IRD (Incoming Read Depth). Currently, no Linux kernel ULP pays attention to the IRD/ORD limits. Rather, they all depend on the RDMA layer and NICs to manage RDMA Read queuing.
Another thought was that the Linux NFS/RDMA client should allow its NIC to retransmit more frequently so that the workflow can detect and recover from lost RDMA operations more quickly.
I've set up an NFS/RDMA client system with a FastLinQ RoCE card on a 25Gbps port, and an NFS/RDMA server system with a CX-5 Ethernet card on a 100Gbps port. The usual workloads seem to run comfortably on this rig without loss of connection. So basic function works, but I'm not able to reproduce significant losses.
The original issue was reported as long delays when handling server failover / reboot scenarios on RoCE. This issue can be explained by the improper retention of stale ARP cache entries on the clients. There does not appear to be an ARP cache flush after connection errors (as usually happens after a TCP connection failure, say). This is being explored separately.
I've discussed the use of explicit ORD/IRD checking with a few upstream RDMA experts, who agree that better performance will be achieved leaving that to drivers and devices.
NFS is frequently deployed on fabrics where the NFS server's NICs are far more capable than the client NICs. This is less true on RDMA fabrics, where most NICs are fast and have similar capabilities.
However with the advent of both RoCE and software-emulated RDMA, it's possible for server NICs to far out-class client capabilities. In these situations, the server can easily overrun a client when sending RDMA Reads or Write, even though the ULP (NFS in this case) does not have much control over the network layer. RDMA is not known for handling network environments that are lossy.