Mellanox / sockperf

Network Benchmarking Utility
Other
585 stars 119 forks source link

issue: 2681317 RX zero-copy flow implementation is limited #156

Closed agalanin-at-nvidia closed 2 years ago

agalanin-at-nvidia commented 2 years ago

Summary of bugs

  1. TCP, zerocopy, packet size > MTU: packet is returned from zcopy_recv as two chunks of 1460+20 bytes but sometimes this is a two packets of 1460 and 20 bytes. In the second case message content pointer is set to beginning of a second packet payload. So packet header is incorrect in this case. This is a possible root cause of #2826990.
  2. zerocopy, TCP, size > MTU: ping-pong data integrity check failed even for the first message because reply sender does not handle fragmented packet data properly: only the first fragment is valid, data from all other fragments are just skipped.
  3. zerocopy: if zerocopy is not performed, the program uses invalid offset in buffer to retrieve data (extra offset to 'fd' bytes).
  4. zerocopy: handles data only from the first packet of recvfrom_zcopy return value. Not reproducible after #1714768 is fixed in libvma.
  5. vmarxfiltercb: callback code always copies input packet data even if message resides entirely in a single data chunk.

Summary of changes

Additional changes

Performance comparison

original sockperf version: 3.7-11.gita57ffb579002 VMA 9.4.0 Message size: 1460 bytes

TCP ping-pong latency, us:

socket VMA zcopy filtercb xtreme
original 19.6 5.1 5.2 4.6 3.8
modified 18.6 5.0 4.9 4.5 3.8

UDP ping-pong latency, us:

socket VMA zcopy filtercb xtreme
original 16.1 3.8 3.8 3.7 3.4
modified 15.9 3.8 3.8 3.8 3.5

TCP throughput, Gbps:

socket VMA zcopy filtercb xtreme
original 7.8 18.5 16.5 🔴 8.2 14.5
modified 7.8 18.7 16.6 🟢15.3 14.4

UDP throughput, Gbps:

socket VMA zcopy filtercb xtreme
original 3.8 22.8 22.8 22.8 22.7
modified 3.8 22.8 22.8 22.8 22.8

I suggest to enable 'hide whitespace' option while reviewing a diff.

agalanin-at-nvidia commented 2 years ago

Just pushed cosmetic commit re-ordering and squash.