search

fixstars / lightning-kit

Reactive data transfer framework
Other
1 stars 0 forks source link

doca tcp frame builder performance improvement #10

Open masaruito110 opened 3 months ago

masaruito110 commented 3 months ago

Purpose

Based on d0ea36e28bdebb977c7f6fe949771fec065d9b40 I measured the performance of docagpunetio.

Current server structure is below

|doca flow| --------(tcp dst_port 1234)--> |frame builder|
                |---(tcp dst_port 1235)--> |frame builder|
                    : (number of server instances I specified)

frame builder structure

|receive_tcp|<--semaphore-->|send_ack| <--semaphore-->|makeframe|<--semaphore-->|notify frame built|

receive_tcp polls doca_gpu_dev_eth_rxq_receive_warp, send_ack send ack to client and calculate the latest seq number for make frame, make frame builds frames using cudaMemcpyAsync.

According to some trial, the throughput is influenced by client side ack checking frequency, and the number of sessions.

Environment

1
|connectx7 on PCIe4|<------>|connectx7 on PCIe3|
|A100 40GB GPU on PCIe4|

2
|connectx7 on PCIe4|<------>|connectx7 on PCIe3|
                    |<----->|connectx6 on PCIe3|
|A100 40GB GPU on PCIe4|

Result

Here is the result. env is the Environment described in Environment session. process means the number of processes. session/process means the number of sessions per process, when process is 2 and session/process is 1, total number of sessions is 2. chunk size is that client checks ack from server every time when it sends this number of bytes. Gbps/session is the throughput per session.

Theoretically, when env is 1, the total throughput is 100Gbps, so we expect we can get 100Gbps when 1 session, 50Gbps/session when 2 sessions. When env is 2, we use 2 ports of connectx7, so the total hroughput is 200Gbps

env process session/process chunk size [MByte] Gbps/session
1 1 1 1 38.9
1 1 1 2 39.39
1 1 1 4 41.72
1 1 1 8 43.46
1 1 1 16 44.36
1 1 2 1 6
1 2 1 1 18.06
2 2 1 1 17.52

The result when env is 1, process is 1 and session/process is 1, over 16MByte chunk size doesn't work because the cyclic buffer handled by doca overwritten. The chunk size increased, the throughput improved. This means that the average RTT is long.

The result when env is 1 and 2, process is 2 and session/process is 1, we get half of result when env is 1, process is 1 and session/process is 1. We expected the throughput is the same because NIC bandwidth, PCIe bandwidth and GPU device memory bandwidth is enough. So there is a limitation or limitations in doca library.

The result when env is 1, process is 1 and session/process is 2, we only get 6Gbps. We expected we can get the same result of when process is 2 and session/process is 1. Maybe cuda kernels affect other kernels in the same process.

masaruito110 commented 3 months ago

Additonal report.

Frame builder part skips frame building and only send ack.

env process session/process chunk size [MByte] Gbps/session
1 1 1 32 90.93
1 1 2 32 49.1
2 2 1 32 55.59
2 2 1 128 88.88

(env, process, session/process) are (1, 1, 1) and (1, 1, 2) is close to theoretical performance. (2,2,1), when chunk size is 128MB, then we got high performance. We can say when there is multiple sessions, the latency get worse, but potentially we can get high throughput.

masaruito110 commented 3 months ago

Addtional report same as https://github.com/fixstars/lightning-kit/issues/12#issuecomment-2066911763

I run heavy mempy kernel while only send ack app (Frame builder without frame building).

The result is below. We measured the performance when there are one or two heavy memcpy runing. The chunk size is enough, then we can get high performance, but the performance with chunk size that frame building can work is slow and similar to the result of actual frame building performance. Also the heavy mempcy increase the performance decrease.

env process session/process chunk size [MByte] Gbps/session with one heavy memcpy Gbps/session with two heavy memcpy
1 1 1 1 27.82 17.72
1 1 1 2 34.63 23.08
1 1 1 4 41.54 27.69
1 1 1 8 41.56 27.69
1 1 1 16 55.43 36.93
1 1 1 32 55.44 36.96
1 1 1 40 69.21 46.16
1 1 1 48 83.13 55.46
1 1 1 64 83.13 73.78
1 1 1 128 88.7 88.53

The performances without heavy memcpy

env process session/process chunk size [MByte] Gbps/session
1 1 1 1 63.02
1 1 1 2 72.31
1 1 1 4 81.92
1 1 1 8 91.04
1 1 1 16 89.52
1 1 1 32 91.13
masaruito110 commented 2 months ago

Performance on VMWare VM.

VM has RTX A6000 Ampere arch, ConnectX6, PCIe gen2, CPU is Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz

process session/process chunk size [MByte] Gbps/session
1 1 1 46.38
1 1 2 55.52
1 1 4 60
1 1 8 60

RTT performance Sent 16k times. RTT is usec.

chunk size [MByte] rtt ave rtt stddev rtt min rtt max
1 147.4199 11.37497 130.888 242.101
2 265.0977 11.41619 240.277 319.813
4 486.8668 19.04014 454.549 549.641
8 945.3377 30.31007 900.802 1025.787