Performance measures with iperf

deneuvillem commented 1 year ago

Hello, I am doing performance measures with the "iperf" tool and I get strange results. Indeed, it looks like the Linux stack server performs better than LwIP-DPDK when using "iperf" (while it was the opposite using "wrk" tool). I used the command iperf -c <server_ip> -p <server_port> -t 20 -P 4 -B <client_ip> on the client side and on the server side, I run the server and set the parameter content_length to 1500 bytes.

Here are results I obtained with this command on my environment (the same as in my previous issue):

For Linux stack:

[ ID] Interval       Transfer     Bandwidth
[  4] 0.0000-20.0035 sec  5.48 GBytes  2.36 Gbits/sec
[  5] 0.0000-20.0038 sec  5.48 GBytes  2.36 Gbits/sec
[  3] 0.0000-20.0059 sec  5.49 GBytes  2.36 Gbits/sec
[  6] 0.0000-20.0008 sec  5.48 GBytes  2.36 Gbits/sec
[SUM] 0.0000-20.0008 sec  21.9 GBytes  9.42 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) = 0.176/0.212/0.263/0.157 ms (tot/err) = 4/0

For LwIP-DPDK

[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-20.0005 sec  2.41 GBytes  1.03 Gbits/sec
[  5] 0.0000-20.0001 sec  2.43 GBytes  1.04 Gbits/sec
[  4] 0.0000-20.0005 sec  2.39 GBytes  1.03 Gbits/sec
[  6] 0.0000-20.0003 sec  2.39 GBytes  1.03 Gbits/sec
[SUM] 0.0000-20.0003 sec  9.62 GBytes  4.13 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) = 0.070/0.113/0.158/0.065 ms (tot/err) = 4/0

I also ran multiple times the experiment with different content_length and the Linux stack was always performing better than LwIP-DPDK in terms of total throughput, average throughput and transfer size.

I was wondering if you get similar results.

yasukata commented 1 year ago

Thank you for your message.

I run the same iperf test, and I got a similar result to the one you provided.

I’m sorry but, for the moment, I do not know the exact reason why this occurs.

While I could not figure out the root cause, let me share my investigation.

First, although I have not confirmed, I believe that the Linux TCP stack should be faster than our LwIP+DPDK implementation for the bulk data transfer because the Linux TCP stack can fully utilize the NIC offloading features (e.g., TSO and checksumming) while our LwIP+DPDK implementation cannot.

However, 1 Gbps for a single TCP connection is a bit too slow and our implementation can be slightly faster, I guess.

To look into a bit more details, I run tcpdump; in the following output, ubuntu runs iperf, and 10.0.0.2 runs the Linux-TCP/LwIP+DPDK web servers (by the way, in this setup, iperf seems to be the primary sender, and our web servers mainly receive the data and just reply TCP ACKs to iperf; in this sense, this iperf test measures the RX performance of our web servers).

Linux TCP case

11:59:27.279248 IP ubuntu.37353 > 10.0.0.2.webmin: Flags [P.], seq 2347814749:2347879909, ack 1, win 502, options [nop,nop,TS val 3358256955 ecr 283892208], length 65160
11:59:27.279319 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347286229, win 23185, options [nop,nop,TS val 283892209 ecr 3358256954], length 0
11:59:27.279319 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347358629, win 22749, options [nop,nop,TS val 283892209 ecr 3358256954], length 0
11:59:27.279326 IP ubuntu.37353 > 10.0.0.2.webmin: Flags [P.], seq 2347879909:2347945069, ack 1, win 502, options [nop,nop,TS val 3358256955 ecr 283892209], length 65160
11:59:27.279335 IP ubuntu.37353 > 10.0.0.2.webmin: Flags [P.], seq 2347945069:2348010229, ack 1, win 502, options [nop,nop,TS val 3358256955 ecr 283892209], length 65160
11:59:27.279401 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347371661, win 23185, options [nop,nop,TS val 283892209 ecr 3358256955], length 0
11:59:27.279401 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347423789, win 22999, options [nop,nop,TS val 283892209 ecr 3358256955], length 0
11:59:27.279401 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347457093, win 22999, options [nop,nop,TS val 283892209 ecr 3358256955], length 0
11:59:27.279408 IP ubuntu.37353 > 10.0.0.2.webmin: Flags [P.], seq 2348010229:2348075389, ack 1, win 502, options [nop,nop,TS val 3358256955 ecr 283892209], length 65160
11:59:27.279482 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347488949, win 23007, options [nop,nop,TS val 283892209 ecr 3358256955], length 0
11:59:27.279482 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347554109, win 23176, options [nop,nop,TS val 283892209 ecr 3358256955], length 0
11:59:27.279488 IP ubuntu.37353 > 10.0.0.2.webmin: Flags [P.], seq 2348075389:2348140549, ack 1, win 502, options [nop,nop,TS val 3358256955 ecr 283892209], length 65160
11:59:27.279500 IP ubuntu.37353 > 10.0.0.2.webmin: Flags [P.], seq 2348140549:2348205709, ack 1, win 502, options [nop,nop,TS val 3358256955 ecr 283892209], length 65160
11:59:27.279566 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347619269, win 23128, options [nop,nop,TS val 283892209 ecr 3358256955], length 0
11:59:27.279566 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347629405, win 23128, options [nop,nop,TS val 283892209 ecr 3358256955], length 0
11:59:27.279573 IP ubuntu.37353 > 10.0.0.2.webmin: Flags [P.], seq 2348205709:2348270869, ack 1, win 502, options [nop,nop,TS val 3358256955 ecr 283892209], length 65160
11:59:27.279651 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347684429, win 22958, options [nop,nop,TS val 283892209 ecr 3358256955], length 0
11:59:27.279651 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347724973, win 22958, options [nop,nop,TS val 283892209 ecr 3358256955], length 0
11:59:27.279652 IP 10.0.0.2.webmin > ubuntu.37353: Flags [.], ack 2347749589, win 23048, options [nop,nop,TS val 283892209 ecr 3358256955], length 0

LwIP+DPDK case

12:04:22.767310 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248786957, win 20480, length 0
12:04:22.767320 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248786997:248788417, ack 1, win 64240, length 1420
12:04:22.767326 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248788417:248795717, ack 1, win 64240, length 7300
12:04:22.767331 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248795717:248798637, ack 1, win 64240, length 2920
12:04:22.767337 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248798637:248807397, ack 1, win 64240, length 8760
12:04:22.767439 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248788417, win 20480, length 0
12:04:22.767439 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248791337, win 20480, length 0
12:04:22.767439 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248794257, win 20480, length 0
12:04:22.767439 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248797177, win 20480, length 0
12:04:22.767439 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248800097, win 20480, length 0
12:04:22.767439 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248803017, win 20480, length 0
12:04:22.767439 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248805937, win 20480, length 0
12:04:22.767448 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248807397:248808857, ack 1, win 64240, length 1460
12:04:22.767454 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248808857:248817657, ack 1, win 64240, length 8800
12:04:22.767460 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248817657:248819077, ack 1, win 64240, length 1420
12:04:22.767464 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248819077:248826377, ack 1, win 64240, length 7300
12:04:22.767567 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248808857, win 20480, length 0
12:04:22.767567 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248811777, win 20480, length 0
12:04:22.767567 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248814697, win 20480, length 0
12:04:22.767567 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248817617, win 20480, length 0
12:04:22.767568 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248819077, win 20480, length 0
12:04:22.767568 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248821997, win 20480, length 0
12:04:22.767568 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248824917, win 20480, length 0
12:04:22.767577 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248826377:248829297, ack 1, win 64240, length 2920
12:04:22.767595 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248829297:248838057, ack 1, win 64240, length 8760
12:04:22.767602 IP ubuntu.34269 > 10.0.0.2.webmin: Flags [P.], seq 248838057:248839517, ack 1, win 64240, length 1460
12:04:22.767696 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248827837, win 20480, length 0
12:04:22.767696 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248830757, win 20480, length 0
12:04:22.767697 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248833677, win 20480, length 0
12:04:22.767697 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248836597, win 20480, length 0
12:04:22.767697 IP 10.0.0.2.webmin > ubuntu.34269: Flags [.], ack 248839517, win 20480, length 0

As seen, in the Linux TCP case, the data size sent by the iperf host (65160 bytes) is bigger than that in the LwIP+DPDK case (1420~8800 bytes).

I think this would be a hint to figure out the reason of the low performance of our current implementation.

I thought TCP configuration in lwipopts.h may resolve this issue and tried several options (e.g., LWIP_WND_SCALE and TCP_RCV_SCALE), but I do not have a good result so far.

I will update the implementation when I find out the solution.

Thank you for reporting this.

deneuvillem commented 1 year ago

Thank your for your investigation and insights.

Something that I tried is to modify the length of the payload the iperf client sends to the server with "-l" parameter. When reducing it, it decreases the observed throughput of both implementations of server but still does not solve the problem.

I also think that payloads sent by iperf client are randomly generated while the server expects to receive the "GET" string to send a proper reponse to the client (I tested it by opening a netcat client and send manually the string "GET" and I receive the correct response). So maybe it would be more accurate to test with a custom payload so that the client can receive reponses other than ACKs.

Another thing that is possible is to disable Nagle algorithm with "-N" parameter. Unfortunately, it did not seem to solve the issue either. Other parameters like TCP window size (-w) and MSS (-M) can also be changed in iperf but I am not sure it changes anything.

yasukata commented 1 year ago

Thank you for sharing your thoughts. They are interesting points. I will keep them in mind for further investigation.

Now, I feel like I need to look into the TCP implementation to understand what limits the transmission data size on the sender side.

I will inform you when I know more.

yasukata / tinyhttpd-lwip-dpdk

Performance measures with iperf #2