search

yasukata / tinyhttpd-lwip-dpdk

A tiny HTTP server built on lwIP and DPDK
Apache License 2.0
33 stars 7 forks source link

Poor performance for large HTTP responses #1

Closed deneuvillem closed 1 year ago

deneuvillem commented 1 year ago

Hello,

First of all, thank you for making this project available, it is very interesting.

I managed to build the project and run it in a cluster of two machines to make some benchmarking like you did in the last section. I obtain similar results for content_length=1 but once I increase it to be more than 1395 bytes, the performance becomes very poor for lwip-dpdk (about 200 requests/s and 40ms of average latency for content_length=1396, 1 client thread, 1 client connection and a delay of 10 seconds).

Those results were obtained using wrk tool on machines with those specifications:

I suspect that the performance issue has a link with the MSS and/or MTU of packets because default MSS=1460 in "lwipopts.h" and default MTU=1500 (I printed it on the main program to be sure). Note that the maximum payload size of packets before obtaining poor performance is 1395 (content_length) + 65 (header size) = 1460, which is the same as the MSS (hence my suspicion).

So I tried to increase the TCP_MSS value in "lwipopts.h" and use rte_eth_dev_set_mtu() function in "main.c" to increase the MTU size but it does not seem to work. I also tried to increase the MTU of my bounded NIC with ifconfig.

I don't know if you have an idea how I could fix this to get better performance.

Thank you!

yasukata commented 1 year ago

Thank you for your detailed report.

I have looked into the program, and I think I could fix this issue.

The issue seemed to come from the Nagle algorithm, which is turned on in lwip by default.

Before disabling the Nagle algorithm (commit id: cb80838), the throughput measured by the following commands was:

./wrk http://10.0.0.2:10000/ -d 10 -t 16 -c 64 -L
sudo LD_LIBRARY_PATH=./dpdk/install/lib/x86_64-linux-gnu ./app -l 0-1 --proc-type=primary --file-prefix=pmd1 -a 0000:04:00.1 -- -a 10.0.0.2 -g 10.0.0.1 -m 255.255.255.0 -l 1500 -p 10000
number passed to -l (content_length) throughput
1000 1.01GB
1500 8.58MB
2000 6.09MB
2500 7.27MB
3000 1.02GB
3500 1.10GB
4000 1.10GB

After disabling the Nagle algorithm, I got the following numbers.

number passed to -l (content_length) throughput
1000 0.98GB
1500 807.32MB
2000 1.07GB
2500 1.09GB
3000 1.08GB
3500 1.09GB
4000 1.10GB

The change, disabling the Nagle algorithm, is pushed to this repository and please find it at commit 309c7ca .

I hope this result could be reproduced in your environment.

Thank you for reporting this issue.

deneuvillem commented 1 year ago

I measured the performance with your fix and I obtain similar results on my environment.

Thank you very much for the help!