search

F-Stack / f-stack

F-Stack is an user space network development kit with high performance based on DPDK, FreeBSD TCP/IP stack and coroutine API.
http://www.f-stack.org
Other
3.87k stars 899 forks source link

Nginx Benchmarking with Linux TCP/IP stack and F-stack #249

Open jeffinjoyvvdn opened 6 years ago

jeffinjoyvvdn commented 6 years ago

Hi F-stack team, I am trying to comparing the performance of Nginx server in Linux TCP/IP stack and F-stack. This is my test setup. Ixia---------40G--------(Intel 40G NIC card)-Dell R730 Server (Ubuntu 16.04, ker: 4.13, RAM: 64GB, processor: Intel Xenon @2.60GHz *52 Linux TCP/IP stack is giving good performance compare to F-stack. Here I am attaching the f-stack.conf and nginx.conf. Could you please check in the configuration that any misses.

Here I am trying to download different size file from web server using Ixia. And I am running Nginx process on different CPU core. The below test result is for 8 CPU core for Nginx (work process=8) in Linux stack.

File size BW RPS CPS
1 KB 248 Mbps 21218 21218
10 KB 2.3 Gbps 26843 26843
100 KB 20.1 Gbps 24441 2441
1 MB 19.1 Gbps 2331 2331
10 MB 20 Gbps 244 244
100 MB 19.5 Gbps 24 24
1 GB 22.4 Gbps 3.6 3.6

The below test result is for F-stack with 8 CPU core.

File size BW RPS CPS
1 KB 7.8 Mbps 669 669
10 KB 76 Mbps 895 895
100 KB 5.7 Gbps 7036 7036
1 MB 21.2 Gbps 2588 2588
10 MB 22.2 Gbps 271 271
100 MB 21.2 Gbps 26 26
1 GB 23.8 Gbps 5 5

Issues:

  1. While starting the Nginx, CPU utilization is getting 100, without any traffic from IXIA. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    4090 root 20 0 8472108 28876 6388 R 100.0 0.0 1:47.38 nginx
    4093 root 20 0 8472124 28304 5796 R 100.0 0.0 1:44.82 nginx
    4097 root 20 0 8472124 28304 5796 R 100.0 0.0 1:44.82 nginx
    4101 root 20 0 8472124 28276 5768 R 100.0 0.0 1:44.81 nginx
    4103 root 20 0 8472124 28304 5796 R 100.0 0.0 1:44.82 nginx
    4105 root 20 0 8472124 28304 5796 R 100.0 0.0 1:44.81 nginx
    4106 root 20 0 8472124 28304 5796 R 100.0 0.0 1:44.81 nginx
    4092 root 20 0 8472124 28304 5796 R 99.7 0.0 1:44.81 nginx

  2. As you can see in the above table, very low performance for the lower size packet(1KB). Anything in configuration needs to enabled for this?

Please fine the attachment. nginx.conf.txt f-stack.conf.txt

jfb8856606 commented 6 years ago
  1. DPDK use poll mode, CPUs utilization always 100%, the real CPU utilization can get use the top tool tools/top/top.
  2. I find some config f-stack.conf need to fix, but all of them do not cause this test result. 
    ...
numa_on=8  ## the value should be 0 or 1
...
## TCP segment offload, default: disabled.
tso=8   ## the value should be 0 or 1
  3. We don't have IXIA, There may be other reasons, but i use wrk as clent, the BW is 270MB * 8 = 2160 Mbps, when the file size is 1KB(1024bytes). 
    curl 192.168.1.2/1k.file -o 1k.file
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
100  1024  100  1024    0     0   730k      0 --:--:-- --:--:-- --:--:-- 1000k
root@server:~/wrk# md5sum 1k.file 
a059d0df7dd9a8ea071521a430003456  1k.file
root@server:~/wrk# ./wrk -c 128 -t 4 -d30s   http://192.168.1.2/1k.file
Running 30s test @ http://192.168.1.2/1k.file
4 threads and 128 connections
Thread Stats   Avg      Stdev     Max   +/- Stdev
Latency   548.96us  122.15us   2.61ms   73.76%
Req/Sec    55.87k     1.64k  103.61k    93.34%
6676360 requests in 30.10s, 7.95GB read
Requests/sec: 221810.74
Transfer/sec:    270.54MB
lining2020x commented 6 years ago

@jeffinjoyvvdn Hi, is there any progress about this issue?

jeffinjoyvvdn commented 6 years ago

HI Scott, The CPU utilization is the expected one. I lost my IXIA setup for further testing, Once I got I will update the test result. I use Linux curl command to download the file from nginx. Still, I am getting the low rate for the low size packet, compared to Linux.

Intel

CPU | FILE SIZE | BW

1 | 1 KB | 8.32 Mbps 10 KB | 77.6 Mbps 100 KB | 780.8 Mbps 1 MB | 2.192 Gbps 10 MB | 4.432 Gbps 100 MB | 7.128 gbps 1 GB | 8.048 Gbps

leopoldsga commented 5 years ago

Hi, could you please tell me how you test the CPS of Nginx using kernel stack? Do you turn on the net.ipv4.tcp_tw_reuse or not? Do you enable nginx with reuseport? It could be better if you could provide some scripts of CPS that result in the CPS numbers in . Thanks!

jfb8856606 commented 5 years ago
  1. Reuseport
  2. Run wrk for testing on multiple client machines simultaneously.
  3. nginx.conf should close log and configreturn 200 "response data".
leopoldsga commented 5 years ago

Thanks a lot!

Besides, if I want to turn off reuseport of nginx, what should be done other than "net.ipv4.tcp_tw_reuse = 0" in kernel? Could just accept_mutex on work?

As for simultaneous clients, I just use one client machine, do you have any suggestion to avoid "port reused" in nginx server?