Open HsuJv opened 4 years ago
Simple solved by change the call odp_cpu_count()
to 1
in the src of nginx_ofp, close it.
--- a/src/event/modules/ngx_ofp_module.c
+++ b/src/event/modules/ngx_ofp_module.c
@@ -178,7 +178,7 @@ int my_webserver(int if_count, char **if_name)
exit(EXIT_FAILURE);
}
- core_count = odp_cpu_count();
+ core_count = 1;
num_workers = core_count;
if (params.core_count)
@@ -211,7 +211,7 @@ int my_webserver(int if_count, char **if_name)
if (num_queues_str) pktin_param.num_queues = atoi(num_queues_str);
odp_pktout_queue_param_init(&pktout_param);
- pktout_param.num_queues = odp_cpu_count();
+ pktout_param.num_queues = 1;
pktout_param.op_mode = ODP_PKTIO_OP_MT_UNSAFE;
for (i = 0; i < params.if_count; i++) {
Reopen this. It seems that ths nginx with ofp has a very terrible performance.
I tried wrk from other host in the same subnet and got
--------------------------nginx_ofp--------------------------
[xxx@yyy ~]$ wrk -c4000 -d120s -t12 --latency "http://myserver//"
Running 2m test @ http://myserver//
12 threads and 4000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 29.50ms 20.13ms 301.76ms 62.44%
Req/Sec 1.51k 759.18 4.08k 64.01%
Latency Distribution
50% 19.28ms
75% 53.42ms
90% 58.77ms
99% 68.88ms
1439503 requests in 2.00m, 229.34MB read
Socket errors: connect 2987, read 0, write 0, timeout 0
Requests/sec: 11988.07
Transfer/sec: 1.91MB
--------------------------original nginx--------------------------
[xxx@yyy ~]$ wrk -c4000 -d120s -t12 --latency "http://myserver//"
Running 2m test @ http://myserver//
12 threads and 4000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 41.19ms 104.87ms 2.00s 96.46%
Req/Sec 2.39k 1.25k 9.29k 68.53%
Latency Distribution
50% 24.46ms
75% 29.89ms
90% 38.71ms
99% 494.92ms
2559265 requests in 2.00m, 409.92MB read
Socket errors: connect 2987, read 0, write 0, timeout 945
Requests/sec: 21313.51
Transfer/sec: 3.41MB
Then I tried wrk from the host which runs nginx itself and got
--------------------------nginx_ofp--------------------------
[root@localhost wrk-master]# ./wrk -c1000 -d120s -t12 --latency "http://myserver//"
Running 2m test @ http://myserver//
12 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 55.50ms 10.34ms 268.90ms 87.49%
Req/Sec 1.34k 214.35 2.35k 69.42%
Latency Distribution
50% 55.35ms
75% 59.35ms
90% 63.86ms
99% 81.24ms
1926373 requests in 2.00m, 306.85MB read
Socket errors: connect 0, read 1, write 0, timeout 0
Requests/sec: 16040.38
Transfer/sec: 2.56MB
--------------------------original nginx--------------------------
[root@localhost wrk-master]# ./wrk -c1000 -d120s -t12 --latency "http://myserver//"
Running 2m test @ http://myserver//
12 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 30.68ms 106.45ms 2.00s 91.86%
Req/Sec 3.27k 1.58k 21.29k 77.98%
Latency Distribution
50% 2.73ms
75% 3.01ms
90% 82.72ms
99% 380.81ms
4664706 requests in 2.00m, 747.14MB read
Socket errors: connect 0, read 826, write 0, timeout 376
Requests/sec: 38840.71
Transfer/sec: 6.22MB
Here's my nginx.conf, both of the original and the ofp one use the same configure
user root;
worker_processes 1;
error_log logs/error.log info;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile off;
keepalive_timeout 65;
server {
listen 80;
server_name localhost;
access_log /dev/null;
location / {
return 200 "Testing";
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
Thanks in advance for your kindly advise
Hi HsuJv,
Maybe I can help you with the 'netwrap' thing...
Take a look at https://github.com/OpenFastPath/ofp/blob/master/example/README (the netwrap section). Now, the "Restrictions" section is outdated. You will need to modify the configuration command:
Next, take a look at script itself (https://github.com/OpenFastPath/ofp/blob/master/scripts/ofp_netwrap.sh): it uses by default 'eth1' and 'ofp_netwrap.cli' configuration file (192.168.100.1). You may need to adapte it to your needs (your interface, etc)
Hi bogdanPricope,
Thanks for your help. I've exactly tried the things you mentioned before, following the guidelines of #225 but still the issue.
But the netwrap thing is not the thing I'm concerned now, I finally made it work with nginx_ofp successfully as said in my comment 2.
And now I'm concerned about the performance issue as said in my comment 3. That's why I reopen this issue after closing it once.
Regards.
Hi, bogdanPricope ,
I also care about the performance test between nginx and ofp-nginx unber same hardware config.
Do we have official performance test about ofp-nginx?
Thanks
Sorry .. I cannot help you with ofp-nginx.
Never mind, I'll leave this issue open to wait someone help.
Anyway thanks for your kindly help.
Hi there,
I'm trying to run nginx with ofp to have a basic performance test
wrk -c4000 -d120s -t12 --latency "http://myserver//"
my playing is:
I find that there is a feature netwrap to do this with the original Nginx #225 , but failed to run it.
I also tried to set the Nginx to
daemon off; master_process off;
, failed either with the same stack.Then I'm going to another project nginx_ofp. Again failed to run it.
I set the nginx.conf to run only on a single core
worker_processes 1;
, but it seems the Nginx setup with CPU mask 0xc. That will not work since I'm having a network probe with E1000. Then I turned to Vmxnet3 but got thisAlso failed on the multi-queue.
Kindly advise how to run it, please.
P.S. I actually not matter how to run nginx with ofp on multi-cores. The thing I want is to make the configure
worker_processes 1;
works since my benchmarks are all on a single core.P.S.2 Also, I succeeded to run the webserver with the help of #241 but it was too slow to compare with original nginx. So I have to figure out how to run nginx with ofp.