Slow requests when proxied

[leearmstrong]

I am suffering from anything proxied via node being incredibly slow compared to direct and I cannot work out why! The code is incredibly simple... but as you will see from these apache bench tests they are MUCH MUCH slower in real life.

The idea of the ab tests is to show that the upstream servers in the round robin don't have any issues at all with the tests running from the server that the node code is running on.

This seems to happen for POST and GET requests.

The runtime of ab for the same request goes from 5 seconds to 105 seconds!

"use strict";

const httpProxy = require('http-proxy');
const http = require('http');
const https = require('https');

var addresses = [
    {
        host: '10.0.1.81',
        port: 80
    },
    {
        host: '10.0.1.82',
        port: 80
    },
    {
        host: '10.0.1.83',
        port: 80
    }
];

var options = {
    xfwd: true
};

var proxy = httpProxy.createServer(options);

http.createServer(function (req, res) {
    //
    // On each request, get the first location from the list...
    //
    var target = {target: addresses.shift()};

    //
    // ...then proxy to the server whose 'turn' it is...
    //
    //console.log('balancing request to: ', target);
    proxy.web(req, res, target);

    proxy.on('error', function (e) {
        console.log("Error:", e)
    });

    //
    // ...and then the server you just used becomes the last item in the list.
    //
    addresses.push(target.target);
}).listen(80);

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.0.1.81 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:        nginx
Server Hostname:        10.0.1.81
Server Port:            80

Document Path:          /
Document Length:        178 bytes

Concurrency Level:      100
Time taken for tests:   4.013 seconds
Complete requests:      100000
Failed requests:        0
Non-2xx responses:      100000
Total transferred:      36400000 bytes
Total body sent:        17800000
HTML transferred:       17800000 bytes
Requests per second:    24916.19 [#/sec] (mean)
Time per request:       4.013 [ms] (mean)
Time per request:       0.040 [ms] (mean, across all concurrent requests)
Transfer rate:          8856.93 [Kbytes/sec] received
                        4331.14 kb/s sent
                        13188.06 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.6      1       4
Processing:     0    3   1.8      3      48
Waiting:        0    3   1.8      2      48
Total:          0    4   1.7      4      48

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      5
  80%      5
  90%      6
  95%      6
  98%      8
  99%     10
 100%     48 (longest request)

 ----------------------------------------------------------------------------------------

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.0.1.82 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:        nginx
Server Hostname:        10.0.1.82
Server Port:            80

Document Path:          /
Document Length:        178 bytes

Concurrency Level:      100
Time taken for tests:   5.008 seconds
Complete requests:      100000
Failed requests:        0
Non-2xx responses:      100000
Total transferred:      36400000 bytes
Total body sent:        17800000
HTML transferred:       17800000 bytes
Requests per second:    19968.71 [#/sec] (mean)
Time per request:       5.008 [ms] (mean)
Time per request:       0.050 [ms] (mean, across all concurrent requests)
Transfer rate:          7098.25 [Kbytes/sec] received
                        3471.12 kb/s sent
                        10569.38 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.5      1       5
Processing:     0    4   3.5      3     205
Waiting:        0    4   3.5      3      59
Total:          1    5   3.4      4     206

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      5
  75%      6
  80%      7
  90%      8
  95%     11
  98%     14
  99%     17
 100%    206 (longest request)

----------------------------------------------------------------------------------------

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.0.1.83 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:        nginx
Server Hostname:        10.0.1.83
Server Port:            80

Document Path:          /
Document Length:        178 bytes

Concurrency Level:      100
Time taken for tests:   4.813 seconds
Complete requests:      100000
Failed requests:        0
Non-2xx responses:      100000
Total transferred:      36400000 bytes
Total body sent:        17800000
HTML transferred:       17800000 bytes
Requests per second:    20774.97 [#/sec] (mean)
Time per request:       4.813 [ms] (mean)
Time per request:       0.048 [ms] (mean, across all concurrent requests)
Transfer rate:          7384.85 [Kbytes/sec] received
                        3611.27 kb/s sent
                        10996.13 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.6      1      10
Processing:     0    4   2.8      3      34
Waiting:        0    4   2.8      3      34
Total:          0    5   2.8      4      35

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      5
  75%      5
  80%      6
  90%      8
  95%     10
  98%     13
  99%     16
 100%     35 (longest request)

----------------------------------------------------------------------------------------

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:        
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        178 bytes

Concurrency Level:      100
Time taken for tests:   103.119 seconds
Complete requests:      100000
Failed requests:        0
Non-2xx responses:      100000
Total transferred:      36400000 bytes
Total body sent:        17800000
HTML transferred:       17800000 bytes
Requests per second:    969.75 [#/sec] (mean)
Time per request:       103.119 [ms] (mean)
Time per request:       1.031 [ms] (mean, across all concurrent requests)
Transfer rate:          344.72 [Kbytes/sec] received
                        168.57 kb/s sent
                        513.29 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       8
Processing:    46  103  16.3    100     233
Waiting:       46  103  16.2    100     233
Total:         46  103  16.3    101     233

Percentage of the requests served within a certain time (ms)
  50%    101
  66%    109
  75%    113
  80%    115
  90%    123
  95%    134
  98%    146
  99%    153
 100%    233 (longest request)

[leearmstrong]

Even a simpler script exhibits the same issue...

var http = require('http'),
httpProxy = require('http-proxy');
//
// Create your proxy server and set the target in the options.
//
httpProxy.createProxyServer({target:'http://10.0.1.81'}).listen(8000); // See (†)

[jcrugzz]

@leearmstrong try using something like wrk rather than apache bench. apache bench doesnt support http 1.1. See if that makes a difference in your benchmarks. I would also suggest using an Agent for this case so there is better socket reuse.

[jcrugzz]

Node version would also be interesting to know

[leearmstrong]

Never used wrk before so will give it a go. All of the client connecting into the nginx server direct (before the proxy I want to implement) are only HTTP 1.0 capable anyway so ab may be a better test.

This is Node 6.10.0

[leearmstrong]

So I added a keep alive agent to it with a timeout of 30000msec and it made a bit of a difference, the test now completes in 75 seconds and not 103 seconds. This is no where near the direct connections from ab to the upstream of closer to 5 seconds.

I can accept there will be some overhead for proxying via node but this seems excessive? Or is this to be expected?

[leearmstrong]

Stats using wrk and using a keepAlive config

wrk -c 100 http://localhost/
Running 10s test @ http://localhost/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    67.99ms   22.95ms 250.93ms   88.57%
    Req/Sec   755.77    179.23     0.97k    75.13%
  14873 requests in 10.01s, 5.24MB read
Requests/sec:   1485.59
Transfer/sec:    536.34KB

wrk -c 100 http://10.0.1.81/
Running 10s test @ http://10.0.1.81/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    52.85ms  146.70ms   1.02s    90.08%
    Req/Sec    26.95k     9.01k   57.15k    73.85%
  524184 requests in 10.01s, 184.46MB read
Requests/sec:  52376.22
Transfer/sec:     18.43MB

[jcrugzz]

I'm honestly not sure if we are comparing apples to apples if your backend is an nginx that can spawn workers. I would run your node process and do some profiling to get some more information. https://nodejs.org/en/docs/guides/simple-profiling/

[guoliangli123]

I meet the same problem,I do not know what happened.