Max HTTP connection count is not configurable, defaults to 250

[robdoesstuff]

As of latest Parity (1.10.3) I don't see a way to set the maximum number of open HTTP connections. Recently WS connection max was added, which is great, but it doesn't appear to apply to HTTP unless I misread the code somehow. Our back-end services are distributed and need more than 250 possible connections, even when pooling a few parity nodes together. I suspect others will also have this need.

Please make it possible to set the maximum number of HTTP connections to any limit so that the OS is the bottleneck, not parity itself.

Thanks!

[tomusdrw]

Can you elaborate on what issues you are running into currently (logs, error messages, etc)? HTTP max connections is not configurable, cause I'm pretty confident there is no hard limit in the server itself.

[robdoesstuff]

Here are some results when I run wrk to load test a parity RPC command. I have no issues until I get to around 240 connections, then wrk reports socket errors.

16 threads and 230 connections Thread Stats Avg Stdev Max +/- Stdev Latency 169.83ms 16.87ms 633.89ms 99.75% Req/Sec 77.78 44.76 141.00 52.21% 3136 requests in 3.08s, 0.97MB read Non-2xx or 3xx responses: 3136

16 threads and 240 connections Thread Stats Avg Stdev Max +/- Stdev Latency 168.66ms 12.02ms 604.47ms 99.88% Req/Sec 81.53 47.08 151.00 55.68% 3295 requests in 3.06s, 1.02MB read Socket errors: connect 5, read 0, write 0, timeout 0 Non-2xx or 3xx responses: 3295

16 threads and 300 connections Thread Stats Avg Stdev Max +/- Stdev Latency 169.51ms 13.48ms 625.54ms 99.88% Req/Sec 90.28 58.94 181.00 48.43% 3296 requests in 3.08s, 1.02MB read Socket errors: connect 53, read 0, write 0, timeout 0 Non-2xx or 3xx responses: 3296

8 threads and 500 connections Thread Stats Avg Stdev Max +/- Stdev Latency 170.06ms 19.17ms 645.15ms 99.82% Req/Sec 158.01 122.27 510.00 66.67% 3301 requests in 3.06s, 1.02MB read Socket errors: connect 253, read 0, write 0, timeout 0 Non-2xx or 3xx responses: 3301

So the magic number appears to be around 250. Once you try to open more than 250 HTTP connections, no more than 250 are opened and the rest appear to be socket errors. We haven't gone beyond this testing because we don't have confidence that this won't also be an issue with our app. In a large deployment, it's not uncommon to have hundreds or thousands of threads or workers spanning many servers. All we want to do is have all these workers be able to query for balance, which, when we benchmarked parity using WS on pure balance checks, we saw some very good numbers (40k/s), so surely just one or two nodes should be able to handle the load, but the bottleneck is somewhere in parity's HTTP configuration. We don't have the option to use websockets for this, so unless it's fixed in Parity, we'll end up writing a multiplexor that accepts HTTP connections and multiplexes them into websockets to parity, which I'd definitely like to avoid if possible as it seems unnecessary.

[robdoesstuff]

To test quickly, install wrk (https://github.com/wg/wrk), run this command ./wrk -t8 -c500 -d3s -s ./jsonrpc.lua <host>:8545 with the following as jsonrpc.lua (this is just an ERC20 balance read eth_call)

wrk.method = "POST"
wrk.body   = '{"jsonrpc": "2.0", "method": "eth_call", "params": [{ "to": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2", "data": "0x70a0823100000000000000000000000014fbca95be7e99c15cc2996c6c9d841e54b79425" }, "latest"], "id": 1}'
wrk.headers["Content-Type"] = "application/json"

[tomusdrw]

tomusdrw@t ~ $ wrk -t8 -c500 -d3s -s ./jsonrpc.lua http://localhost:8545
Running 3s test @ http://localhost:8545
  8 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    15.96ms    4.63ms 226.81ms   91.03%
    Req/Sec     3.59k     1.58k   24.82k    94.61%
  86121 requests in 3.10s, 9.12MB read
Requests/sec:  27783.87
Transfer/sec:      2.94MB
tomusdrw@t ~ $ wrk -t8 -c1000 -d3s -s ./jsonrpc.lua http://localhost:8545
Running 3s test @ http://localhost:8545
  8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    33.56ms   40.42ms 531.00ms   97.20%
    Req/Sec     3.37k     0.98k    6.80k    74.17%
  80559 requests in 3.01s, 8.53MB read
Requests/sec:  26747.21
Transfer/sec:      2.83MB
tomusdrw@t ~ $ wrk -t8 -c5000 -d3s -s ./jsonrpc.lua http://localhost:8545
Running 3s test @ http://localhost:8545
  8 threads and 5000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    36.11ms   11.43ms 348.86ms   98.24%
    Req/Sec     3.41k     1.54k    6.41k    58.08%
  77749 requests in 3.01s, 8.23MB read
  Socket errors: connect 3987, read 0, write 0, timeout 0
Requests/sec:  25831.69
Transfer/sec:      2.73MB

Definitely not a server property, since as you can see I have different results on my machine. If you inspect dmesg|tail after running wrk you might find something like this:

[364403.312078] TCP: request_sock_TCP: Possible SYN flooding on port 8545. Sending cookies.  Check SNMP counters.
``
^^^ That means reaching limit of the rate of open TCP connections, the limitation might also be caused by opening client-side connections.

[robdoesstuff]

tomusdrw, thank you for digging into it and attempting to reproduce. I did some googling and found that with wrk at least, my ulimit on OSX is what limits how many connections I can have open, so after making a few sysctl changes and a ulimit change, I was able to test Parity up to 16000 connections. While there probably is a reason to limit the max number of connections, it appears as though it's either unlimited or at least 16000, either of which work, so this is probably a non-issue for my case but I'll leave it open in case you want to add a configuration option to explicitly set the max number of http connections.

[dvdplm]

@robdoesstuff going to close this now. Feel free to open a new issue if you feel the problem needs further work. Thanks!