High load - SSL Handshake errors + timeouts

[gbhrdt]

Hi! I've been searching all over the internet for the best HAProxy performance tweaks, modified the sysctl etc but we are still getting all these SSL handshake timeouts/failures and I rather think it's a configuration issue. This is my last place of hope someone could help me and I would be grateful for ANY hint on what might cause this. Our traffic is increasing and can be around 50-100k reqs per second at peaks.

What bothers me is that the logs are continuing even if the traffic is not at a "peak". There are about 5-10 errors per second on each node.

Current setup:

• Docker swarm, docker 18.09
• 1 manager, 5 workers, 8 vCPUs each
• each node runs a DFP instance with 8 threads

Logs:

2019/02/23 12:28:58 HAPRoxy: 10.255.0.224:50250 [23/Feb/2019:12:28:48.830] services~ services/<NOSRV> -1/-1/-1/-1/10019 408 1995 - - cR-- 1105/1105/0/0/0 0/0 "<BADREQ>"
2019/02/23 12:28:59 HAPRoxy: 10.255.0.3:53975 [23/Feb/2019:12:28:58.677] services/2: SSL handshake failure
2019/02/23 12:28:59 HAPRoxy: 10.255.0.224:50336 [23/Feb/2019:12:28:53.216] services/2: Connection closed during SSL handshake
2019/02/23 12:28:59 HAPRoxy: 10.255.0.2:62480 [23/Feb/2019:12:28:49.339] services~ services/<NOSRV> -1/-1/-1/-1/10038 408 1995 - - cR-- 1110/1110/0/0/0 0/0 "<BADREQ>"
2019/02/23 12:28:59 HAPRoxy: 10.255.0.2:47399 [23/Feb/2019:12:28:30.565] services/2: Connection closed during SSL handshake
2019/02/23 12:28:59 HAPRoxy: 10.255.0.3:54300 [23/Feb/2019:12:28:29.618] services/2: Timeout during SSL handshake
2019/02/23 12:28:59 HAPRoxy: 10.255.0.224:53561 [23/Feb/2019:12:28:29.711] services/2: Timeout during SSL handshake
2019/02/23 12:28:59 HAPRoxy: 10.255.0.224:63195 [23/Feb/2019:12:28:49.318] services/2: Connection closed during SSL handshake
2019/02/23 12:29:00 HAPRoxy: 10.255.0.224:62714 [23/Feb/2019:12:29:00.151] services/2: SSL handshake failure
2019/02/23 12:29:00 HAPRoxy: 10.255.0.224:37902 [23/Feb/2019:12:28:31.788] services/2: Connection closed during SSL handshake
2019/02/23 12:29:00 HAPRoxy: 10.255.0.224:53258 [23/Feb/2019:12:28:50.150] services~ services/<NOSRV> -1/-1/-1/-1/10036 408 1995 - - cR-- 1123/1123/0/0/0 0/0 "<BADREQ>"
2019/02/23 12:29:00 HAPRoxy: 10.255.0.224:39304 [23/Feb/2019:12:28:30.261] services/2: Timeout during SSL handshake
2019/02/23 12:29:00 HAPRoxy: 10.255.0.2:58524 [23/Feb/2019:12:28:30.499] services/2: Timeout during SSL handshake
2019/02/23 12:29:00 HAPRoxy: 10.255.0.224:7350 [23/Feb/2019:12:28:59.926] services/2: SSL handshake failure
2019/02/23 12:29:00 HAPRoxy: 10.255.0.2:13298 [23/Feb/2019:12:28:41.158] services/2: Connection closed during SSL handshake

Env:

    - RELOAD_INTERVAL=30000
    - DO_NOT_RESOLVE_ADDR=true
    - SKIP_ADDRESS_VALIDATION=true
    - LISTENER_ADDRESS=swarm-listener
    - MODE=swarm
    - TIMEOUT_CLIENT=60
    - TIMEOUT_SERVER=120
    - TIMEOUT_CONNECT=15
    - TIMEOUT_QUEUE=60
    - TIMEOUT_HTTP_REQUEST=15
    - TIMEOUT_HTTP_KEEP_ALIVE=15
    - EXTRA_GLOBAL=maxconn 50000,tune.ssl.cachesize 2000000,nbthread 8,cpu-map auto:1/1-8 0-7,stats bind-process 8
    - EXTRA_FRONTEND=maxconn 200000
    - STATS_URI=/metrics
    - DEBUG=true
    - DEBUG_ERRORS_ONLY=true
    - CONNECTION_MODE=http-server-close

Apache bench:

$ ab -c 250 -n 2000 https://api.example.com/
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking api.example.com (be patient)
Completed 200 requests
Completed 400 requests
SSL read failed (5) - closing connection
Completed 600 requests
Completed 800 requests
Completed 1000 requests
SSL handshake failed (5).
Completed 1200 requests
SSL handshake failed (5).
SSL read failed (5) - closing connection
Completed 1400 requests
SSL handshake failed (5).
Completed 1600 requests
SSL handshake failed (5).
SSL handshake failed (5).
Completed 1800 requests
^C

Server Software:
Server Hostname:        api.example.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
TLS Server Name:        api.example.com

Document Path:          /
Document Length:        34807 bytes

Concurrency Level:      250
Time taken for tests:   38.126 seconds
Complete requests:      1935
Failed requests:        9
   (Connect: 2, Receive: 0, Length: 7, Exceptions: 0)
Total transferred:      67543848 bytes
HTML transferred:       67157848 bytes
Requests per second:    50.75 [#/sec] (mean)
Time per request:       4925.820 [ms] (mean)
Time per request:       19.703 [ms] (mean, across all concurrent requests)
Transfer rate:          1730.08 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0 3332 3820.3   2203   36882
Processing:    48  496 1299.3    281   28691
Waiting:        0  126 184.6     79    5380
Total:        173 3828 3975.2   2859   37434

Percentage of the requests served within a certain time (ms)
  50%   2858
  66%   3648
  75%   4812
  80%   5207
  90%   8057
  95%   9334
  98%  18661
  99%  23430
 100%  37434 (longest request)

[vfarcic]

This project needs adoption. I moved to Kubernetes and cannot dedicate time to this project anymore. Similarly, involvement from other contributors dropped as well. Please consider contributing yourself if you think this project is useful.

[lle0x]

Hi @gbhrdt

If this issue is still relevant, please feel free to open an issue in the fork I created. https://github.com/freedomwatchers/docker-flow-proxy/issues/new

[lle0x]

Dear @gbhrdt

If this issue is still relevant, please feel free to leave a comment here.

[lle0x]

Closed due to inactivity