Socket Hang up when requests are reverse-proxied with HA Proxy

[perrefe]

Using SSL HAProxy frontend between newman and application server causes intermitently socket hang up errors.

➜ newman -v
3.4.3

➜ uname -a 
Linux jenkins 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

➜ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Newman command:

➜ newman run app.postman_collection --reporters cli --globals globals.postman_globals --environment kube.postman_environment

app.postman_collection

{
    "variables": [],
    "info": {
        "name": "APP",
        "_postman_id": "ba523361-986d-8262-c06e-31b8f6338f9b",
        "description": "",
        "schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json"
    },
    "item": [
        {
            "name": "Login",
            "event": [
                {
                    "listen": "test",
                    "script": {
                        "type": "text/javascript",
                        "exec": [
                            "var jsonData = JSON.parse(responseBody);",
                            "",
                            "/* Tests */",
                            "tests[\"HTTP Status code is 200\"] = responseCode.code === 200;",
                            "tests[\"x-auth-token is present\"] = postman.getResponseHeader(\"x-auth-token\");",
                            "tests[\"Response Data code\"] = jsonData.code === \"0\";",

                            "",
                            "/* Environment set */",
                            "postman.setEnvironmentVariable(\"x-auth-token\", jsonData.code);"
                        ]
                    }
                }
            ],
            "request": {
                "url": "http://{{appHost}}:{{appPort}}/{{appCtx}}/api/login",
                "method": "POST",
                "header": [],
                "body": {
                    "mode": "urlencoded",
                    "urlencoded": [
                        {
                            "key": "username",
                            "value": "{{user}}",
                            "type": "text",
                            "enabled": true
                        },
                        {
                            "key": "password",
                            "value": "{{password}}",
                            "type": "text",
                            "enabled": true
                        }
                    ]
                },
                "description": ""
            },
            "response": []
        },
        {
            "name": "Logout",
            "event": [
                {
                    "listen": "test",
                    "script": {
                        "type": "text/javascript",
                        "exec": [
                            "/* Tests */",
                            "tests[\"HTTP Status code is 200\"] = responseCode.code === 200;"
                        ]
                    }
                }
            ],
            "request": {
                "url": "http://{{appHost}}:{{appPort}}/{{appCtx}}/api/logout",
                "method": "GET",
                "header": [
                    {
                        "key": "x-auth-token",
                        "value": "{{x-auth-token}}",
                        "description": ""
                    }
                ],
                "body": {},
                "description": ""
            },
            "response": []
        }
    ]
}

globals globals.postman_globals

{
  "id": "224d6f23-f1b9-e483-f9cf-1863c0e5eff8",
  "name": "Postman Globals",
  "values": [
    {
      "key": "appPort",
      "value": "80",
      "type": "text",
      "enabled": true
    },
    {
      "key": "appCtx",
      "value": "app",
      "type": "text",
      "enabled": true
    },
    {
      "key": "user",
      "value": "user-01",
      "type": "text",
      "enabled": true
    },
    {
      "key": "password",
      "value": "password-01",
      "type": "text",
      "enabled": true
    }
  ],
  "_postman_variable_scope": "globals",
  "_postman_exported_at": "2017-02-06T19:14:07.935Z",
  "_postman_exported_using": "Postman/4.9.2"
}

Env.postman_environment

{
  "id": "2f8547b1-6e18-d079-59a7-eaa0a747a596",
  "name": "Env",
  "values": [
    {
      "key": "appHost",
      "value": "app.mycompany.cl",
      "type": "text",
      "enabled": true
    }
  ],
  "timestamp": 1486407576675,
  "_postman_variable_scope": "environment",
  "_postman_exported_at": "2017-02-06T19:06:24.748Z",
  "_postman_exported_using": "Postman/4.9.2"
}

HA Proxy config

global
   log /dev/log local0
   log /dev/log local1 notice
   chroot /var/lib/haproxy
   stats socket /run/haproxy/admin.sock mode 660 level admin
   stats timeout 30s
   user nonroot
   group nonroot
   tune.ssl.default-dh-param 2048
   daemon

defaults
   log global
   mode http
   option httplog
   option dontlognull
   timeout connect 5000
   timeout client 50000
   timeout server 50000

frontend http_front
   bind *:80
   stats uri /haproxy?stats
   default_backend http_back

frontend https_front
   bind *:443 ssl crt /etc/haproxy/certs/app.pem
   reqadd X-Forwarded-Proto:\ https
   acl letsencrypt-acl path_beg /.well-known/acme-challenge/
   use_backend letsencrypt-backend if letsencrypt-acl
   default_backend http_back

backend http_back
   server kube-01 10.0.0.98:32026 check
   server kube-02 10.0.0.74:32026 check

backend letsencrypt-backend
   server letsencrypt 127.0.0.1:54321

Scenarios:

• variables appHost=app.mycompany.cl and appPort=80, works fine
• variables appHost=10.0.0.98 and appPort=32026, works fine
• variables appHost=10.0.0.74 and appPort=32026, works fine
• variables appHost=app.mycompany.cl and appPort=443 (and obviously https in request.url), some requests fails with:

APP

→ Login
  POST https://app.mycompany.cl:443/app/api/login [200 OK, 838B, 4.6s]
  ✓  HTTP Status code is 200
  ✓  x-auth-token is present
  ✓  Response Data code

→ Logout
  GET https://app.mycompany.cl:443/app/api/logout [errored]
  ✓  HTTP Status code is 200

┌─────────────────────────┬──────────┬──────────┐
│                         │ executed │   failed │
├─────────────────────────┼──────────┼──────────┤
│              iterations │        1 │        0 │
├─────────────────────────┼──────────┼──────────┤
│                requests │        2 │        1 │
├─────────────────────────┼──────────┼──────────┤
│            test-scripts │        2 │        0 │
├─────────────────────────┼──────────┼──────────┤
│      prerequest-scripts │        0 │        0 │
├─────────────────────────┼──────────┼──────────┤
│              assertions │        4 │        0 │
├─────────────────────────┴──────────┴──────────┤
│ total run duration: 4.7s                      │
├───────────────────────────────────────────────┤
│ total data received: 78B (approx)             │
├───────────────────────────────────────────────┤
│ average response time: 4.6s                   │
└───────────────────────────────────────────────┘

  #  failure                                 detail

 1.  Error                                   socket hang up
                                             at request
                                             inside "Logout" of "APP"

This similar issue #818 reopened recently, but involves a forward-proxy. Also, this scenarios tested directly from Postman or Postman's Collection Runner works well.

Thank you guys!

[kunagpal]

Possibly related to #818

[shamasis]

Damn... we definitely missed updating these issues. Ok, here's something from some of my old notes from a different HAProxy test that I was doing:

- HAProxy install and configured fine
- setup local node server for test with 2 instances
- configured haproxy to to add the two node servers in cfg
- collection to call server and ran fine
- runs with 1000 iteration but takes ages after 500-ish iteration
- configured a bunch of global timeout_* and conn count cfg in haproxy
- 1000 iterations works fines

It seems that I did not face socket hangup, but I did configure a bunch of global settings while stressing the system. Sadly, I didn't take note of which configs. :-( Anyway, I also recall socket hangup, but from something else in HAProxy, caused when one of the upstream servers were shutting down.

I presume that it has been quite a while and if the issue is still not resolved, please feel free to reopen.