End to End SSL

[distributorofpain]

I have marathon-lb setup with a valid SSL certificate and its currently working with a wordpress site i spun up in mesosphere. So i know traffic is passing and things are healthy.

Everything is using vhosts, so i have another vhost entry for another application i am trying to run. This application needs to have SSL working like so:

Client ----> marathon-lb with valid cert -----> application with self signed cert

Essentially, end to end SSL.

When i spin up the application, i get a "502 bad gateway The server returned an invalid or incomplete response.", so i figure marathon-lb doesnt like the self signed certificate. I check the application directly: https://public-ip-of-agent-server:port and i get an invalid cert warning, but after i click through, the application is working.

So i then modify the labels for the application, but i think i am missing something to bypass the self signed SSL cert. ( i dont want to use lets encrypt for this as i will have too many applications running, but i would like to use SSL to ensure i have end to end SSL ).

Here is the application configuration, perhaps i am doing something wrong:

{
  "id": "/identityserver",
  "cmd": null,
  "cpus": 1,
  "mem": 1536,
  "disk": 0,
  "instances": 1,
  "constraints": [
    [
      "hostname",
      "UNIQUE"
    ]
  ],
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": "withheld/esb/identityserver:v6",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 443,
          "hostPort": 0,
          "servicePort": 10056,
          "protocol": "tcp",
          "labels": {}
        }
      ],
      "privileged": false,
      "parameters": [],
      "forcePullImage": false
    }
  },
  "env": {
    "server.session.timeout": "43200"
  },
  "healthChecks": [
    {
      "protocol": "COMMAND",
      "command": {
        "value": "bash -c \\\"</dev/tcp/$HOST/$PORT0\\\""
      },
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3,
      "ignoreHttp1xx": false
    }
  ],
  "labels": {
    "HAPROXY_0_USE_HSTS": "false",
    "HAPROXY_0_REDIRECT_TO_HTTPS": "true",
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_BACKEND_SERVER_TCP_HEALTHCHECK_OPTIONS": "server {serverName} {host_ipv4}:{port}{cookieOptions} ssl verify none{healthCheckOptions}{otherOptions}",
    "HAPROXY_0_VHOST": "identityserver.loadbalancer.withheld.com"
  },
  "portDefinitions": [
    {
      "port": 10056,
      "protocol": "tcp",
      "labels": {}
    }
  ],
  "uris": [
    "http://withheld/public/withheld.tar.gz"
  ],
  "fetch": [
    {
      "uri": "http://withheld/public/withheld.tar.gz",
      "extract": true,
      "executable": false,
      "cache": false
    }
  ]
}

Also, on a side note, or perhaps in addition to the labels. I was trying to figure out how to make the health check work as well, so the stats page shows a healthy application ( might be a two birds one stone thing here ).

[distributorofpain]

when i dig into the marathon-lb haproxy.cfg file, this is what i see for this application:

backend identityserver_10056
  balance roundrobin
  mode http
  option forwardfor
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  server 10_128_32_106_24272 10.128.32.106:24272

If i modify HAPROXY_0_BACKEND_HTTP_HEALTHCHECK_OPTIONS to be this:

server {serverName} {host_ipv4}:{port}{cookieOptions} ssl verify none {healthCheckOptions}{otherOptions}

nothing changes....

The log shows this:

[ALERT] 056/055442 (24577) : parsing [/tmp/tmpi1c7gcpk:102] : character '{' is not permitted in server name '{serverName}'.
[ALERT] 056/055442 (24577) : Error(s) found in configuration file : /tmp/tmpi1c7gcpk
[ALERT] 056/055442 (24577) : Fatal errors found in configuration.

This of course tells me that it doesnt accept the line with the variables in it... so how can you insert variables in it?

I thought the labels override the defaults, but that doesnt seem to be the case. If this must be done in a template file, can the template file contain just the template setting i want to modify (any good examples of the file?)

[distributorofpain]

Ok, so i banged on this some more and found this to work:

Label: HAPROXY_0_BACKEND_SERVER_OPTIONS

  server {serverName} {host_ipv4}:{port} {cookieOptions} check inter 5s ssl verify none \n

Strangely it gives me this error when loading but still loads(my healthcheck addition works as well):

2017-02-26 06:26:11,407 marathon_lb: checking config with command: ['haproxy', '-f', '/tmp/tmpmcn_dcho', '-c']
[ALERT] 056/062611 (30810) : parsing [/tmp/tmpmcn_dcho:105] : 'server 10_128_37_164_9484' unknown keyword '
'. Registered keywords :
    [ ALL] id <arg>
    [ TCP] tcp-ut <arg>
    [ SSL] ca-file <arg>
    [ SSL] check-ssl
    [ SSL] ciphers <arg>
    [ SSL] crl-file <arg>
    [ SSL] crt <arg>
    [ SSL] force-sslv3
    [ SSL] force-tlsv10
    [ SSL] force-tlsv11
    [ SSL] force-tlsv12
    [ SSL] no-ssl-reuse
    [ SSL] no-sslv3
    [ SSL] no-tlsv10
    [ SSL] no-tlsv11
    [ SSL] no-tlsv12
    [ SSL] no-tls-tickets
    [ SSL] send-proxy-v2-ssl
    [ SSL] send-proxy-v2-ssl-cn
    [ SSL] sni <arg>
    [ SSL] ssl
    [ SSL] verify <arg>
    [ SSL] verifyhost <arg>
[ALERT] 056/062611 (30810) : Error(s) found in configuration file : /tmp/tmpmcn_dcho
[ALERT] 056/062611 (30810) : Fatal errors found in configuration.

I was guessing that it is complaining about the \n and i removed it from the end of the line, but it still errors, this time with"

2017-02-26 06:40:51,122 marathon_lb: checking config with command: ['haproxy', '-f', '/tmp/tmp63jullmv', '-c']
[ALERT] 056/064051 (1510) : parsing [/tmp/tmp63jullmv:105] : 'server 10_128_31_230_11214' unknown keyword 'server'. Registered keywords :
    [ ALL] id <arg>
    [ TCP] tcp-ut <arg>
    [ SSL] ca-file <arg>
    [ SSL] check-ssl
    [ SSL] ciphers <arg>
    [ SSL] crl-file <arg>
    [ SSL] crt <arg>
    [ SSL] force-sslv3
    [ SSL] force-tlsv10
    [ SSL] force-tlsv11
    [ SSL] force-tlsv12
    [ SSL] no-ssl-reuse
    [ SSL] no-sslv3
    [ SSL] no-tlsv10
    [ SSL] no-tlsv11
    [ SSL] no-tlsv12
    [ SSL] no-tls-tickets
    [ SSL] send-proxy-v2-ssl
    [ SSL] send-proxy-v2-ssl-cn
    [ SSL] sni <arg>
    [ SSL] ssl
    [ SSL] verify <arg>
    [ SSL] verifyhost <arg>
[ALERT] 056/064051 (1510) : Error(s) found in configuration file : /tmp/tmp63jullmv
[ALERT] 056/064051 (1510) : Fatal errors found in configuration.

So it seems that when it reloads that it adds another entry for the new server location, but that causes an error during the reload. I am wondering if i need to add some sort of other "\n" type entry to get it to stop showing that error?

[brndnmtthws]

It probably makes more sense to just pass through haproxy, IMO.

Have a look at this: https://github.com/brndnmtthws/mlb-nginx-http2

It's intended as an example for HTTP/2, but it's the same principle.

[distributorofpain]

I forgot this was still open. I have since switched to only editing tags in json mode. Adding the /n in the right places and escaping slashes with two slashes works perfectly in json mode. All the errors were related to the configuration file not having the needed return. I thought i had been applying it, but i didnt realize until i really dug deep into the temp config files for haproxy, that it wasnt applying the labels properly when they were added in the web form with escape characters.

[imesh]

@distributorofpain Thanks for sharing above information, it worked for me on DC/OS 1.10:

"HAPROXY_X_BACKEND_SERVER_OPTIONS": "server {serverName} {host_ipv4}:{port} {cookieOptions} check inter 5s ssl verify none"

[distributorofpain]

awesome glad we are not the only ones needing this and that it works in 1.10(havnt upgraded yet due to the CPU restriction preparation )!

[abhishekraghav]

Hello, I too stuck on the same issue. Just a quick question, are you doing ssl-passthrough here or reestablishing the SSL connection from marathon lb to backend. Do you really need certs to be configured here as you are terminating the SSL on backend.

Do you have any other component siting in front of marathon Lb as well. e.g. ALB or NGINX. etc.

[distributorofpain]

We have a signed certificate in marathon-lb ( i think its godaddy, but its 100% internet usable ). Then for the applications, we use self signed certificates built into the application when its dockerized. In the marathon-lb configuration noted above, we use the switch for marathon-lb to ignore the self signed certificate(basically it will trust the self signed cert).

ssl verify none

[abhishekraghav]

@distributorofpain Thanks for the clarification. I have one small doubt. In below configuration "HAPROXY_X_BACKEND_SERVER_OPTIONS": "server {serverName} {host_ipv4}:{port} {cookieOptions} check inter 5s ssl verify none" What this 5s mean here. Since after applying this configuration, we are getting some intermittent 502 error reported on ELB (that sits before Marathon LB) while I am running performance tests. . I am seeing any trace of error on my service logs, but ELB. Could it be due to any specific option ..

Any help is much appreciated. Thanks.