mesosphere / marathon-lb

Marathon-lb is a service discovery & load balancing tool for DC/OS
Apache License 2.0
449 stars 300 forks source link

Marathon-lb wrong configuration with Mesos Tcp #483

Closed maauso closed 5 years ago

maauso commented 7 years ago

Hello,

Versions Marathon-lb v1.5.0 Marathon 1.4.3 Mesos 1.30

we found a problem when we tried to migrate Marathon health check to Mesos health check in order to be ready for https://insight.io/github.com/mesosphere/marathon/blob/HEAD/docs/docs/health-checks.md?line=45

we give you a example because will be easier

Previous configuration

In marathon

    {
      "protocol": "TCP",
      "portIndex": 0,
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3
    }
  ],
  "labels": {
    "HAPROXY_0_MODE": "http",
    "HAPROXY_0_BALANCE": "roundrobin",
    "HAPROXY_0_GROUP": "group1",
    "HAPROXY_0_VHOST": "service-test.dom.dom"
  },

Configuration in marathon-lb

backend services_test_10222
  balance roundrobin
  mode http
  option forwardfor
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  server  {something} check inter 60s fall 4

we can see that we have a Layer7 service and HaProxy is doing tcp checks.

Current configuration

    {
      "protocol": "MESOS_TCP",
      "portIndex": 0,
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3
    }
  ],
  "labels": {
    "HAPROXY_0_MODE": "http",
    "HAPROXY_0_BALANCE": "roundrobin",
    "HAPROXY_0_GROUP": "group1",
    "HAPROXY_0_VHOST": "service-test.dom.dom"
  },

Configuration in marathon-lb

backend services_test_10222
  balance roundrobin
  mode http
  option forwardfor
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  option  httpchk GET /
  timeout check 20s
   server  {something} check inter 60s fall 4

we can see that now HaProxy has httpchk directly to "/", but we didn't configure this option in any place.

The question is,

The idea is to have the same that we had with marathon Heathcheck

drewkerrigan commented 7 years ago

@maauso the logic causing the problem is here: https://github.com/mesosphere/marathon-lb/blob/master/marathon_lb.py#L518-L524

Psuedo-code:

if healthcheck's protocol is "TCP": use haproxy tcp healthcheck else if HAPROXY_0_MODE is "http": use haproxy http healthcheck

Basically the logic in the first part of the if statement should be changed to also check for MESOS_TCP. Because it doesn't, it fell through to see that your app does in fact specify it's mode as "http".

Is there a reason your HAPROXY_0_MODE is not tcp? I assume there's a good reason but just wanted to verify that. I'll also make a change for the next release to also check for MESOS_TCP.

drewkerrigan commented 7 years ago

PR: https://github.com/mesosphere/marathon-lb/pull/509

maauso commented 7 years ago

@drewkerrigan

We found some applications that these don't have a URL to do a light health check, but they are working in Layer 7, then we need to use a TCP health check.

Thanks for the PR.