mesosphere / marathon-lb

Marathon-lb is a service discovery & load balancing tool for DC/OS
Apache License 2.0
449 stars 300 forks source link

HAPROXY_0_BACKEND_HEAD newline character cause problem #542

Closed agate closed 5 years ago

agate commented 6 years ago

I try to follow this way to increase my backend service's timeout: https://github.com/mesosphere/marathon-lb/wiki#setting-timeout-for-long-lived-socket-connections

I found haproxy config be generated properly from: :9090/_haproxy_getconfig which is good. But I found my marathon application failed to start up. It's because in the label HAPROXY_0_BACKEND_HEAD contains \n which cause mesos not able to launch docker application(mesos try to set this lable as a docker's env var). The error shows in mesos: docker: poorly formatted environment: variable 'balance {balance}' has white spaces.

So I try to avoid use \n in label HAPROXY_0_BACKEND_HEAD. But that cause haproxy config syntax error.

Any help?

justinrlee commented 6 years ago

Hi, Where are you configuring this? The labels from that example should be configured as labels on the target application, not on Marathon-LB. Also, they should be configured as labels, not as environment variables.

Can you show me the marathon json definition you're trying to use?

Thanks, Justin Lee

joecox commented 6 years ago

I am having the same issue. I set the HAPROXY_0_BACKEND_HEAD in my app to:

{
  "labels": {
    "HAPROXY_0_BACKEND_HEAD": "backend {backend}\n  balance {balance}\n  mode {mode}\n  timeout server 30m\n  timeout client 30m\n"
  }
}

Here is my full app JSON:

{
  "id": "/http-server",
  "cmd": "python -m SimpleHTTPServer",
  "cpus": 0.1,
  "mem": 128,
  "disk": 0,
  "instances": 0,
  "acceptedResourceRoles": [
    "*"
  ],
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": "python:2.7",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 8000,
          "hostPort": 0,
          "servicePort": 0,
          "protocol": "tcp",
          "labels": {}
        }
      ],
      "privileged": false,
      "parameters": [],
      "forcePullImage": false
    }
  },
  "labels": {
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_BACKEND_HEAD": "backend {backend}\n  balance {balance}\n  mode {mode}\n  timeout server 30m\n  timeout client 30m\n"
  }
}

My app fails to start. In my app's stderr, I see:

docker: poorly formatted environment: variable 'balance {balance}' has white spaces.

To the best of my understanding, Mesos tries to inject the environment variable MARATHON_APP_LABEL_HAPROXY_0_BACKEND_HEAD. My guess is this is where it fails. But I would appreciate some insight about whether this is true or not.

Mesos version: 1.2.0 Marathon version: 1.4.11 Docker version: 17.03.1-ce build c6d412e

agate commented 6 years ago

OK. I confirmed that this is because of the mesos version. I upgraded my mesos slave version to 1.3.2 and it solved the problem.