Zookeeper config not being updated with automatically created ACLs

[dwstevens]

We are using the following template to automatically create ACLs for our mesos cluster resources:

frontend http-in
        bind *:80
        {{ $services := .Services }}
        {{ range $index, $app := .Apps }} {{ if hasKey $services $app.Id }} {{ $service := getService $services $app.Id }}
        acl {{ $app.EscapedId }}-aclrule {{ $service.Acl}}
        use_backend {{ $app.EscapedId }}-cluster if {{ $app.EscapedId }}-aclrule
        {{ else }}

        # This is the default proxy criteria
        acl {{ $app.EscapedId }}-aclrule hdr(host) -i {{ $app.Env.SUBDOMAIN }}.domain.com
        use_backend {{ $app.EscapedId }}-cluster if {{ $app.EscapedId }}-aclrule
        {{ end }} {{ end }}

When we add a new Marathon task with the SUBDOMAIN ENV, the /etc/haproxy/haproxy.cfg is successfully updated.

        # This is the default proxy criteria
        acl ::mcp-aclrule hdr(host) -i mcp.domain.com
        use_backend ::mcp-cluster if ::mcp-aclrule

But /api/state does not have a defined service for MCP

{
  "Apps": [{
    "Id": "/mcp",
    "EscapedId": "::mcp",
    "HealthCheckPath": "/status",
    "HealthChecks": [{
      "Protocol": "HTTP",
      "Path": "/status",
      "PortIndex": 0
    }],
    "Tasks": [{
      "Host": "XXX.compute-1.amazonaws.com",
      "Port": 31000,
      "Ports": [31000]
    }],
    "ServicePort": 10004,
    "ServicePorts": [10004],
    "Env": {
      "MONGO": "XXX.compute-1.amazonaws.com:31002",
      "NODE_ENV": "staging",
      "SUBDOMAIN": "mcp"
    },
    "Labels": {}
  }, {
    "Id": "/mongo",
    "EscapedId": "::mongo",
    "HealthCheckPath": "",
    "HealthChecks": [{
      "Protocol": "TCP",
      "Path": "/",
      "PortIndex": 0
    }],
    "Tasks": [{
      "Host": "XXX.compute-1.amazonaws.com",
      "Port": 31002,
      "Ports": [31002]
    }],
    "ServicePort": 10001,
    "ServicePorts": [10001],
    "Env": {
      "SUBDOMAIN": "mongo"
    },
    "Labels": {}
  }, {
    "Id": "/npm-registry",
    "EscapedId": "::npm-registry",
    "HealthCheckPath": "/",
    "HealthChecks": [{
      "Protocol": "HTTP",
      "Path": "/",
      "PortIndex": 0
    }],
    "Tasks": [{
      "Host": "XXX.compute-1.amazonaws.com",
      "Port": 31000,
      "Ports": [31000]
    }],
    "ServicePort": 10000,
    "ServicePorts": [10000],
    "Env": {},
    "Labels": {}
  }, {
    "Id": "/sark",
    "EscapedId": "::sark",
    "HealthCheckPath": "/status",
    "HealthChecks": [{
      "Protocol": "HTTP",
      "Path": "/status",
      "PortIndex": 0
    }],
    "Tasks": [{
      "Host": "XXX.compute-1.amazonaws.com",
      "Port": 31001,
      "Ports": [31001]
    }],
    "ServicePort": 10002,
    "ServicePorts": [10002],
    "Env": {
      "MONGO": "XXX.compute-1.amazonaws.com:31002",
      "NODE_ENV": "staging",
      "SUBDOMAIN": "sark"
    },
    "Labels": {}
  }, {
    "Id": "/tron",
    "EscapedId": "::tron",
    "HealthCheckPath": "/",
    "HealthChecks": [{
      "Protocol": "HTTP",
      "Path": "/",
      "PortIndex": 0
    }],
    "Tasks": [{
      "Host": "XXX.compute-1.amazonaws.com",
      "Port": 31000,
      "Ports": [31000]
    }],
    "ServicePort": 10003,
    "ServicePorts": [10003],
    "Env": {
      "NODE_ENV": "staging",
      "SUBDOMAIN": "app"
    },
    "Labels": {}
  }],
  "Services": {
    "/mongo": {
      "Id": "/mongo",
      "Acl": "hdr(host) -i mongo.domain.com"
    },
    "/npm-registry": {
      "Id": "/npm-registry",
      "Acl": "hdr(host) -i npm.domain.com"
    },
    "/sark": {
      "Id": "/sark",
      "Acl": "hdr(host) -i sark.quantumbio.com"
    },
    "/tron": {
      "Id": "/tron",
      "Acl": "hdr(host) -i app.domain.com"
    }
  }
}

And if we look in Zookeeper, we see that MCP is not stored there.

ubuntu@ip-10-0-31-124:/usr/lib/zookeeper/bin$ ./cli_mt internal-zookeeper-XXXXus-east-1.elb.amazonaws.com:80 
Watcher SESSION_EVENT state = CONNECTED_STATE
Got a new session id: 0xdf4e154e0f670007
ls /bamboo
time = 2 msec
/bamboo: rc = 0
    %2Fsark
    %2Fnpm-registry
    %2Ftron
    %2Fmongo
time = 2 msec

Do we need to do something special to get Bamboo to add an automatically created ACL to zookeeper?

MCP is listed but with no ACL, even though haproxy.cfg contains correct ACL info.

[j1n6]

Were you able to access mcp service via haproxy? The user interface only shows information for custom configured apps (when it's green). When the configuration is set in template without any custom ACL rules, it shows white background.

Services section in /api/state only list custom configuration for each Marathon app. The custom configuration overrides the default specified in the template.

[xiaods]

not bug?

[j1n6]

not quite seen any behaviour in our production yet. @dwstevens any updates?