mesos-dns ignore ipAddress{} when a healthchecks was setted

[ricardoareis]

Hello,

I've used marathon+mesos-dns+calico in a PoC with good results, however when the calico was configured as a docker network plugins and also, a healthchecks marathon was configured mesos-dns ignore ipAddress clause, changing any record with a hosts address.

with healthchecks

[root@ip-172-16-1-170 ~]# dig @172.16.1.170 test.marathon.mesos +short 172.16.252.219 172.16.252.91

without healthchecks

[root@ip-172-16-1-170 ~]# dig @172.16.1.170 test.marathon.mesos +short 10.100.1.147 10.100.1.108 10.100.1.154 10.100.1.90

Marathon app.json -> http://pastebin.com/S1gTfhGC

Docker Network + Calico -> http://pastebin.com/v6Y7m2yB

mesos-dns-conf -> http://pastebin.com/zxjhTfF6

[sargun]

I'm not sure we have any logic that looked at healthchecks. What do you mean exactly?

[ricardoareis]

Sargun,

When any kind of health check was passed, mesos-dns answer has only a host addresses, instead of the calico ipam pool (used as docker network plugin).

I've attached a sample file app.json

[jdef]

would you mind providing a copy of your mesos-dns config?

On Mon, Apr 11, 2016 at 8:10 PM, ricardoareis notifications@github.com wrote:

Sargun,

When any kind of health check was passed, mesos-dns response with a host address, instead of the calico ipam (used as docker network plugin).

I've attached a sample file app.json

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/mesosphere/mesos-dns/issues/417#issuecomment-208631808

[ricardoareis]

Hello jdef,

I've updated the issue with a mesos-dns config.

tks Ricardo

[frosenberg]

I see the same issue. I have mesos-dns deployed via marathon and a couple of apps that used DNS for discovery. Now that I turned on health checks on each app, the SRV record now returns the IP of the Mesos slave and not of the docker container that is running on the slave.

[sargun]

@frosenberg Can you publish your state.json with and without healthchecks?

[frosenberg]

@sargun https://gist.github.com/frosenberg/39d104a45a13015a555d665d650fa2b9

There are 4 services in my current deployment that I run locally (one mesos slave). Please not that when I enable the health checks, the rest-api service is not healthy, thus being killed an restarted, because its trying to check whether its dependent gRPC services are healthy (which they are individually) but the rest-api cannot connect b/c mesos-dns is giving the wrong IP (the one of the mesos slave not the docker container).

Without health checks everything works fine.

This is my mesos-dns config:

{
  "zk": "zk://172.17.0.1:2181/mesos",
  "refreshSeconds": 60,
  "ttl": 60,
  "domain": "mesos",
  "port": 53,
  "resolvers": ["8.8.8.8","8.8.4.4"],
  "timeout": 5,
  "httpon": false,
  "dnson": true,
  "httpport": 8123,
  "externalon": true,
  "listener": "172.17.0.1",
  "SOAMname": "ns1.mesos",
  "SOARname": "root.ns1.mesos",
  "SOARefresh": 60,
  "SOARetry":   600,
  "SOAExpire":  86400,
  "SOAMinttl": 60,
  "IPSources": ["netinfo", "host", "mesos"]
}

[rajholla]

I hit the same issue today . Mesos dns resolves to slave ip instead of calico ipam assigned ip for the container. Here is my mesos-dns config

{ "zk": "zk://127.0.0.1:2181/mesos", "refreshSeconds": 30, "ttl": 60, "domain": "mesos", "port": 53, "resolvers": ["144.202.8.11", "144.202.8.12", "144.202.54.10"], "timeout": 5, "listener": "0.0.0.0", "email": "root.mesos-dns.mesos", "IPSources": ["netinfo", "host"] }