DNS Resolution issues for Pods in the same namespace as Consul

[lawliet89]

When Consul Catalog Sync is configured to sync Consul services to Kubernetes, consul-k8s will create ExternalName services in Kubernetes for Consul services.

This causes DNS resolution issues when resolving x.service.consul DNS names from a pod in the same namespace as Consul.

Consider Consul running in some namespace called core and we are trying to access a service called prometheus-server with 2 replicas. CoreDNS is configured as per the documentation.

From a pod in the default namespace, when we query DNS dig:

# dig +search prometheus-server.service.consul

; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> +search prometheus-server.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5046
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;prometheus-server.service.consul. IN   A

;; ANSWER SECTION:
prometheus-server.service.consul. 5 IN  A   10.17.18.167
prometheus-server.service.consul. 5 IN  A   10.17.17.36

;; ADDITIONAL SECTION:
prometheus-server.service.consul. 5 IN  TXT "external-source=kubernetes"
prometheus-server.service.consul. 5 IN  TXT "external-source=kubernetes"

;; Query time: 2 msec
;; SERVER: 172.20.0.10#53(172.20.0.10)
;; WHEN: Thu Jul 18 05:04:17 UTC 2019
;; MSG SIZE  rcvd: 299

compared to a pod in the core namespace (same as Consul):

dig +search prometheus-server.service.consul         

; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> +search prometheus-server.service.consul
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44728
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;prometheus-server.service.consul.core.svc.cluster.local. IN A

;; ANSWER SECTION:
prometheus-server.service.consul.core.svc.cluster.local. 5 IN CNAME consul.service.consul.
consul.service.consul.    5    IN    A    10.17.17.208
consul.service.consul.    5    IN    A    10.17.16.39
consul.service.consul.    5    IN    A    10.17.17.231

;; Query time: 1 msec
;; SERVER: 172.20.0.10#53(172.20.0.10)
;; WHEN: Thu Jul 18 01:54:28 UTC 2019
;; MSG SIZE  rcvd: 274

This is because of the /etc/resolv.conf for default namespace pods configured by Kubernetes:

# cat /etc/resolv.conf 
nameserver 172.20.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ap-southeast-1.compute.internal
options ndots:5

v.s pods in the core namespace:

# cat /etc/resolv.conf 
nameserver 172.20.0.10
search core.svc.cluster.local svc.cluster.local cluster.local ap-southeast-1.compute.internal
options ndots:5

What is happening is that in the core namespace, xxx.service.consul is being searched for in the core.svc.cluster.local domain and this resolves to the consul ExternalName service that has been synced to Kubernetes by Consul Catalog Sync. Thus, the DNS returned from CoreDNS is unexpected.

This does not happen in a cluster running kube-dns, as expected because ExternalName only works with CoreDNS. Turning off syncing of services from Consul to Kubernetes and deleting the ExternalName services left behind fixes this issue.

If it helps, I deployed Consul using the Helm chart provided by HashiCorp.

Not sure what can be done about this. The only solution I can come up with is to disable syncing from Consul to Kubernetes.

[lkysow]

Hi Yong, thanks for the detailed write-up.

If your services are using {service name}.service.consul then you can turn off syncing

syncCatalog:
  toK8S: false

The syncing is only used so that {service name} resolves but if you're always adding the .service.consul then you don't need those services registered since the CoreDNS rule will take any .consul lookups and redirect them to the Consul servers.

Another workaround is to set

syncCatalog:
  k8sPrefix: consul

This will cause the service we register to be consul-consul and so it won't match xxx.service.consul.

[lawliet89]

Thanks for the response. I think turning off syncing to Kubernetes seems like the right solution for me.

I guess this isn't really a bug per se but more of something that might be useful to be documented since it took me a whole day to figure out what was going on.

[lkysow]

I hear you! I think it really is a bug because we shouldn't be syncing the consul service over since that borks DNS.

[lkysow]

@lawliet89 I'm looking at this ticket again and wondering what the use case is for using catalog sync and consul DNS. Because if catalog sync is used, then I think the idea is that you don't need to use Consul DNS.

The only solution I can think of is adding a flag -sync-consul-service that defaults to false and controls whether we sync the consul service into kube.

[lawliet89]

There isn't one per se. I just had it enabled for "completeness" sake.

[lkysow]

Okay I think we should close this then since it's not required and the workaround is confusing.