DNS names for pods using headless service and deployment

stroop23 commented 7 years ago

BUG REPORT:

/kind bug @kubernetes/sig-network

What happened: When setting up a headless service pointing to the pods managed by a deployment/replicaset there are there are either

no DNS names provisioned, (hostname field empty in podspec) or
there's single dns name pointing to the ip of the last pod to be brought up (hostname field was set in the podspec)

What you expected to happen: As per the kube-dns spec, i expect dns names to be set up for each pod in the replicaset using the pod name as hostname if the hostname field in the pod isn't explicitly set, provided that the subdomain field is set equal to the service name (as per documentation) This behaviour is actually present elsewhere: In this set up the /etc/hosts file of containers in the pod does have the expected host name

How to reproduce it (as minimally and precisely as possible):

Applying the yaml in the attached zip, creates a single pod, a deployment and headless services for each. The deployment has no hostname set, the pod does. Inspecting the addresses in the endpoints of the services, show that for the service attached to the deployment no hostname was added to the address, and as such kube-dns doesn't pick up the entry. However the hosts files of the containers do have the expected entry, for example:

$ kubectl exec -ti pod-1 ping test-deployment-813627333-chfqr.deployment-service.default.svc.cluster.local
ping: unknown host

 $ kubectl exec -ti test-deployment-813627333-chfqr  cat /etc/hosts | grep test
10.56.0.83  test-deployment-813627333-chfqr.deployment-service.default.svc.cluster.local    test-deployment-813627333-chfqr

If you look at the single pod there is a dns entry which is equal to the hostname in /etc/hosts

$ kubectl exec -ti test-deployment-813627333-chfqr getent hosts pod-1.pod-service.default.svc.cluster.local
10.56.0.87      pod-1.pod-service.default.svc.cluster.local
$ kubectl exec -ti pod-1 cat /etc/hosts | grep pod-1
10.56.0.87  pod-1.pod-service.default.svc.cluster.local pod-1

pod-dns.zip

Anything else we need to know?: Not sure if this should be a fix in kubernetes or the kube-dns addon. Kube-dns checks if the hostname field of the addresses in the endpoint is set. However it might make sense to set that field with the pod name, if no hostname was defined.

Environment:

Kubernetes version (use kubectl version): 1.6.4
Cloud provider or hardware configuration**: GKE
OS (e.g. from /etc/os-release): cos version 57
Kernel (e.g. uname -a): 4.4.35+ #1 SMP Wed Apr 5 13:00:57 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.50GHz GenuineIntel GNU/Linux

k8s-github-robot commented 7 years ago

@spil-dennis There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
e.g., @kubernetes/sig-api-machinery-misc for API Machinery
(2) specifying the label manually: /sig <label>
e.g., /sig scalability for sig/scalability

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

stroop23 commented 7 years ago

/sig network

dgageot commented 7 years ago

I've seen the same behaviour. We have pods that have a hostname set in the spec. DNS resolution seems to work until we scale up the number of pods. nslookup on the hostname only returns the IP of the last pod to be brought up instead of the IPs of all the pods with the same hostname.

stroop23 commented 7 years ago

If you want the ips of all the pods, you should use the DNS name of the service. That works as expected.

Op 24 jun. 2017 2:30 p.m. schreef David Gageot notifications@github.com:

I've seen the same behaviour. We have pods that have a hostname set in the spec. DNS resolution seems to work until we scale up the number of pods. nslookup on the hostname only returns the IP of the last pod to be brought up instead of the IPs of all the pods with the same hostname.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/kubernetes/kubernetes/issues/47992#issuecomment-310835820, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABPRRCr1nVlfd8IH7PQU0eP5jVwmSrdzks5sHQFPgaJpZM4OECxh.

thockin commented 7 years ago

As per the kube-dns spec, i expect dns names to be set up for each pod in the replicaset using the pod name as hostname if the hostname field in the pod isn't explicitly set

https://github.com/kubernetes/dns/blob/master/docs/specification.md

The spec makes no such guarantee. It says it requires an A for the hostname, which is defines as either "The value of the endpoint's hostname field" or "A unique, system-assigned identifier for the endpoint. The exact format and source of this identifier is not prescribed by this specification"

Now, I suppose we could use the pod-name, since we know it's going to be unique because we are selecting within a single namespace. Today we use a hash of other metadata. To change this, we would need the main endpoints controller to copy the Pod name into the EndpointAddress struct. The obvious implication is that the Endpoint struct overall would get bigger because duplicate-folding would become impossible.

On further thought, I bet we're already passing that in as the targetRef, which means duplicate folding has already been defeated (I forget why we did that, but the cost has been somewhat high). DNS could maybe use the targetRef.name iff the targetRef.namespace is the same as the Service in question. That would at least do what you expect (even if the spec doesn't demand it). I'm not sure it's the right thing to do, but it doesn't sound impossible. @bowei might have ideas why not to do that.

I've seen the same behaviour. We have pods that have a hostname set in the spec. DNS resolution seems to work until we scale up the number of pods. nslookup on the hostname only returns the IP of the last pod to be brought up instead of the IPs of all the pods with the same hostname.

The spec says "If there are multiple IP addresses for a given hostname, then there must be one such A record returned for each IP", so it sounds like this may be a straight up bug. Can I get you to file a bug against https://github.com/kubernetes/dns with enough to repro JUST this part?

stroop23 commented 7 years ago

The spec makes no such guarantee. It says it requires an A for the hostname, which is defines as either "The value of the endpoint's hostname field" or "A unique, system-assigned identifier for the endpoint. The exact format and source of this identifier is not prescribed by this specification"

I agree that the spec doesn't state my assumption explicitly. Somehow I expected that the "unique, system-assigned identifier" would be the system assigned pod name.

Today we use a hash of other metadata.

Looking through the code, i see that there's a hash created from the ip and port. There is however no (obvious) way to actually retrieve the dns name generated, which raises the question why the record is created in the first place.

I do not know exactly where, but somewhere there's code that generates the hostname for the pod and inserts this into /etc/hosts. I think it would make sense that at that point it also sets the hostname field of the pod, since that's the hostname the pod itself thinks it has. This would in turn fix the issue with the endpoint.

The spec says "If there are multiple IP addresses for a given hostname, then there must be one such A record returned for each IP", so it sounds like this may be a straight up bug. Can I get you to file a bug against https://github.com/kubernetes/dns with enough to repro JUST this part?

Indeed if an explicit hostname is set in the pod template of replicaset, then there's only one DNS record. If no hostname was set, it does properly create multiple A records. I can file a bug for that.

thockin commented 7 years ago

which raises the question why the record is created in the first place

Implementation detail of SkyDNS that leaked through.

that's the hostname the pod itself thinks it has

The pod doesn't know a full DNS-resolvable name unless you set hostname AND subdomain. Only hostname can be automatically set :( That would give you a DNS-resolvable name, but the pod itself doesn't know that name.

stroop23 commented 7 years ago

Implementation detail of SkyDNS that leaked through.

I don't see how, not adding a record shouldn't cause any issues? Better not create records that will never actually be used.

The pod doesn't know a full DNS-resolvable name unless you set hostname AND subdomain. Only hostname can be automatically set :( That would give you a DNS-resolvable name, but the pod itself doesn't know that name.

That doesn't seem true. If you set only a subdomain then /etc/hosts in the pod will contain the full resolvable DNS name, built out of the pod name, subdomain and namespace:

root@test-deployment-813627333-94087:/# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.56.0.92  test-deployment-813627333-94087.deployment-service.default.svc.cluster.local    test-deployment-813627333-94087

And for example an airflow worker actually communicates this name to the master. It would make sense to me, that DNS would follow the same behaviour.

stroop23 commented 7 years ago

Some further info to elaborate my assumptions: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ states:

Currently when a pod is created, its hostname is the Pod’s metadata.name value.

Then goes on to explain what to do to override this hostname. This text also explains the requirement of the subdomain field to be equal to the service name, for a pod DNS name to be created, as well as what the FQDN for the pod would be.

Given the above description, i think the most logical thing would be to have the endpoint controller use the pod name, in case the hostname wasn't explicitly defined, but the subdomain rule was satisfied.

Sidenote: kube-dns does not care about this subdomain, if there's a hostname set on the endpoint record it will create the dns record with it, otherwise it'll generate something. Interestingly it will add a DNS record, but not a PTR one if it generates it's own name.

stroop23 commented 7 years ago

Related bug reports on kubernetes/dns: https://github.com/kubernetes/dns/issues/116 https://github.com/kubernetes/dns/issues/117

thockin commented 7 years ago

On Mon, Jun 26, 2017 at 4:10 AM, Dennis Docter notifications@github.com wrote:

Some further info to elaborate my assumptions: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ states:

Currently when a pod is created, its hostname is the Pod’s metadata.name value.

This is true - log into a pod and run hostname

$ k run -ti --restart=OnFailure --rm --image=busybox sh
If you don't see a command prompt, try pressing enter.
/ # hostname
sh-lv0w0

Then goes on to explain what to do to override this hostname. This text also explains the requirement of the subdomain field to be equal to the service name for a pod DNS name to be created, as well as what the FQDN the pod would be.

Given the above description, i think the most logical thing would be to have the endpoint controller copy the pod name in case the hostname wasn't explicitly defined, but the subdomain rule was satisfied.

Sidenote: kube-dns does not care about this subdomain, if there's a hostname set on the endpoint record it will create the dns record with it, otherwise it'll generate something. Interestingly it will add a DNS record, but not a PTR one if it generates it's own name.

We then run into the issue of overlapping hostnames, which has an open PR under consideration.

So the proposal here is, if I understand

if endpoint.hostname != "" && endpoint.subdomain != "" && service.name
== endpoint.subdomain{
  A = hostname
} else if targetRef.name != "" {
  A = targetRef.name
}

I am less comfortable using hostname if subdomain doesn't match. Hostname was only specified to be unique within a subdomain. It seems wrong to use hostname without that. I could MAYBE see using hostname if it exists and subdomain is "". Maybe.

Remind me why this matters? If you care about DNS names, why can't you just set hostname and subdomain in your pod spec?

stroop23 commented 7 years ago

Something along those lines, however i'd keep the current validation like:

hostname := pod.Spec.Hostname
if len(hostname) == 0 {
    hostname = pod.ObjectMeta.Name
}

if len(hostname) > 0 &&
    pod.Spec.Subdomain == service.Name &&
    service.Namespace == pod.Namespace {
    epa.Hostname = hostname
}

I am less comfortable using hostname if subdomain doesn't match. Hostname was only specified to be unique within a subdomain. It seems wrong to use hostname without that. I could MAYBE see using hostname if it exists and subdomain is "". Maybe.

Since that's what the spec is currently, it makes sense not to change that. That being said, if you fall back to the pod name, that one is guaranteed to be unique within the namespace, which is also part of the DNS name, where as for the hostname there are no restrictions at all afaik.

Remind me why this matters? If you care about DNS names, why can't you just set hostname and subdomain in your pod spec?

For single pods that's fine. However in a deployment/rs if you set a hostname, all instances in the deployment/rs will then share the same hostname. DNS will then resolve to the last pod to be handled by kube-dns. (And due to a hashing issue in kube-dns, the service dns name will also only have that single A record)

In many cases there is no issue, as the pod will only use this name internally. However if pods communicate their hostname outside, then this could result in issues if the other party actually tries to use it. (concrete example: Airflow workers communicate their hostname and the master connects to the workers using this hostname)

I think the least intrusive solution to solve this problem, is to just use the podname, if explicit hostname are not possible, like in deployments. Especially since that seems to be the accepted behaviour elsewhere in kubernetes.

premkumar-masilamani commented 7 years ago

Hi - Is there any update on this?. We have the exact requirements where I need to access the individual pods created by deployment, via FQDN. Its a surprise that the FQDN & IP address is set in the Pod, but No DNS entry to resolve them, from other pods.

@spil-dennis Did you manage to find any work-arounds?

stroop23 commented 7 years ago

For now i'm setting hostname and updating /etc/hosts with <pod-ip-with-dashes>.<namespace>.pod.cluster.local on pod startup.

However that hostname is deprecated behaviour afaik, but for now it's still added to DNS.

Cheers, Dennis

premkumar-masilamani commented 7 years ago

Ok. In my case, the ip address of the other pods are not known at creation time (will be created later) and with auto-scaling, it would change frequently. Thanks for the reply.

barrywhart commented 7 years ago

I agree, this is a surprising limitation. We run Airflow on Kubernetes, and it needs the worker pods to have DNS entries, otherwise it can't access logs on the worker pods.

thockin commented 7 years ago

I guess I wouldn't object too hard to using the pod name as hostname if the subdomain was provided. @bowei for thoughts.

PRs welcome...

On Thu, Nov 2, 2017 at 5:19 AM, Barry Hart notifications@github.com wrote:

I agree, this is a surprising limitation. We run Airflow on Kubernetes, and it needs the worker pods to have DNS entries, otherwise it can't access logs on the worker pods https://github.com/apache/incubator-airflow/blob/1.8.2/airflow/www/views.py#L730-L757 .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubernetes/issues/47992#issuecomment-341403240, or mute the thread https://github.com/notifications/unsubscribe-auth/AFVgVH3_tzrcT82pWIcKAbvSA8blFCadks5sybM0gaJpZM4OECxh .

bowei commented 7 years ago

This looks ok, can you make a PR against

https://github.com/kubernetes/dns/blob/master/docs/specification.md

with the proposal?

stroop23 commented 7 years ago

The desired behaviour is already documented in there for headless services:

It's mainly the way of hostname assignment that needs to change. So you can set the subdomain in a template for a pod, but leave the hostname field empty. In that case it should take the pod name as hostname. I'm not sure if that should go into the DNS spec.

smarterclayton commented 7 years ago

There is no requirement that endpoints include pods in a single namespace. It's just that's the default behavior that the controller has today. Someone could easily implement a cross namespace endpoints updater.

On Fri, Nov 3, 2017 at 3:43 AM, Dennis Docter notifications@github.com wrote:

The desired behaviour is already documented in there for headless services:

...svc.. IN A

It's mainly the way of hostname assignment that needs to change. So you can set the subdomain in a template for a pod, but leave the hostname field empty. In that case it should take the pod name as hostname. I'm not sure if that should go into the DNS spec.

— You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubernetes/issues/47992#issuecomment-341638193, or mute the thread https://github.com/notifications/unsubscribe-auth/ABG_p9Xk7lgqS5pYT_pIJDHA6AeNczm6ks5sysQjgaJpZM4OECxh .

mlowery commented 6 years ago

Essentially, this is a discrepancy between /etc/hosts-based lookup vs. nameserver-based lookup if subdomain is specified but hostname is not.

In the short term, I've added an admission controller (which is opt-in via an annotation) that sets hostname to match pod name. This seems to work fine with naked pods and deployments. Any gotchas here?

tomplus commented 6 years ago

Can we extend definition of hostname from https://github.com/kubernetes/dns/blob/master/docs/specification.md by adding step where a name of Pod can be used if a namespace is the same ?

hostname

In order of precedence, the hostname of an endpoint is:
- The value of the endpoint's hostname field.
- A name of Pod if a Pod and a Service are in the same namespace.
- A unique, system-assigned identifier for the endpoint. The exact format and source of this identifier is not prescribed by this specification. However, it must be possible to use this to identify a specific endpoint in the context of a Service. This is used in the event no explicit endpoint hostname is defined.

How does it look ?

fejta-bot commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

thockin commented 6 years ago

/lifecycle frozen /remove-lifecycle stale

apuchitnis commented 6 years ago

@bowei and @thockin, any thoughts on @tomplus ' suggestion?

chrisohaver commented 6 years ago

CoreDNS has the option to use pod names instead of endpoint hostnames/ip.

(Added in this PR: coredns/coredns#1190)

krmayankk commented 6 years ago

/area dns

krmayankk commented 6 years ago

/assign @krmayankk

athenabot commented 5 years ago

@krmayankk If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

athenabot commented 5 years ago

@johnbelamaric @krmayankk If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

johnbelamaric commented 5 years ago

/remove-triage unresolved

This is a feature request, but a similar feature is available in CoreDNS. We can consider adding it to the spec but frankly I'd rather do that if we deprecate kube-dns. I think our time is better spent elsewhere than adding this feature there.

@chrisohaver I noticed while looking at this that I don't think hostname (and endpoint_pod_names) is properly restricting to only when the Pod namespace == Service namespace: https://github.com/coredns/coredns/blob/master/plugin/kubernetes/kubernetes.go#L291

Can you take a look?

chrisohaver commented 5 years ago

@johnbelamaric, I thought all Pods in an Endpoint had to be in the same namespace as the Service. Is that not the case?

johnbelamaric commented 5 years ago

See Clayton's comment above. The current controller works that way, but there is not real reason it has to. Since that is how it works, this hasn't come up in the wild as far as I know.

To test it you can manually create endpoints with a targetRef pointing to a pod in another namespace.

On Thu, Jun 6, 2019 at 10:11 AM chrisohaver notifications@github.com wrote:

@johnbelamaric https://github.com/johnbelamaric, I thought all Pods in an Endpoint had to be in the same namespace as the Service. Is that not the case?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubernetes/issues/47992?email_source=notifications&email_token=ACIHRM72XMX64M6FP25RPV3PZFAKLA5CNFSM4DQQFRQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXDQYTA#issuecomment-499584076, or mute the thread https://github.com/notifications/unsubscribe-auth/ACIHRM5NULFKQPNUQ3YPYJDPZFAKLANCNFSM4DQQFRQQ .

thockin commented 5 years ago

I'm a bit confused. This issue is both a bug report and a feature request:

Bug: a deployment which sets hostname ends up with weird DNS under a headless service.

Feature: if a pod does not set hostname use the pod name.

The remainder is the first time I have thought about this:

Whether the pod behind an endpoint is in the same namespace or not shouldn't matter. The DNS name is relative to the Service. I don't care how an endpoint came to be in the Endpoints of a given Service, whoever put it there believed it belongs there. The name is still relative to a Service.

johnbelamaric commented 5 years ago

I guess I would argue this:

1) Feature is already supported in CoreDNS, it's not part of the spec but it's there. If you want that behavior, go ahead and deploy CoreDNS.

2) The bug is in kube-dns, in that it gives only one IP instead of all IPs. This is also fixed in CoreDNS and the mitigation is to move to that.

/close

k8s-ci-robot commented 5 years ago

@johnbelamaric: Closing this issue.

In response to [this](https://github.com/kubernetes/kubernetes/issues/47992#issuecomment-500049548): >I guess I would argue this: > >1) Feature is already supported in CoreDNS, it's not part of the spec but it's there. If you want that behavior, go ahead and deploy CoreDNS. > >2) The bug is in kube-dns, in that it gives only one IP instead of all IPs. This is also fixed in CoreDNS and the mitigation is to move to that. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

kubernetes / kubernetes

DNS names for pods using headless service and deployment #47992