search

coredns / rfc

Propsals and request for comments for CoreDNS
0 stars 11 forks source link

CoreDNS Projects for Summer of Code 2020 #2

Closed yongtang closed 3 years ago

yongtang commented 4 years ago

Please Note: This is a tracking issue for Summer of Code. Anyone interested in this implementation should check link there.

Please Note: This is reserved for Summer of Code, we don't accept PRs outside of Summer of Code, or without coordination with maintainers.

Below are the list of candidate CoreDNS projects for Summer of Code 2020:

External health check and orchestration of CoreDNS in Kubernetes clusters

Anomaly detection of CoreDNS server through machine learning

For anyone who is interested in GSoC, please follow the RFC process set forth in this repo.

wjayesh commented 4 years ago

Hello, I am interested in working on CoreDNS for the SOC. Can you help me understand this line? :

While CoreDNS exposes a health check itself, the health check is not UDP (DNS) based.

I have looked at the unregistered plugin "health" but I am not clear about the UDP part. Are the requests sent over TCP now? Apart from this, I am unable to access the contributing.md file. Please let me know of the steps I should be taking next and whether this is the approriate channel for this discussion. Thank you!

yongtang commented 4 years ago

@WJayesh Thanks for the interest. Please see https://github.com/coredns/coredns/issues/3617#issue-554621515 for additional information related to the motivation of this project.

yongtang commented 4 years ago

For anyone that might be interested, I am working on a tutorial of TF+Prometheus+CoreDNS for reading metrics into tf.data.Dataset and used with tf.keras for training a vanilla LSTM model as a starter.

See https://github.com/tensorflow/io/pull/774 for the PR.

You can also check the tutorial from Google Colab in https://colab.research.google.com/github/yongtang/io/blob/prometheus/docs/tutorials/prometheus.ipynb

yongtang commented 4 years ago

@WJayesh Please check https://github.com/cncf/soc for more details about Summer of Code and how to apply.

Syn3rman commented 4 years ago

Hey, I'm interested in participating under CoreDNS for GSoC.

Being a beginner in k8's, I've been reading about networking in k8's for the specific issue at hand and have some doubts regarding the same:

Could you also tell if I'm looking at things the right way and if not, point me in the right direction? Thanks!

Syn3rman commented 4 years ago

..and here's the link to the CONTRIBUTING.md file @WJayesh :smile:

wjayesh commented 4 years ago

Hi Aditya!

Is the health check supposed to be deployed within the cluster or external to the cluster?

This comment suggests that the health check is external.

yongtang commented 4 years ago

To elaborate further on the project:

Let's just look into the problem this project try to solve.

As was mentioned in the project description, coredns is a critical component of Kubernetes and any failure or misbehavior means Kubernetes cluster is in jeopardy.

Since there are limitations on Kubernetes healthcheck, what could we do if we are assigned task of maintaining the health of coredns in a real production server? If all we have is a dig tools and a kubectl, then we could check manually:

  1. Use kubectl to find out the IP address (cluster ip and pod ip) of the coredns service.
  2. Make sure coredns service is up and in good state according to Kubernetes
  3. Use dig to ping pod cluster ip and pod ip through DNS.
  4. Restart coredns pods through kubectl, if pods are up and running but DNS reply does not match the expectation.

The above is the exact problem we solve in a manual way. Of course devops is all about automation so ideally the above steps need to be automated. We could use shell+dig+kubecl to automate everything. But, shell is not good in error handling and may not be reliable depending on which shell (bash/csh/zsh) to use. A golang program will be much more robust--and that is all for this project.

wjayesh commented 4 years ago

Use dig to ping pod cluster ip and pod ip through DNS.

Is there a reason for pinging the pod ip too in addition to the cluster ip? The service would anyway be directing traffic to those ips.

A golang program will be much more robust--and that is all for this project.

I will study golang and start work on this idea if it is ok 😃

wjayesh commented 4 years ago

Can it be designed as an unregistered plugin that when triggered(externally) by an http call, performs an internal self check and returns success/failure?

So, unlike the one step process that exists now (TCP call to 8080/health), it'll be a two step process; first a call to some endpoint of the plugin(similar to existing action) and then, that plugin looks for the IPs and then pings (UDP/dns) them. If the response is unsatisfactory, the plugin will restart the pod itself or send some signal to do the same.

================================ I'll have to test this but it might be possible to configure the livenessProbe on this new plugin as opposed to the health plugin currently. If this is successful, we won't have to worry about restarting the pods ourselves; returning a non success code to the probe will make k8s restart them automatically.

miekg commented 4 years ago

[ Quoting notifications@github.com in "Re: [coredns/coredns] CoreDNS Proje..." ]

Can it be designed as an unregistered plugin that when triggered(externally) by an http call, performs an internal self check and returns success/failure?

So, unlike the one step process that exists now (TCP call to 8080/health), it'll be a two step process; first a call to some endpoint of the plugin(similar to existing action) and then, that plugin looks for the IPs and then pings (UDP/dns) them. If the response is unsatisfactory, the plugin will restart the pod itself or send some signal to do the same.

================================ I'll have to test this but it might be possible to configure the livenessProbe on this new plugin as opposed to the health plugin currently. If this is successful, we won't have to worry about restarting the pods ourselves; returning a non success code to the probe will make k8s restart them automatically.

This could be done, however it hinges on something that's very hard to determine:

What's my external IP?

(And several minor ones). Discussion on these in the original health check via UDP issue thread.

yongtang commented 4 years ago

Is there a reason for pinging the pod ip too in addition to the cluster ip? The service would anyway be directing traffic to those ips.

Pinging a pod ip does not equal to cluster ip. In real production systems, any point of the system could fail. We can not make the assumption that k8s networking will work all the time (in fact in real scenario k8s networking is much less reliable than a standard router/switch).

As part of the project scope discussion, the deployment of the external health check could live outside of a Kubernetes cluster itself. If coredns is ever exposed to outside world (e.g., nodeport/etc) then the external program actually has to be deployed outside as well (note coredns is capable of connect to k8s from outside of the cluster), just to ping coredns the same way it is expected.

chrisohaver commented 4 years ago

Is there a reason for pinging the pod ip too in addition to the cluster ip? The service would anyway be directing traffic to those ips.

Also as far as remediation goes you'll need to know which Pods are sick, so they can be remedied individually. With checks via the Cluster IP of a service with 4 pods behind it, you might be able to determine that for example 25% of checks fail, and estimate that 1 of the 4 pods is failing, but you would not know which one unless you also check the pod IPs directly.

wjayesh commented 4 years ago

@miekg

What's my external IP?

Will using a service(say NodePort) not work? I'm thinking of a plugin similar to the health plugin in that it listens on a port (say 9000) and is process wide. The difference is, we use a service to expose this port 9000) externally. The health plugin port(8080) isn't exposed externally but accessible within the cluster and has the livenessProbe configured on it.

Also, if livenessProbe could be configured(will have to test) on this new plugin(which internally makes udp checks as desired), we won't need the external IP to conduct checks. The kubelet will ping the endpoint by itself and restart whenever the plugin sends a failure code.

wjayesh commented 4 years ago

Discussion on these in the original health check via UDP issue thread.

I had read through the discussion and I admit I didn't understand it all. But if localhost can't be used due to networking complexities, can't we make use of the fact that the plugin is a part of the binary and hence in the same container as the rest of coredns. Using golang's command line execution, we can get the IP for the pod using hostname -i. We could then ping port 53 on this IP through udp, from inside the plugin. I am, however, not sure if it there's a way to restart the pod from within the pod itself

wjayesh commented 4 years ago

If coredns is ever exposed to outside world (e.g., nodeport/etc) then the external program actually has to be deployed outside as well

If I understand it correctly, an external application would be used if we exposed the udp port 53 directly. This program could check the response it recieves from the port and if unsatisfacory, use kubectl to restart the pods.

What my plugin approach assumes is, we don't expose the dns port directly but some other port which the plugin listens on. This plugin then exploits its position(being present in the container) to ping the dns port internally and then somehow restart the pod if the response is not as expected.

If it is ok to expose the dns port to the outside world, then the first approach(external application) could do the job, if I'm not terribly wrong.

yongtang commented 4 years ago

Using golang's command line execution, we can get the IP for the pod using hostname -i.

@WJayesh as was mentioned, getting a pod ip is less of a issues there are too many ways to find network interfaces ip.. However, you cannot get a cluster ip which is not available from container itself. Also, multiple IPs might be exposed, how to figure out which one to use?

To clarify, the Summer-of-Code project (this itself itself) tries to solve a very specific problem of external health check in Kubernetes environment, and is only applied to Kubernetes environment. This project does not intend to cover all scenarios of health-check.

Issue coredns/coredns#3617 tries to address a separate issue of a standalone health check without restricted to the deployment environment. Issue coredns/coredns#3617 does not make any assumption about if coredns is in Kubernetes or not.

chrisohaver commented 4 years ago

However, you cannot get a cluster ip which is not available from container itself.

A CoreDNS Pod can learn the Cluster IP(s) in a round-about way, via the Kubernetes API. A Pod can get it's own IP(s), look those IPs up in the Endpoints API, then look up the Services in the API for the matching Endpoints.

Monitoring the Cluster IP from the Pods is problematic for other reasons though. E.g. the frequency of ClusterIP health checks should not scale up with the number of pods, which would happen if each pod monitors independently.

wjayesh commented 4 years ago

However, you cannot get a cluster ip which is not available from container itself.

The cluster ip we need is the dns service ip and from what I've read "Kubernetes DNS schedules a DNS Pod and Service on the cluster, and configures the kubelets to tell individual containers to use the DNS Service’s IP to resolve DNS names". If I've understood it correctly, this information resides in the /etc/resolv.conf file.

Nonetheless, keeping all of the points discussed here in mind, I'll experiment with a few ideas and implement them this weekend and report if I can get something to work 😃

yongtang commented 4 years ago

@WJayesh just keep in mind while this external health check is targeting for CoreDNS deployment in Kubernetes clusters, the health check program itself does not necessarily deployed in Kubernetes. Also, CoreDNS deployment on Kubernetes does not necessarily follow the "recommended" way mentioned in the link.

chrisohaver commented 4 years ago

The cluster ip we need is the dns service ip and from what I've read "Kubernetes DNS schedules a DNS Pod and Service on the cluster, and configures the kubelets to tell individual containers to use the DNS Service’s IP to resolve DNS names". If I've understood it correctly, this information resides in the /etc/resolv.conf file.

Thats true for Pods that use the clusterFirst dns policy. But CoreDNS doesn't use that dns policy. For good reason, doing so would result in forwarding upstream requests to itself, which would create a forwarding loop.

wjayesh commented 4 years ago

Thats true for Pods that use the clusterFirst dns policy. But CoreDNS doesn't use that dns policy.

Thanks! I missed this detail. I have noticed that the dnsPolicy is set to Default. All I know about this policy is "If a Pod’s dnsPolicy is set to “default”, it inherits the name resolution configuration from the node that the Pod runs on. The Pod’s DNS resolution should behave the same as the node."

This means it inherits the/etc/resolv.conf from its node but I'm not very clear how this policy helps. What are the contents of that file? Does it have the cluster IP of the dns service? If not, how does the pod resolve service names? Again, I'll go hands on and discover these things among others but I'm just putting it out here.

chrisohaver commented 4 years ago

I'm not very clear how this policy helps

In CoreDNS's case, it helps by letting CoreDNS know what upstream DNS servers to use.

wjayesh commented 4 years ago

In CoreDNS's case, it helps by letting CoreDNS know what upstream DNS servers to use.

So, I just verified that the pods with the Default policy have the upstream server set in their /etc/resolv.conf and those with ClusterFirst have the dns service IP.

wjayesh commented 4 years ago

An update: I could test whether the dns service was up, externally. I created a NodePort service that exposed the dns/udp port of the coredns deployment. Then, dig the nodeip -p nodeport.

I remember that an issue of which external IP to choose(in the case where there are multiple) was discussed. If NodePort service is used, then the request can be made to any node in the cluster; what would matter is the port and that can be set to a fixed value while defining the service. This way, we can be sure that any external IP (pertaining to the cluster) that we would be pinging at the given port would take us to the dns/udp endpoint. Is this line of thinking correct?

yongtang commented 4 years ago

@WJayesh Keep in mind the scope of this project is to "add external health-check for all deployment scenarios" (with/without cluster ip, with/without nodeport/etc) The project does not intend to recommend one specific deployment for the user (as we could not control what user want to do in their cluster).

yongtang commented 4 years ago

@WJayesh In case of k8s deployment, what we want to do, is to use kubectl (eventually in client-go/k8s Api in golang) to find out what is the deployment scenario by user, and perform health check accordingly.

wjayesh commented 4 years ago

Understood :+1: So, I will come up with an approach for each possible scenario that the deployment may have and then eventually integrate them into a golang program

wjayesh commented 4 years ago

I want your inputs on an idea: I am creating a lightweight single-container pod(with bash, dnsutils and other tools installed) in the kube-system namespace of an existing cluster. This is achieved through calls to the api-server from the golang application or through kubectl. The IP addr of the service and the coredns pods can also be obtained through kubectl get and describe which is stored for a limited time in the golang app Then I run a shell in that container and use dig to determine if the dns server is up or not.

This approach should work(?) for all cases as it does not depend on the cluster configuration of the user.

wjayesh commented 4 years ago

Another approach could've been "ssh"ing directly into the coredns pods but for some reason(maybe the path isn't set or bash isn't configured in the image), the executables don't run in the two coredns pods. And thus the need to create a new pod for the health check.

yongtang commented 4 years ago

@WJayesh You don't need a live pod (with kubectl and dig) inside Kubernetes, you only need kubectl and dig in an environment where your kubectl is able to access k8s api server, and your dig is able to access k8s networking (pod and cluster ips).

wjayesh commented 4 years ago

I am sorry for the ambiguity in my earlier comment, I'll try to be more explicit: The kubectl is with the client itself(not in the pod). We know that the external golang application can connect to the api-server (just like kubectl does) so accessing a k8s api-server from external environment isn't an issue.

But I imagined the dig command to be inside the pod in the cluster. For dig to be able to access the isolated private network from outside, we can either have a service exposing the ports or have a shell running inside the cluster from where we can access cluster-local services and pods. I chose the second approach of bashing into a container and using it to access the coredns pods.

Is there a different approach(apart from the two mentioned) that I am missing and which can be used to access k8s networking externally?

yongtang commented 4 years ago

@WJayesh While a lot of the standard k8s networking assumes k8s networking is an isolated CIDR, there is no such absolute restrictions. Your pod is accessible as long as the CIDR of the k8s cluster is routable, which in many cases like multi-clusters. You also could use kube-proxy to attach to the network I think.

wjayesh commented 4 years ago

Your pod is accessible as long as the CIDR of the k8s cluster is routable

Are we talking about a route table which has routes set up for the CIDR block? In any case, I'll have to research a bit on this. Any resources on how to do this would be helpful although I would be scraping the web for it too

You also could use kube-proxy to attach to the network I think

I think it is kubectl proxy that establishes connection to the api server on the local machine. This makes the api endpoints accessible on the localhost but I'm not sure if we can direct traffic to the cluster IPs (i can't find a mechanism for it)

yongtang commented 4 years ago

@WJayesh

Are we talking about a route table which has routes set up for the CIDR block? In any case, I'll have to research a bit on this. Any resources on how to do this would be helpful although I would be scraping the web for it too

This is not the focus of this project. The focus of the project is just to make sure the binary you build could be launched with or without being part of a k8s pod.

wjayesh commented 4 years ago

The binary I build will be an external golang program, not part of a k8s pod. what i had proposed earlier was that this ext program dynamically creates a lightweight pod (through communicating with the api-server) . The only purpose of this pod inside the cluster would be to execute dig command(and maybe other debug commands) that the ext golang application sends to this pod. We can also get rid of the pod right after a check and create one the next time we are running a check so the main application doesn't actually live on the cluster and isn't affected by it.

the pod is just created so that accessing the private k8s cluster network becomes simple (no routing or proxy required). It is straightforward to run a shell in a container(through kubectl exec) and then equally simple to run the dig commands in the shell.

yongtang commented 4 years ago

@WJayesh this is a fine approach. On the other hand, this is also based on the assumption that ops will be ok to see a program launch pods up and downs (and consumes cpu/memory on the cluster). Not every ops will be convinced to allow that. We are trying to make minimal assumption of the k8s deployment or how k8s should be operated because we are not the person maintaining every cluster every day. Our focus is to built a program that could be easily adopted by more ops (which may have different preferences of deployment). The approach you proposed could be recommended but it should not be "the way" the binary will be used.

wjayesh commented 4 years ago

I agree; there should be a least intrusive approach to achieving the same objective. I'll research and experiment all possible ways this can be done and report it here (i have exams next week ~so might be off for a short while~)

wjayesh commented 4 years ago

There really are limited options to access the udp port inside the cluster without modifying the cluster slightly. There's an option of port-forward to route requests from a local port to one on the desired pod, which could've been the ideal solution but it doesn't work for UDP yet. We can enquire about the status but looks like a long-term issue.

wjayesh commented 4 years ago

if we don't want a pod to be created inside the cluster that takes up memory and cpu, we can explore adding a service(NodePort/LoadBalancer) or having an ingress controller serve the existing ClusterIP. I've asked the #kubernetes-dev slack for suggestions but so far no tangible solutions

yongtang commented 4 years ago

@WJayesh I am not sure about your confusion here. When you provide a binary to ops, ops will deploy the binary in a way such that the healthcheck binary will be able to access both the api server and the coredns pod ips.

How they make this happen? There are plenty of ways. They could just launch a health-check pod with k8s yaml and problem solved. if they are not feeling comfortable about create health check pods to compete resource with production usage pods, they could also change the k8s networking configuration so that a binary could access coredns pod ips from outside. They could add several routes in their route table, they could also adds iptable rules if they want.

The key here is that the way ops deploy the healthcheck binary you provide, is the their choice. You are not forcing them to create a pod or forcing them to do one way or another. And you really should not care how they make it happen.

greenpau commented 4 years ago

My $.02, one way of monitoring name resolution here is having an independent service capable of accessing all network namespaces on a server.

As I am writing this, I would say it could be a Prometheus exporter tailored for the above use case. A scrape triggers the enumeration of network namespaces and identification of the ones related to k8s. Then, the exporters performs DNS queries within the namespaces.

Background ref: http://www.admin-magazine.com/Articles/The-practical-benefits-of-network-namespaces/(offset)/3

wjayesh commented 4 years ago

The key here is that the way ops deploy the healthcheck binary you provide, is the their choice. You are not forcing them to create a pod or forcing them to do one way or another. And you really should not care how they make it happen.

Very clear. The binary will be flexible in that it could choose its approach depending on the environment it is deployed in and the access that has been provided to it. And the two cases you've mentioned can be taken care of through determinate solutions we've discussed earlier(one where ops deploys a pod and the other where it changes the k8s networking config to make coredns available to the binary).

My previous comments pertained to a case when ops does not want to create a pod and also does not want to change their k8s networking config (i.e. zero change to the cluster). In such a case, my binary(deployed somewhere externally) would not have any means to perform the health check and I was researching if something could be done to address this specific case. If no solution is found, we can also just tell ops that the binary won't function in such a situation beforehand.

wjayesh commented 4 years ago

I will start a draft proposal addressing this problem and its solution in detail and post it here when done, so any changes or conflicts in understanding could be resolved better. Thoughts?

yongtang commented 4 years ago

The binary will be flexible in that it could choose its approach depending on the environment it is deployed in and the access that has been provided to it. And the two cases you've mentioned can be taken care of through determinate solutions we've discussed earlier(one where ops deploys a pod and the other where it changes the k8s networking config to make coredns available to the binary).

@WJayesh glad this is sorted out 👍

omkarprabhu-98 commented 4 years ago

@yongtang
so to clarify, External health check and orchestration of CoreDNS in Kubernetes clusters requires to develop a go program possibly having the following steps:

  1. connect to the cluster (a k8s client would be appropriate I think)
  2. use the k8s api to check status of coredns service/pods
  3. perform a dns lookup (probably for kubernetes.default.svc.cluster.local ?) within the cluster - need to figure out a way to perform it via the k8s networking
  4. If it is unsuccessful/not what is expected, restart the pods using k8s api Am I on the right track?
omkarprabhu-98 commented 4 years ago

@yongtang
so to clarify, External health check and orchestration of CoreDNS in Kubernetes clusters requires to develop a go program possibly having the following steps:

  1. connect to the cluster (a k8s client would be appropriate I think)
  2. use the k8s api to check status of coredns service/pods
  3. perform a dns lookup (probably for kubernetes.default.svc.cluster.local ?) within the cluster - need to figure out a way to perform it via the k8s networking
  4. If it is unsuccessful/not what is expected, restart the pods using k8s api Am I on the right track?
yongtang commented 4 years ago

@omkarprabhu-98 Yes it is almost there, except:

perform a dns lookup (probably for kubernetes.default.svc.cluster.local ?) within the cluster - need to figure out a way to perform it via the k8s networking should be:

  • perform a DNS look up agains the cluster IP and pod IPs of the coredns pods, and see if any coredns is not in good health.
wjayesh commented 4 years ago

@yongtang I've prepared an initial draft of the proposal for the project. Where should I share it with the team for review and feedback? Thanks!

yongtang commented 4 years ago

@WJayesh @omkarprabhu-98 All discussions could happen here. Ideally we want to have most of the discussions public.