Closed yongtang closed 3 years ago
Hello, I am interested in working on CoreDNS for the SOC. Can you help me understand this line? :
While CoreDNS exposes a health check itself, the health check is not UDP (DNS) based.
I have looked at the unregistered plugin "health" but I am not clear about the UDP part. Are the requests sent over TCP now? Apart from this, I am unable to access the contributing.md file. Please let me know of the steps I should be taking next and whether this is the approriate channel for this discussion. Thank you!
@WJayesh Thanks for the interest. Please see https://github.com/coredns/coredns/issues/3617#issue-554621515 for additional information related to the motivation of this project.
For anyone that might be interested, I am working on a tutorial of TF+Prometheus+CoreDNS for reading metrics into tf.data.Dataset and used with tf.keras for training a vanilla LSTM model as a starter.
See https://github.com/tensorflow/io/pull/774 for the PR.
You can also check the tutorial from Google Colab in https://colab.research.google.com/github/yongtang/io/blob/prometheus/docs/tutorials/prometheus.ipynb
@WJayesh Please check https://github.com/cncf/soc for more details about Summer of Code and how to apply.
Hey, I'm interested in participating under CoreDNS for GSoC.
Being a beginner in k8's, I've been reading about networking in k8's for the specific issue at hand and have some doubts regarding the same:
Could you also tell if I'm looking at things the right way and if not, point me in the right direction? Thanks!
Hi Aditya!
Is the health check supposed to be deployed within the cluster or external to the cluster?
This comment suggests that the health check is external.
To elaborate further on the project:
Let's just look into the problem this project try to solve.
As was mentioned in the project description, coredns is a critical component of Kubernetes and any failure or misbehavior means Kubernetes cluster is in jeopardy.
Since there are limitations on Kubernetes healthcheck, what could we do if we are assigned task of maintaining the health of coredns in a real production server?
If all we have is a dig
tools and a kubectl
, then we could check manually:
The above is the exact problem we solve in a manual way. Of course devops is all about automation so ideally the above steps need to be automated. We could use shell+dig+kubecl to automate everything. But, shell is not good in error handling and may not be reliable depending on which shell (bash/csh/zsh) to use. A golang program will be much more robust--and that is all for this project.
Use dig to ping pod cluster ip and pod ip through DNS.
Is there a reason for pinging the pod ip too in addition to the cluster ip? The service would anyway be directing traffic to those ips.
A golang program will be much more robust--and that is all for this project.
I will study golang and start work on this idea if it is ok 😃
Can it be designed as an unregistered plugin that when triggered(externally) by an http call, performs an internal self check and returns success/failure?
So, unlike the one step process that exists now (TCP call to 8080/health), it'll be a two step process; first a call to some endpoint of the plugin(similar to existing action) and then, that plugin looks for the IPs and then pings (UDP/dns) them. If the response is unsatisfactory, the plugin will restart the pod itself or send some signal to do the same.
================================ I'll have to test this but it might be possible to configure the livenessProbe on this new plugin as opposed to the health plugin currently. If this is successful, we won't have to worry about restarting the pods ourselves; returning a non success code to the probe will make k8s restart them automatically.
[ Quoting notifications@github.com in "Re: [coredns/coredns] CoreDNS Proje..." ]
Can it be designed as an unregistered plugin that when triggered(externally) by an http call, performs an internal self check and returns success/failure?
So, unlike the one step process that exists now (TCP call to 8080/health), it'll be a two step process; first a call to some endpoint of the plugin(similar to existing action) and then, that plugin looks for the IPs and then pings (UDP/dns) them. If the response is unsatisfactory, the plugin will restart the pod itself or send some signal to do the same.
================================ I'll have to test this but it might be possible to configure the livenessProbe on this new plugin as opposed to the health plugin currently. If this is successful, we won't have to worry about restarting the pods ourselves; returning a non success code to the probe will make k8s restart them automatically.
This could be done, however it hinges on something that's very hard to determine:
What's my external IP?
(And several minor ones). Discussion on these in the original health check via UDP issue thread.
Is there a reason for pinging the pod ip too in addition to the cluster ip? The service would anyway be directing traffic to those ips.
Pinging a pod ip does not equal to cluster ip. In real production systems, any point of the system could fail. We can not make the assumption that k8s networking will work all the time (in fact in real scenario k8s networking is much less reliable than a standard router/switch).
As part of the project scope discussion, the deployment of the external health check could live outside of a Kubernetes cluster itself. If coredns is ever exposed to outside world (e.g., nodeport/etc) then the external program actually has to be deployed outside as well (note coredns is capable of connect to k8s from outside of the cluster), just to ping coredns the same way it is expected.
Is there a reason for pinging the pod ip too in addition to the cluster ip? The service would anyway be directing traffic to those ips.
Also as far as remediation goes you'll need to know which Pods are sick, so they can be remedied individually. With checks via the Cluster IP of a service with 4 pods behind it, you might be able to determine that for example 25% of checks fail, and estimate that 1 of the 4 pods is failing, but you would not know which one unless you also check the pod IPs directly.
@miekg
What's my external IP?
Will using a service(say NodePort) not work? I'm thinking of a plugin similar to the health plugin in that it listens on a port (say 9000) and is process wide. The difference is, we use a service to expose this port 9000) externally. The health plugin port(8080) isn't exposed externally but accessible within the cluster and has the livenessProbe configured on it.
Also, if livenessProbe could be configured(will have to test) on this new plugin(which internally makes udp checks as desired), we won't need the external IP to conduct checks. The kubelet will ping the endpoint by itself and restart whenever the plugin sends a failure code.
Discussion on these in the original health check via UDP issue thread.
I had read through the discussion and I admit I didn't understand it all. But if localhost
can't be used due to networking complexities, can't we make use of the fact that the plugin is a part of the binary and hence in the same container as the rest of coredns. Using golang's command line execution, we can get the IP for the pod using hostname -i
. We could then ping port 53 on this IP through udp, from inside the plugin.
I am, however, not sure if it there's a way to restart the pod from within the pod itself
If coredns is ever exposed to outside world (e.g., nodeport/etc) then the external program actually has to be deployed outside as well
If I understand it correctly, an external application would be used if we exposed the udp port 53 directly. This program could check the response it recieves from the port and if unsatisfacory, use kubectl
to restart the pods.
What my plugin approach assumes is, we don't expose the dns port directly but some other port which the plugin listens on. This plugin then exploits its position(being present in the container) to ping the dns port internally and then somehow restart the pod if the response is not as expected.
If it is ok to expose the dns port to the outside world, then the first approach(external application) could do the job, if I'm not terribly wrong.
Using golang's command line execution, we can get the IP for the pod using hostname -i.
@WJayesh as was mentioned, getting a pod ip is less of a issues there are too many ways to find network interfaces ip.. However, you cannot get a cluster ip which is not available from container itself. Also, multiple IPs might be exposed, how to figure out which one to use?
To clarify, the Summer-of-Code project (this itself itself) tries to solve a very specific problem of external health check in Kubernetes environment, and is only applied to Kubernetes environment. This project does not intend to cover all scenarios of health-check.
Issue coredns/coredns#3617 tries to address a separate issue of a standalone health check without restricted to the deployment environment. Issue coredns/coredns#3617 does not make any assumption about if coredns is in Kubernetes or not.
However, you cannot get a cluster ip which is not available from container itself.
A CoreDNS Pod can learn the Cluster IP(s) in a round-about way, via the Kubernetes API. A Pod can get it's own IP(s), look those IPs up in the Endpoints API, then look up the Services in the API for the matching Endpoints.
Monitoring the Cluster IP from the Pods is problematic for other reasons though. E.g. the frequency of ClusterIP health checks should not scale up with the number of pods, which would happen if each pod monitors independently.
However, you cannot get a cluster ip which is not available from container itself.
The cluster ip we need is the dns service ip and from what I've read "Kubernetes DNS schedules a DNS Pod and Service on the cluster, and configures the kubelets to tell individual containers to use the DNS Service’s IP to resolve DNS names". If I've understood it correctly, this information resides in the /etc/resolv.conf
file.
Nonetheless, keeping all of the points discussed here in mind, I'll experiment with a few ideas and implement them this weekend and report if I can get something to work 😃
@WJayesh just keep in mind while this external health check is targeting for CoreDNS deployment in Kubernetes clusters, the health check program itself does not necessarily deployed in Kubernetes. Also, CoreDNS deployment on Kubernetes does not necessarily follow the "recommended" way mentioned in the link.
The cluster ip we need is the dns service ip and from what I've read "Kubernetes DNS schedules a DNS Pod and Service on the cluster, and configures the kubelets to tell individual containers to use the DNS Service’s IP to resolve DNS names". If I've understood it correctly, this information resides in the /etc/resolv.conf file.
Thats true for Pods that use the clusterFirst dns policy. But CoreDNS doesn't use that dns policy. For good reason, doing so would result in forwarding upstream requests to itself, which would create a forwarding loop.
Thats true for Pods that use the clusterFirst dns policy. But CoreDNS doesn't use that dns policy.
Thanks! I missed this detail. I have noticed that the dnsPolicy
is set to Default. All I know about this policy is "If a Pod’s dnsPolicy is set to “default”, it inherits the name resolution configuration from the node that the Pod runs on. The Pod’s DNS resolution should behave the same as the node."
This means it inherits the/etc/resolv.conf
from its node but I'm not very clear how this policy helps. What are the contents of that file? Does it have the cluster IP of the dns service? If not, how does the pod resolve service names?
Again, I'll go hands on and discover these things among others but I'm just putting it out here.
I'm not very clear how this policy helps
In CoreDNS's case, it helps by letting CoreDNS know what upstream DNS servers to use.
In CoreDNS's case, it helps by letting CoreDNS know what upstream DNS servers to use.
So, I just verified that the pods with the Default policy have the upstream server set in their /etc/resolv.conf
and those with ClusterFirst have the dns service IP.
An update: I could test whether the dns service was up, externally. I created a NodePort service that exposed the dns/udp port of the coredns deployment. Then, dig the nodeip -p nodeport.
I remember that an issue of which external IP to choose(in the case where there are multiple) was discussed. If NodePort service is used, then the request can be made to any node in the cluster; what would matter is the port and that can be set to a fixed value while defining the service. This way, we can be sure that any external IP (pertaining to the cluster) that we would be pinging at the given port would take us to the dns/udp endpoint. Is this line of thinking correct?
@WJayesh Keep in mind the scope of this project is to "add external health-check for all deployment scenarios" (with/without cluster ip, with/without nodeport/etc) The project does not intend to recommend one specific deployment for the user (as we could not control what user want to do in their cluster).
@WJayesh In case of k8s deployment, what we want to do, is to use kubectl (eventually in client-go/k8s Api in golang) to find out what is the deployment scenario by user, and perform health check accordingly.
Understood :+1: So, I will come up with an approach for each possible scenario that the deployment may have and then eventually integrate them into a golang program
I want your inputs on an idea:
I am creating a lightweight single-container pod(with bash, dnsutils and other tools installed) in the kube-system
namespace of an existing cluster. This is achieved through calls to the api-server
from the golang application or through kubectl
. The IP addr of the service and the coredns pods can also be obtained through kubectl get and describe which is stored for a limited time in the golang app
Then I run a shell in that container and use dig to determine if the dns server is up or not.
This approach should work(?) for all cases as it does not depend on the cluster configuration of the user.
Another approach could've been "ssh"ing directly into the coredns pods but for some reason(maybe the path isn't set or bash isn't configured in the image), the executables don't run in the two coredns pods. And thus the need to create a new pod for the health check.
@WJayesh You don't need a live pod (with kubectl and dig) inside Kubernetes, you only need kubectl and dig in an environment where your kubectl is able to access k8s api server, and your dig is able to access k8s networking (pod and cluster ips).
I am sorry for the ambiguity in my earlier comment, I'll try to be more explicit: The kubectl is with the client itself(not in the pod). We know that the external golang application can connect to the api-server (just like kubectl does) so accessing a k8s api-server from external environment isn't an issue.
But I imagined the dig command to be inside the pod in the cluster. For dig to be able to access the isolated private network from outside, we can either have a service exposing the ports or have a shell running inside the cluster from where we can access cluster-local services and pods. I chose the second approach of bashing into a container and using it to access the coredns pods.
Is there a different approach(apart from the two mentioned) that I am missing and which can be used to access k8s networking externally?
@WJayesh While a lot of the standard k8s networking assumes k8s networking is an isolated CIDR, there is no such absolute restrictions. Your pod is accessible as long as the CIDR of the k8s cluster is routable, which in many cases like multi-clusters. You also could use kube-proxy to attach to the network I think.
Your pod is accessible as long as the CIDR of the k8s cluster is routable
Are we talking about a route table which has routes set up for the CIDR block? In any case, I'll have to research a bit on this. Any resources on how to do this would be helpful although I would be scraping the web for it too
You also could use kube-proxy to attach to the network I think
I think it is kubectl proxy
that establishes connection to the api server on the local machine. This makes the api endpoints accessible on the localhost but I'm not sure if we can direct traffic to the cluster IPs (i can't find a mechanism for it)
@WJayesh
Are we talking about a route table which has routes set up for the CIDR block? In any case, I'll have to research a bit on this. Any resources on how to do this would be helpful although I would be scraping the web for it too
This is not the focus of this project. The focus of the project is just to make sure the binary you build could be launched with or without being part of a k8s pod.
The binary I build will be an external golang program, not part of a k8s pod. what i had proposed earlier was that this ext program dynamically creates a lightweight pod (through communicating with the api-server) . The only purpose of this pod inside the cluster would be to execute dig command(and maybe other debug commands) that the ext golang application sends to this pod. We can also get rid of the pod right after a check and create one the next time we are running a check so the main application doesn't actually live on the cluster and isn't affected by it.
the pod is just created so that accessing the private k8s cluster network becomes simple (no routing or proxy required). It is straightforward to run a shell in a container(through kubectl exec) and then equally simple to run the dig commands in the shell.
@WJayesh this is a fine approach. On the other hand, this is also based on the assumption that ops will be ok to see a program launch pods up and downs (and consumes cpu/memory on the cluster). Not every ops will be convinced to allow that. We are trying to make minimal assumption of the k8s deployment or how k8s should be operated because we are not the person maintaining every cluster every day. Our focus is to built a program that could be easily adopted by more ops (which may have different preferences of deployment). The approach you proposed could be recommended but it should not be "the way" the binary will be used.
I agree; there should be a least intrusive approach to achieving the same objective. I'll research and experiment all possible ways this can be done and report it here (i have exams next week ~so might be off for a short while~)
There really are limited options to access the udp port inside the cluster without modifying the cluster slightly. There's an option of port-forward
to route requests from a local port to one on the desired pod, which could've been the ideal solution but it doesn't work for UDP yet.
We can enquire about the status but looks like a long-term issue.
if we don't want a pod to be created inside the cluster that takes up memory and cpu, we can explore adding a service(NodePort/LoadBalancer) or having an ingress controller serve the existing ClusterIP. I've asked the #kubernetes-dev slack for suggestions but so far no tangible solutions
@WJayesh I am not sure about your confusion here. When you provide a binary to ops, ops will deploy the binary in a way such that the healthcheck binary will be able to access both the api server and the coredns pod ips.
How they make this happen? There are plenty of ways. They could just launch a health-check pod with k8s yaml and problem solved. if they are not feeling comfortable about create health check pods to compete resource with production usage pods, they could also change the k8s networking configuration so that a binary could access coredns pod ips from outside. They could add several routes in their route table, they could also adds iptable rules if they want.
The key here is that the way ops deploy the healthcheck binary you provide, is the their choice. You are not forcing them to create a pod or forcing them to do one way or another. And you really should not care how they make it happen.
My $.02, one way of monitoring name resolution here is having an independent service capable of accessing all network namespaces on a server.
As I am writing this, I would say it could be a Prometheus exporter tailored for the above use case. A scrape triggers the enumeration of network namespaces and identification of the ones related to k8s. Then, the exporters performs DNS queries within the namespaces.
Background ref: http://www.admin-magazine.com/Articles/The-practical-benefits-of-network-namespaces/(offset)/3
The key here is that the way ops deploy the healthcheck binary you provide, is the their choice. You are not forcing them to create a pod or forcing them to do one way or another. And you really should not care how they make it happen.
Very clear. The binary will be flexible in that it could choose its approach depending on the environment it is deployed in and the access that has been provided to it. And the two cases you've mentioned can be taken care of through determinate solutions we've discussed earlier(one where ops deploys a pod and the other where it changes the k8s networking config to make coredns available to the binary).
My previous comments pertained to a case when ops does not want to create a pod and also does not want to change their k8s networking config (i.e. zero change to the cluster). In such a case, my binary(deployed somewhere externally) would not have any means to perform the health check and I was researching if something could be done to address this specific case. If no solution is found, we can also just tell ops that the binary won't function in such a situation beforehand.
I will start a draft proposal addressing this problem and its solution in detail and post it here when done, so any changes or conflicts in understanding could be resolved better. Thoughts?
The binary will be flexible in that it could choose its approach depending on the environment it is deployed in and the access that has been provided to it. And the two cases you've mentioned can be taken care of through determinate solutions we've discussed earlier(one where ops deploys a pod and the other where it changes the k8s networking config to make coredns available to the binary).
@WJayesh glad this is sorted out 👍
@yongtang
so to clarify, External health check and orchestration of CoreDNS in Kubernetes clusters requires to develop a go program possibly having the following steps:
@yongtang
so to clarify, External health check and orchestration of CoreDNS in Kubernetes clusters requires to develop a go program possibly having the following steps:
@omkarprabhu-98 Yes it is almost there, except:
perform a dns lookup (probably for kubernetes.default.svc.cluster.local ?) within the cluster - need to figure out a way to perform it via the k8s networking should be:
- perform a DNS look up agains the cluster IP and pod IPs of the coredns pods, and see if any coredns is not in good health.
@yongtang I've prepared an initial draft of the proposal for the project. Where should I share it with the team for review and feedback? Thanks!
@WJayesh @omkarprabhu-98 All discussions could happen here. Ideally we want to have most of the discussions public.
Please Note: This is a tracking issue for Summer of Code. Anyone interested in this implementation should check link there.
Please Note: This is reserved for Summer of Code, we don't accept PRs outside of Summer of Code, or without coordination with maintainers.
Below are the list of candidate CoreDNS projects for Summer of Code 2020:
External health check and orchestration of CoreDNS in Kubernetes clusters
Anomaly detection of CoreDNS server through machine learning
For anyone who is interested in GSoC, please follow the RFC process set forth in this repo.