search

deis / monitor

Monitoring for Deis Workflow
https://deis.com
MIT License
22 stars 32 forks source link

Create a mechanism for determining leader election within the telegraf pod #120

Closed jchauncey closed 7 years ago

jchauncey commented 8 years ago

If we can determine a leader within the telegraf pod it will allow us to only poll the api server from 1 pod which cuts down on traffic and memory stress for the api server.

Thoughts are to use something like this - http://blog.kubernetes.io/2016/01/simple-leader-election-with-Kubernetes.html

arschles commented 8 years ago

Would it be reasonable to run a single telegraf pod specifically configured to poll and API server? My rationale behind this alternative strategy is that setting up and maintaining leader election to poll the API server adds a lot of complexity for a single feature, and it would be a win if we can get away with not doing it for now.

jchauncey commented 8 years ago

Potentially but taht means we have a completely different telegraf manifest for polling the api server

arschles commented 8 years ago

IMO that'll still be a lot less overall complexity. That'd mean there was a RC for just the kube API telegraf (deis-monitor-telegraf-kube-api) and then the already existing daemonset

jchauncey commented 8 years ago

Ok after some spiking this ended up not being worth it (using k8s leader election). Basically the problem we had was that we could determine a leader but if that leader ever went away we were not polling to find a new leader.

Going to keep investigating a better way to handle this

jchauncey commented 7 years ago

When we implement the solution in #139 this will not be an issue