Have 3 node cluster
Ubuntu/Kubernetes/Deis-Workflow-v2.0.0
Each node has good compute resources..
Withing 10-15 mins of fresh installation of deis workflow, k8s dashboard and k8s apis hang up.
After investing, found out that deis-monitor-telegraf (3 pods) were opening up a lot of connections to kube-api server to collect metrics but not closing them.
Continuous, lsof -P -p showed opened sockets increasing continuously until it hit 1024 (this is the default ulimit), at which point, kube-apiserver started logging "too many open files" on 6443 and 8080 ports and the entire system hangs.
I tried increasing the ulimit but it did not help.
I shutdown deis-monitor-telegraf daemon set and lsof came to its stable minimal value.
Tried both v2 and v2-beta versions of telegraf image. Both have the same problem.
Is there a way to collect the metrics on demand instead of collecting them by auto polling?
Have 3 node cluster Ubuntu/Kubernetes/Deis-Workflow-v2.0.0 Each node has good compute resources..
Withing 10-15 mins of fresh installation of deis workflow, k8s dashboard and k8s apis hang up. After investing, found out that deis-monitor-telegraf (3 pods) were opening up a lot of connections to kube-api server to collect metrics but not closing them. Continuous, lsof -P -p showed opened sockets increasing continuously until it hit 1024 (this is the default ulimit), at which point, kube-apiserver started logging "too many open files" on 6443 and 8080 ports and the entire system hangs.
I tried increasing the ulimit but it did not help.
I shutdown deis-monitor-telegraf daemon set and lsof came to its stable minimal value.
Tried both v2 and v2-beta versions of telegraf image. Both have the same problem.
Is there a way to collect the metrics on demand instead of collecting them by auto polling?
Thanks!