search

influxdata / tick-charts

A repository for Helm Charts for the full TICK Stack
Apache License 2.0
90 stars 74 forks source link

Telegraf to collect StatsD #116

Closed yishaihl closed 4 years ago

yishaihl commented 4 years ago

Hey guys, I have k8s cluster with Telegraf deployed as DaemonSet. We want to collect StatsD from our apps cross the cluster. I have added the following Telegraf configuration to my Telegraf DaemonSet:

[[inputs.statsd]] service_address = "localhost:8125" delete_gauges = true delete_counters = true delete_sets = true delete_timings = true percentiles = [90] metricseparator = ""

And there is no StatsD collections.. what i'm missing here? I should set StatsD collector for each pod individualy?

Thanks 🙏

russorat commented 4 years ago

@yishaihl thanks for the issue. When Telegraf is deployed as a DaemonSet, it deploys a Telegraf Pod to each Node of the cluster. By default, each Pod in k8s is given it's own private IP address, so when you say localhost:8125, you are referencing the Pod that telegraf is running on, not where statsD is. See: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/ for a better explanation.

It sounds like you might want to run Telegraf as a sidecar in each Pod that is sending StatsD metrics. That way, it would run as a container inside each pod, and it would be able to listen for statsD messages on localhost. This blog describes it: https://www.influxdata.com/blog/monitoring-kubernetes-architecture/

yishaihl commented 4 years ago

Thanks a lot! I indeed deployed telegraf as a sidecar.