search

grafana / helm-charts

Apache License 2.0
1.66k stars 2.28k forks source link

Grafana-Tempo Distributed chart pod issue(s) #1213

Open nreddipalle opened 2 years ago

nreddipalle commented 2 years ago

I am spinning up grafana-tempo in AKS using the default config..

helm upgrade --install tempo --namespace=grafana-tempo grafana/tempo-distributed

Only two pods tempo-tempo-distributed-memcached-0 and tempo-tempo-distributed-query-frontend-55b7cb5cdb-7zss9 came up fine and all the other pods are in CrashLoopBackOff state with the below error.

image

image

Anything wrong in my config?? or I need to pass anything while installing.

level=warn ts=2022-04-11T04:06:11.681189008Z caller=ingester.go:159 msg="failed to stop ingester lifecycler" err="failed to join the ring ingester: invalid service state: Failed, expected: Running, failure: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"

warrenhodg commented 2 years ago

You are probably experiencing the same problem as others have when using Loki and Thanos - using an IP address that gets filtered out, so the memberlist doesn't know what to expose as its IP. See https://github.com/thanos-io/thanos/issues/615