Grafana-Loki No Private Ip Address Found

[p1-repo-sync-bot[bot]]

Bug

Description

Describe the problem, what were you doing when you noticed the bug?

Attempting to use NAT'd public subnets; loki appears to not have a good way of handling this without adding in additional configurations. This is something we can make easier for the customer.

level=error ts=2024-05-21T22:17:19.78109034Z caller=loki.go:517 msg="module failed" module=ruler error="failed to start ruler, because it depends on module index-gateway-ring, which has failed: context canceled"
level=error ts=2024-05-21T22:17:19.781095067Z caller=loki.go:517 msg="module failed" module=runtime-config error="context canceled"
level=error ts=2024-05-21T22:17:19.781127277Z caller=loki.go:517 msg="module failed" module=ring error="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0005db360 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [logging-loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false loki false {    false   <nil>}} loki [{ringDesc 0xead6e0} {}]} 0xc00188c0a0 0x43b1820 0xc000ac02a0 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xead6e0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0008b0fc0 0xc0004e63c0 0xc0004e6480 0xc0004e6540 0xc0004e6600 0xc0004e6840 0xc0004e66c0 0xc0004e6900 0xc0004e6780 0xc0008d32c0 0xc0008d3300 0xc0004e6a80 0xc0004e6b40 0xc0004e6cc0 0xc0004e6c00 0xc000892580 0xc000ac1a40 0xc000892570 0xc000892578 0xc0008d3340 0xc0008d3380 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-05-21T22:17:19.781133218Z caller=loki.go:517 msg="module failed" module=compactor error="failed to start compact

Provide any steps possible used to reproduce the error (ideally in an isolated fashion). Use a Public Subnet, the problems and solutions are described in the following articles

https://community.grafana.com/t/gossip-ring-memberlist-no-private-ip-address-found/52209 https://medium.com/@ramasharma_7120/grafana-loki-deployment-error-no-private-ip-address-found-051d9316b861

BigBang Version

What version of BigBang were you running?

This can be retrieved multiple ways:

# via helm
helm ls -n bigbang

# via the deployed umbrella git tag
kubectl get gitrepository -n bigbang

[p1-repo-sync-bot[bot]]

jimmyungerman commented:

---

@jeffv what type of Loki deployment are you seeing this on? Monolith or scalable?

[p1-repo-sync-bot[bot]]

jimmyungerman commented:

---

@jeffv I've added the changes described in the links you provided, and here, to the bigbang universe chart at branch 145-grafana-loki-no-private-ip-address-found.

Do you have the ability to test it on EKS like how you found this error?

[p1-repo-sync-bot[bot]]

jimmyungerman commented:

---

@jeffv I've added the changes described in the links you provided and expanded on here to the bigbang universe chart at branch 145-grafana-loki-no-private-ip-address-found.

Do you have the ability to test it on EKS like how you found this error?