search

rancher / opni

Multi Cluster Observability with AIOps
https://opni.io
Apache License 2.0
327 stars 52 forks source link

Panic in gateway when node is resource constrained #1853

Open dbason opened 7 months ago

dbason commented 7 months ago 
2023-11-20T06:17:19.793345396Z 2023 Nov 20 06:17:19 DEBUG mgmt management/exte
nsions.go:118 rule descriptors name=alerting.node.NodeAlertingCapability rules="[http:{post:\"/v1/node/sync\"  body:\"*\"}  method:{name:\"Sync\"  input_type:\".alerting.node.AlertingCapabilityConfig\"  output_type:\".alerting.node.SyncResponse\"  options:{[google.api.http]:{post:\"/v1/node/sync\"  body:\"*\"}}}]"
Received terminated, shutting down...panic: unreachable
2023-11-20T06:17:20.078729396Z 
2023-11-20T06:17:20.078737146Z runtime/debug.Stack()
2023-11-20T06:17:20.078750479Z  runtime/debug/stack.go:24 +0x5e
2023-11-20T06:17:20.078758104Z golang.org/x/sync/singleflight.newPanicError({0x4246880?, 0x66d4f70})
2023-11-20T06:17:20.078764854Z  golang.org/x/sync@v0.3.0/singleflight/singleflight.go:35 +0x26
2023-11-20T06:17:20.078772187Z golang.org/x/sync/singleflight.(*Group).doCall.func2.1()
2023-11-20T06:17:20.078778896Z  golang.org/x/sync@v0.3.0/singleflight/singleflight.go:184 +0x34
2023-11-20T06:17:20.078785562Z panic({0x4246880?, 0x66d4f70?})
2023-11-20T06:17:20.078792396Z  runtime/panic.go:914 +0x21f
2023-11-20T06:17:20.078798187Z github.com/rancher/opni/pkg/auth/openid.(*OpenidMiddleware).tryConfigureKeyRefresher.func1()
2023-11-20T06:17:20.078804396Z  github.com/rancher/opni/pkg/auth/openid/openid.go:170 +0x9da
2023-11-20T06:17:20.078810854Z golang.org/x/sync/singleflight.(*Group).doCall.func2(0xc0010a7d1e, 0xc001080500, 0x4000803501?)
2023-11-20T06:17:20.078817812Z  golang.org/x/sync@v0.3.0/singleflight/singleflight.go:189 +0x64
2023-11-20T06:17:20.078824771Z golang.org/x/sync/singleflight.(*Group).doCall(0x44f9be0?, 0xc00108f920?, {0xc000284280?, 0x20?}, 0x0?)
2023-11-20T06:17:20.078831021Z  golang.org/x/sync@v0.3.0/singleflight/singleflight.go:191 +0x96
2023-11-20T06:17:20.078837729Z golang.org/x/sync/singleflight.(*Group).Do(0x9642620, {0xc000284280, 0x20}, 0x0?)
2023-11-20T06:17:20.078844146Z  golang.org/x/sync@v0.3.0/singleflight/singleflight.go:104 +0x15a
2023-11-20T06:17:20.078851187Z github.com/rancher/opni/pkg/auth/openid.(*OpenidMiddleware).tryConfigureKeyRefresher(0xc00100e300, {0x67319b8?, 0xc000eca3c0?})
2023-11-20T06:17:20.078858271Z  github.com/rancher/opni/pkg/auth/openid/openid.go:144 +0xb3
2023-11-20T06:17:20.078864604Z created by github.com/rancher/opni/pkg/auth/openid.New in goroutine 1
2023-11-20T06:17:20.078866854Z  github.com/rancher/opni/pkg/auth/openid/open
+0x2f9
2023-11-20T06:17:20.078870437Z 
2023-11-20T06:17:20.079034312Z 
goroutine 63 [running]:
golang.org/x/sync/singleflight.(*Group).doCall.func1()
2023-11-20T06:17:20.082505104Z  golang.org/x/sync@v0.3.0/singleflight/singleflight.go:161 +0x2b3
2023-11-20T06:17:20.097151646Z golang.org/x/sync/singleflight.(*Group).doCall(0x44f9be0?, 0xc00108f920?, {0xc000284280?, 0x20?}, 0x0?)
2023-11-20T06:17:20.097189437Z  golang.org/x/sync@v0.3.0/singleflight/singleflight.go:196 +0xb1
2023-11-20T06:17:20.097197854Z golang.org/x/sync/singleflight.(*Group).Do(0x9642620, {0xc000284280, 0x20}, 0x0?)
2023-11-20T06:17:20.097199479Z  golang.org/x/sync@v0.3.0/singleflight/singleflight.go:104 +0x15a
2023-11-20T06:17:20.097336104Z github.com/rancher/opni/pkg/auth/openid.(*OpenidMiddleware).tryConfigureKeyRefresher(0xc00100e300, {0x67319b8?, 0xc000eca3c0?})
2023-11-20T06:17:20.097340854Z  github.com/rancher/opni/pkg/auth/openid/openid.go:144 +0xb3
2023-11-20T06:17:20.097351854Z created by github.com/rancher/opni/pkg/auth/openid.New in goroutine 1
2023-11-20T06:17:20.097353437Z  github.com/rancher/opni/pkg/auth/openid/openid.go:70 +0x2f9
kralicky commented 7 months ago

The only panic("unreachable") is here: https://github.com/rancher/opni/tree/main/pkg/auth/openid/openid.go#L170 which is called from the noauth setup code. Is this being removed anyway in your auth branch?