search

resmoio / kubernetes-event-exporter

Export Kubernetes events to multiple destinations with routing and filtering
Apache License 2.0
789 stars 158 forks source link

nil pointer dereference with 1.27 cluster #91

Open dmolik opened 1 year ago

dmolik commented 1 year ago

the the release of 1.27 I'm getting the following:

E0526 10:47:05.249149       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 49 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2685340?, 0x46d49b0})
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x20?})
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/runtime/runtime.go:49 +0x75
panic({0x2685340, 0x46d49b0})
    /usr/local/go/src/runtime/panic.go:884 +0x212
k8s.io/client-go/discovery.convertAPIResource(...)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc000129980, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc000128330, 0x15}, {0xc0004532e0, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0x0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x3059430, 0xc0007059e0})
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc000767690)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x3059430?, 0xc0007059e0?})
    /go/pkg/mod/k8s.io/client-go@v0.26.0/restmapper/discovery.go:148 +0x42
github.com/resmoio/kubernetes-event-exporter/pkg/kube.GetObject(0xc000677360, 0xc0001c4ea0, {0x302f540, 0xc0004a4410})
    /app/pkg/kube/objects.go:28 +0x108
github.com/resmoio/kubernetes-event-exporter/pkg/kube.(*LabelCache).GetLabelsWithCache(0xc000361e40, 0xc000677360)
    /app/pkg/kube/labels.go:40 +0xab
github.com/resmoio/kubernetes-event-exporter/pkg/kube.(*EventWatcher).onEvent(0xc0001595c0, 0xc000677258)
    /app/pkg/kube/watcher.go:95 +0x29d
github.com/resmoio/kubernetes-event-exporter/pkg/kube.(*EventWatcher).OnAdd(0x0?, {0x2a86d00?, 0xc000677258?})
    /app/pkg/kube/watcher.go:62 +0x32
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /go/pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/shared_informer.go:911 +0x134
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x100047378a0?)
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000074f38?, {0x302f500, 0xc00059a000}, 0x1, 0xc000168000)
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000379d00?)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/shared_informer.go:905 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:73 +0x85
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12d1050]

goroutine 49 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x20?})
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x2685340, 0x46d49b0})
    /usr/local/go/src/runtime/panic.go:884 +0x212
k8s.io/client-go/discovery.convertAPIResource(...)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc000129980, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc000128330, 0x15}, {0xc0004532e0, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0x0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x3059430, 0xc0007059e0})
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc000767690)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x3059430?, 0xc0007059e0?})
    /go/pkg/mod/k8s.io/client-go@v0.26.0/restmapper/discovery.go:148 +0x42
github.com/resmoio/kubernetes-event-exporter/pkg/kube.GetObject(0xc000677360, 0xc0001c4ea0, {0x302f540, 0xc0004a4410})
    /app/pkg/kube/objects.go:28 +0x108
github.com/resmoio/kubernetes-event-exporter/pkg/kube.(*LabelCache).GetLabelsWithCache(0xc000361e40, 0xc000677360)
    /app/pkg/kube/labels.go:40 +0xab
github.com/resmoio/kubernetes-event-exporter/pkg/kube.(*EventWatcher).onEvent(0xc0001595c0, 0xc000677258)
    /app/pkg/kube/watcher.go:95 +0x29d
github.com/resmoio/kubernetes-event-exporter/pkg/kube.(*EventWatcher).OnAdd(0x0?, {0x2a86d00?, 0xc000677258?})
    /app/pkg/kube/watcher.go:62 +0x32
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /go/pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/shared_informer.go:911 +0x134
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x100047378a0?)
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000074f38?, {0x302f500, 0xc00059a000}, 0x1, 0xc000168000)
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000379d00?)
    /go/pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/shared_informer.go:905 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.26.0/pkg/util/wait/wait.go:73 +0x85
Joibel commented 1 year ago

I have two clusters on 1.27, with identical event exporters. Only the one with prometheus adapter in it is doing this.

sherifkayad commented 1 year ago

Hello, even with the latest Docker Image version ghcr.io/resmoio/kubernetes-event-exporter:v1.2 I am still getting the panic on my Kubernetes v1.27 cluster .. any ideas? 😢

sherifkayad commented 1 year ago

Also like @Joibel I have it working on one my clusters and crashing on others .. How can I even get the reason? Have you guys tested already with Kubernetes clusters >= 1.27 ??

Joibel commented 1 year ago

@sherifkayad Do you have prometheus-adapter installed on the failing one? Or some other metrics provider?

sherifkayad commented 1 year ago

Yes, I do .. but actually the Prometheus Adapter is installed on both clusters 😄 .. that's the part that I find really crazy

sherifkayad commented 1 year ago

@Joibel @dmolik I guess after some readings in tons of issues, I came down to the root cause; that's a weird combination of Kubernetes 1.27 + Prometheus Adapter pushing metrics in a wrong format to the k8s API + a faulty Go k8s API Lib version ..

The last piece of the puzzle is the Prometheus Adapter (https://github.com/kubernetes-sigs/prometheus-adapter/issues/581), luckily the folks issued a fixed release yesterday (v0.11.0) and I can confirm that upon upgrading to that one everything is back to normal!

dmolik commented 1 year ago

Yes, we upgraded Prometheus adapters and clusters yesterday, no issues to report, we'll be updating k-e-e today or tomorrow.On Jul 26, 2023 4:11 PM, Sherif Ayad @.***> wrote: @Joibel @dmolik I guess after some readings in tons of issues, I came down to the root cause; that's a weird combination of Kubernetes 1.27 + Prometheus Adapter pushing metrics in a wrong format to the k8s API + a faulty Go k8s API Lib version .. The last piece of the puzzle is the Prometheus Adapter (kubernetes-sigs/prometheus-adapter#581), luckily the folks issued a fixed release yesterday (v0.11.0) and I can confirm that upon upgrading to that one everything is back to normal!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>