search

opsgenie / kubernetes-event-exporter

Export Kubernetes events to multiple destinations with routing and filtering
Apache License 2.0
1.04k stars 352 forks source link

Failed to export events after running for a while #121

Open ediezh opened 3 years ago

ediezh commented 3 years ago

The only log I see from the container log is the below warning after the pod has been running for a while

W0504 07:57:16.735926       1 reflector.go:302] pkg/kube/watcher.go:95: watch of *v1.Event ended with: The resourceVersion for the provided watch is too old.

Restarting the pod would fix the problem.

But the event exporter should be able to automatically refresh the watched resources list.

timansky commented 3 years ago

having the same problem, is there any healthcheck or example of livinessprobe?

aauren commented 3 years ago

Not sure that this will be the problem for everyone, but the linked issue above appears to be the solution for me when I encountered this problem.

Basically, you should not use the file exporter for stdout:

...
receivers:
  - name: "dump"
     file:
       path: "/dev/stdout"

This is because kubernetes-event-exporter uses a library called lumberjack which will automatically attempt to rotate the file path after ~12 hours. Obviously, /dev/stdout cannot rotated.

People that want to put events on stdout should instead use the stdout receiver:

receivers:
  - name: "dump"
     stdout: {}
youyongsong commented 2 years ago

@mustafaakin-atl upgrade client-go version may solve this issue. https://github.com/kubernetes/kubernetes/pull/95981 this pr add watch health check function.