resmoio / kubernetes-event-exporter

Export Kubernetes events to multiple destinations with routing and filtering
Apache License 2.0
753 stars 149 forks source link

publish updates from the event watch #187

Open tommyzli opened 2 months ago

tommyzli commented 2 months ago

Hi

in the watcher, event updates are ignored: https://github.com/resmoio/kubernetes-event-exporter/blob/master/pkg/kube/watcher.go#L65-L67 That seems to have come from this PR which states:

OnUpdate is called when there was a modification to an object. Events are only spawned/populated once and not modified later. Even if they were to be modified manually, their EventTime and LastTimestamp are going to stay the same as at an event creation (unless modified manually too, which goes against event logic), and so they are going to be discarded by kubernetes-event-exporter anyway

However, the actual spec on event compression states:

If the key for the new event matches the key for a previously generated event (meaning all of the above fields match between the new event and some previously generated event), then the event is considered to be a duplicate and the existing event entry is updated in etcd:

  • The new PUT (update) event API is called to update the existing event entry in etcd with the new last seen timestamp and count.

I confirmed with kubectl get events -w --output-watch-events -A -o json on k8s version v1.27.13-eks-3af4770

{"type":"ADDED","object":{"apiVersion":"v1","count":27710,"eventTime":null,"firstTimestamp":"2024-06-21T14:35:45Z","involvedObject":{"apiVersion":"v1","fieldPath":"spec.containers{mycontainer}","kind":"Pod","name":"mypod-jtxyds7","namespace":"mynamespace","resourceVersion":"294442300","uid":"b9f29a46-52c0-4c49-b443-faf3250fc756"},"kind":"Event","lastTimestamp":"2024-06-25T20:15:41Z","message":"Back-off restarting failed container mycontainer in pod mypod-jtxyds7_mynamespace(b9f29a46-52c0-4c49-b443-faf3250fc756)","metadata":{"creationTimestamp":"2024-06-21T14:35:45Z","managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}},"manager":"kubelet","operation":"Update","time":"2024-06-25T20:15:41Z"}],"name":"mypod-jtxyds7.17db0bb2d6d540c0","namespace":"mynamespace","resourceVersion":"299342233","uid":"e2998bb1-d85f-48c1-8ee7-ba06b90301ed"},"reason":"BackOff","reportingComponent":"kubelet","reportingInstance":"ip-11-11-111-11.us-east-2.compute.internal","source":{"component":"kubelet","host":"ip-11-11-111-11.us-east-2.compute.internal"},"type":"Warning"}}
{"type":"MODIFIED","object":{"apiVersion":"v1","count":27733,"eventTime":null,"firstTimestamp":"2024-06-21T14:35:45Z","involvedObject":{"apiVersion":"v1","fieldPath":"spec.containers{mycontainer}","kind":"Pod","name":"mypod-jtxyds7","namespace":"mynamespace","resourceVersion":"294442300","uid":"b9f29a46-52c0-4c49-b443-faf3250fc756"},"kind":"Event","lastTimestamp":"2024-06-25T20:20:41Z","message":"Back-off restarting failed container mycontainer in pod mypod-jtxyds7_mynamespace(b9f29a46-52c0-4c49-b443-faf3250fc756)","metadata":{"creationTimestamp":"2024-06-21T14:35:45Z","managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}},"manager":"kubelet","operation":"Update","time":"2024-06-25T20:20:41Z"}],"name":"mypod-jtxyds7.17db0bb2d6d540c0","namespace":"mynamespace","resourceVersion":"299346282","uid":"e2998bb1-d85f-48c1-8ee7-ba06b90301ed"},"reason":"BackOff","reportingComponent":"kubelet","reportingInstance":"ip-11-11-111-11.us-east-2.compute.internal","source":{"component":"kubelet","host":"ip-11-11-111-11.us-east-2.compute.internal"},"type":"Warning"}}

You can see the differences between the ADDED and MODIFIED events are the count (27710 to 27733) and the lastTimestamp (2024-06-25T20:15:41Z to 2024-06-25T20:20:41Z)

For another example, the Datadog agent only filters away deleted events

partially related to https://github.com/resmoio/kubernetes-event-exporter/issues/51

choon94 commented 1 month ago

Hey @mustafaakin,

Can we get a review for this? I also meet this issue and i think this PR can fix the issue.