Missed events from kubernetes_events in Elasticsearch

Bug Report

Describe the bug kubectl give us much more events than Fluent Bit kubernetes_events plugin logged to Elasticsearch

To Reproduce

Deploy Fluent Bit with IN/kubernetes_events and OUT/es,stdout with default configurations

configuration
Deployment via [Terraform](https://www.terraform.io/) and [Fluent Bit Helm chart](https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit) **`fluent-bit.tf`** ```terraform # Fluent Bit resource "helm_release" "fluent-bit" { name = "fluent-bit" repository = "https://fluent.github.io/helm-charts" chart = "fluent-bit" version = "0.44.0" namespace = logs timeout = 300 values = [file("${path.module}/values/fluent-bit.yaml")] } ``` **`fluent-bit.yaml`** ```yaml # Chart - https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit # Main kind: Deployment replicaCount: 1 image: repository: cr.fluentbit.io/fluent/fluent-bit tag: 3.0.0-debug testFramework: enabled: false # Permissions rbac: create: true nodeAccess: true eventsAccess: true # Configuration config: service: | [SERVICE] flush 1 daemon Off LogLevel info Config_Watch On Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 inputs: | [INPUT] Name kubernetes_events Tag kube_events Kube_url https://kubernetes.default.svc outputs: | [OUTPUT] Name es Match kube_events Host elastic Port 9200 Compress gzip Type _doc Suppress_Type_Name On Logstash_Format On Logstash_Prefix kube-events tls On tls.verify Off [OUTPUT] name stdout match kube_events Format json ```

Kill a regular Pod

kubectl delete pod vector-867p4 -n logging

Check events grabbed by Fluent Bit and pushed to the Elasticsearch and shown in the stdout

screenshot and json
**`Elasticsearch`** ```json { "_index": "kube-events-2024.03.25", "_id": "H6OedY4BPsgOOuw_T6Me", "_version": 1, "_score": 0, "_source": { "@timestamp": "2024-03-25T12:38:11.000Z", "metadata": { "name": "vector-xkfct.17c00236833eea4f", "namespace": "logging", "uid": "95305702-4aaf-4756-ba9d-90c46fe30ea3", "resourceVersion": "5587321", "creationTimestamp": "2024-03-25T12:38:11Z", "managedFields": [ { "manager": "kube-scheduler", "operation": "Update", "apiVersion": "events.k8s.io/v1", "time": "2024-03-25T12:38:11Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:action": {}, "f:eventTime": {}, "f:note": {}, "f:reason": {}, "f:regarding": {}, "f:reportingController": {}, "f:reportingInstance": {}, "f:type": {} } } ] }, "involvedObject": { "kind": "Pod", "namespace": "logging", "name": "vector-xkfct", "uid": "e4ffec7b-15ad-4dd9-b4cb-9ba3f685cf2e", "apiVersion": "v1", "resourceVersion": "5587317" }, "reason": "Scheduled", "message": "Successfully assigned logging/vector-xkfct to node-s-4vcpu-16gb-amd-oid1q", "source": {}, "firstTimestamp": null, "lastTimestamp": null, "type": "Normal", "eventTime": "2024-03-25T12:38:11.554210Z", "action": "Binding", "reportingComponent": "default-scheduler", "reportingInstance": "default-scheduler-kube-scheduler-84f7bb9b87-j7n4s" }, "fields": { "metadata.managedFields.apiVersion": [ "events.k8s.io/v1" ], "involvedObject.name": [ "vector-xkfct" ], "reason": [ "Scheduled" ], "metadata.managedFields.manager.keyword": [ "kube-scheduler" ], "reportingComponent.keyword": [ "default-scheduler" ], "involvedObject.namespace": [ "logging" ], "metadata.managedFields.operation": [ "Update" ], "type": [ "Normal" ], "involvedObject.kind": [ "Pod" ], "involvedObject.uid.keyword": [ "e4ffec7b-15ad-4dd9-b4cb-9ba3f685cf2e" ], "metadata.resourceVersion.keyword": [ "5587321" ], "reason.keyword": [ "Scheduled" ], "type.keyword": [ "Normal" ], "involvedObject.apiVersion.keyword": [ "v1" ], "metadata.managedFields.fieldsType.keyword": [ "FieldsV1" ], "metadata.managedFields.time": [ "2024-03-25T12:38:11.000Z" ], "eventTime": [ "2024-03-25T12:38:11.554Z" ], "action": [ "Binding" ], "involvedObject.name.keyword": [ "vector-xkfct" ], "metadata.name": [ "vector-xkfct.17c00236833eea4f" ], "action.keyword": [ "Binding" ], "metadata.uid.keyword": [ "95305702-4aaf-4756-ba9d-90c46fe30ea3" ], "metadata.resourceVersion": [ "5587321" ], "involvedObject.uid": [ "e4ffec7b-15ad-4dd9-b4cb-9ba3f685cf2e" ], "reportingInstance": [ "default-scheduler-kube-scheduler-84f7bb9b87-j7n4s" ], "metadata.managedFields.manager": [ "kube-scheduler" ], "metadata.name.keyword": [ "vector-xkfct.17c00236833eea4f" ], "involvedObject.resourceVersion": [ "5587317" ], "metadata.managedFields.operation.keyword": [ "Update" ], "metadata.creationTimestamp": [ "2024-03-25T12:38:11.000Z" ], "message": [ "Successfully assigned logging/vector-xkfct to node-s-4vcpu-16gb-amd-oid1q" ], "reportingComponent": [ "default-scheduler" ], "involvedObject.namespace.keyword": [ "logging" ], "reportingInstance.keyword": [ "default-scheduler-kube-scheduler-84f7bb9b87-j7n4s" ], "involvedObject.resourceVersion.keyword": [ "5587317" ], "@timestamp": [ "2024-03-25T12:38:11.000Z" ], "involvedObject.apiVersion": [ "v1" ], "metadata.managedFields.apiVersion.keyword": [ "events.k8s.io/v1" ], "metadata.uid": [ "95305702-4aaf-4756-ba9d-90c46fe30ea3" ], "message.keyword": [ "Successfully assigned logging/vector-xkfct to node-s-4vcpu-16gb-amd-oid1q" ], "metadata.namespace.keyword": [ "logging" ], "metadata.managedFields.fieldsType": [ "FieldsV1" ], "involvedObject.kind.keyword": [ "Pod" ], "metadata.namespace": [ "logging" ] } } ``` **`stdout`** > `[{"date":1711370291.0,"metadata":{"name":"vector-867p4.17c002366ae4a446","namespace":"logging","uid":"caf04aca-c580-45f9-8f5f-d77bf01f59b3","resourceVersion":"5587313","creationTimestamp":"2024-03-25T12:38:11Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-03-25T12:38:11Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"logging","name":"vector-867p4","uid":"754aef8e-911d-4a5c-ab31-36cf38597a71","apiVersion":"v1","resourceVersion":"3918692","fieldPath":"spec.containers{vector}"},"reason":"Killing","message":"Stopping container vector","source":{"component":"kubelet","host":"node-s-4vcpu-16gb-amd-oid1q"},"firstTimestamp":"2024-03-25T12:38:11Z","lastTimestamp":"2024-03-25T12:38:11Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"node-s-4vcpu-16gb-amd-oid1q"}]` > > `[{"date":1711370291.0,"metadata":{"name":"vector-xkfct.17c00236833eea4f","namespace":"logging","uid":"95305702-4aaf-4756-ba9d-90c46fe30ea3","resourceVersion":"5587321","creationTimestamp":"2024-03-25T12:38:11Z","managedFields":[{"manager":"kube-scheduler","operation":"Update","apiVersion":"events.k8s.io/v1","time":"2024-03-25T12:38:11Z","fieldsType":"FieldsV1","fieldsV1":{"f:action":{},"f:eventTime":{},"f:note":{},"f:reason":{},"f:regarding":{},"f:reportingController":{},"f:reportingInstance":{},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"logging","name":"vector-xkfct","uid":"e4ffec7b-15ad-4dd9-b4cb-9ba3f685cf2e","apiVersion":"v1","resourceVersion":"5587317"},"reason":"Scheduled","message":"Successfully assigned logging/vector-xkfct to node-s-4vcpu-16gb-amd-oid1q","source":{},"firstTimestamp":null,"lastTimestamp":null,"type":"Normal","eventTime":"2024-03-25T12:38:11.554210Z","action":"Binding","reportingComponent":"default-scheduler","reportingInstance":"default-scheduler-kube-scheduler-84f7bb9b87-j7n4s"},{"date":1711370291.0,"metadata":{"name":"vector.17c00236828c0f1d","namespace":"logging","uid":"6816408c-8756-4d52-a6e9-098e4c8446c4","resourceVersion":"5587319","creationTimestamp":"2024-03-25T12:38:11Z","managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2024-03-25T12:38:11Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:source":{"f:component":{}},"f:type":{}}}]},"involvedObject":{"kind":"DaemonSet","namespace":"logging","name":"vector","uid":"b0468ef7-2b18-4c66-8983-f1a70d3befa9","apiVersion":"apps/v1","resourceVersion":"5568369"},"reason":"SuccessfulCreate","message":"Created pod: vector-xkfct","source":{"component":"daemonset-controller"},"firstTimestamp":"2024-03-25T12:38:11Z","lastTimestamp":"2024-03-25T12:38:11Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"daemonset-controller","reportingInstance":""},{"date":1711370292.0,"metadata":{"name":"vector-xkfct.17c00236ac070e70","namespace":"logging","uid":"795428a6-8f94-4f6a-96b0-ee2b2be91424","resourceVersion":"5587332","creationTimestamp":"2024-03-25T12:38:12Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-03-25T12:38:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"logging","name":"vector-xkfct","uid":"e4ffec7b-15ad-4dd9-b4cb-9ba3f685cf2e","apiVersion":"v1","resourceVersion":"5587318","fieldPath":"spec.containers{vector}"},"reason":"Pulled","message":"Container image \"timberio/vector:0.36.1-distroless-libc\" already present on machine","source":{"component":"kubelet","host":"node-s-4vcpu-16gb-amd-oid1q"},"firstTimestamp":"2024-03-25T12:38:12Z","lastTimestamp":"2024-03-25T12:38:12Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"node-s-4vcpu-16gb-amd-oid1q"},{"date":1711370292.0,"metadata":{"name":"vector-xkfct.17c00236adb426c5","namespace":"logging","uid":"050df0e0-d5e7-45f4-b0bd-1c567de3fc79","resourceVersion":"5587333","creationTimestamp":"2024-03-25T12:38:12Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-03-25T12:38:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"logging","name":"vector-xkfct","uid":"e4ffec7b-15ad-4dd9-b4cb-9ba3f685cf2e","apiVersion":"v1","resourceVersion":"5587318","fieldPath":"spec.containers{vector}"},"reason":"Created","message":"Created container vector","source":{"component":"kubelet","host":"node-s-4vcpu-16gb-amd-oid1q"},"firstTimestamp":"2024-03-25T12:38:12Z","lastTimestamp":"2024-03-25T12:38:12Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"node-s-4vcpu-16gb-amd-oid1q"},{"date":1711370292.0,"metadata":{"name":"vector-xkfct.17c00236b292a35b","namespace":"logging","uid":"e3fe5500-b94c-4249-8ff7-ebe5ec0a1dd8","resourceVersion":"5587334","creationTimestamp":"2024-03-25T12:38:12Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-03-25T12:38:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"logging","name":"vector-xkfct","uid":"e4ffec7b-15ad-4dd9-b4cb-9ba3f685cf2e","apiVersion":"v1","resourceVersion":"5587318","fieldPath":"spec.containers{vector}"},"reason":"Started","message":"Started container vector","source":{"component":"kubelet","host":"node-s-4vcpu-16gb-amd-oid1q"},"firstTimestamp":"2024-03-25T12:38:12Z","lastTimestamp":"2024-03-25T12:38:12Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"node-s-4vcpu-16gb-amd-oid1q"}]`

Get events from the cluster

# kubectl get events -n logging
4s          Normal   Killing             pod/vector-867p4                   Stopping container vector
3s          Normal   Scheduled           pod/vector-xkfct                   Successfully assigned logging/vector-xkfct to node-s-4vcpu-16gb-amd-oid1q
3s          Normal   Pulled              pod/vector-xkfct                   Container image "timberio/vector:0.36.1-distroless-libc" already present on machine
3s          Normal   Created             pod/vector-xkfct                   Created container vector
3s          Normal   Started             pod/vector-xkfct                   Started container vector
4s          Normal   SuccessfulCreate    daemonset/vector                   Created pod: vector-xkfct

Expected behavior

We should get events in Elasticsearch similar to the ones we get by kubectl or Fluent Bit stdout

Your Environment

Configuration: attached to the 'To Reproduce' section

Environment name and version (e.g. Kubernetes? What version?):

# kubeclt version
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1

# cr.fluentbit.io/fluent/fluent-bit:3.0.0-debug
# fluent-bit --version
Fluent Bit v3.0.0
Git commit: f499a4fbe1a58e3230cf1f8dce51145f1f721cf0

# elasticsearch --version
Version: 8.12.2, Build: docker/48a287ab9497e852de30327444b0809e55d46466/2024-02-19T10:04:32.774273190Z, JVM: 21.0.2

Filters and plugins:
- IN - Kubernetes Events
- OUT - Elasticsearch
- OUT - Standard Output

Additional context We are trying to collect Kubernetes events because they are stored just one hour and then purged and we need to save a whole history.

fluent / fluent-bit

Missed events from kubernetes_events in Elasticsearch #8633

Bug Report