Filebeat can't collect the k8s metadata

KevinLiangX commented 1 year ago

vesion: filebeat:8.7.1

env: kubeneretes 1.24

configuration: filebeat.yml: | filebeat.inputs:

type: container enabled: true paths:
- /var/log/containers/ exclude_files: ["filebeat"] processors:
- add_id: ~
- add_kubernetes_metadata: in_cluster: true host: ${NODE_NAME} matchers:
  - logs_path: logs_path: "/var/log/containers/"
- add_fields: target: kubernetes fields: cluster: ${CLUSTER_NAME} output: kafka: enabled: true hosts: ["10.64.8.83:30011","10.64.8.42:30012","10.64.8.18:30013"] topic: filebeat max_message_bytes: 5242880 partition.round_robin: reachable_only: true keep-alive: 120 required_acks: 1

we use filebeat to kafka to clickhouse, there are no k8s metadata in the event.

some message in kafka:

{"@timestamp":"2023-07-03T03:06:34.456Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.7.1","_id":"gjm7GYkBFBpxZ8vOU_bb"},"message":"I0703 03:06:34.456328 1 server.go:558] external host was not specified, using 10.64.8.83","input":{"type":"container"},"kubernetes":{"cluster":"local-1"},"ecs":{"version":"8.0.0"},"host":{"name":"imp-filebeat-m42jk"},"agent":{"version":"8.7.1","ephemeral_id":"3c5cdc3d-d4ff-4d27-b5e8-598516a939f9","id":"8addecc7-2408-4999-a156-76fa1dd1b577","name":"imp-filebeat-m42jk","type":"filebeat"},"log":{"offset":0,"file":{"path":"/var/log/containers/kube-apiserver-orion2_kube-system_kube-apiserver-f7293aa9c53e2774859c2c772b2335e62eaa787822195816220c1db99a6f8e6e.log"}},"stream":"stderr"}

some filebeat log for metadata 2023-07-03T13:54:11.33627537+08:00 stderr F {"log.level":"info","@timestamp":"2023-07-03T05:54:11.335Z","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":73},"message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.24.13","service.name":"filebeat","ecs.version":"1.6.0"}

2023-07-03T13:54:11.350535465+08:00 stderr F {"log.level":"info","@timestamp":"2023-07-03T05:54:11.349Z","log.logger":"kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":129},"message":"kubernetes: Node master1 discovered by in cluster pod node query","service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}

2023-07-03T13:54:11.310521667+08:00 stderr F {"log.level":"info","@timestamp":"2023-07-03T05:54:11.309Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.exclude_files.0 filebeat.inputs.0.paths.0 filebeat.inputs.0.processors.1.add_kubernetes_metadata.host filebeat.inputs.0.processors.1.add_kubernetes_metadata.in_cluster filebeat.inputs.0.processors.1.add_kubernetes_metadata.matchers.0.logs_path.logs_path filebeat.inputs.0.processors.2.add_fields.fields.cluster filebeat.inputs.0.processors.2.add_fields.target filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}

is there ideas for it ?

plus
the normal event

{"@timestamp":"2023-07-03T02:50:58.761Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.7.1","_id":"SJ67GYkBHeXtMnisXMfn"},"stream":"stderr","kubernetes":{"pod":{"uid":"20a60e92-154a-4723-ab49-dbf61bd217c3","ip":"10.64.8.18","name":"kube-apiserver-arm-orion1"},"cluster":"local-1","namespace":"kube-system","namespace_uid":"5effb7a2-a126-40b4-a494-102311e82c24","namespace_labels":{"field_cattle_io/projectId":"p-bptlm","kubernetes_io/metadata_name":"kube-system"},"labels":{"component":"kube-apiserver","tier":"control-plane"},"container":{"name":"kube-apiserver"},"node":{"name":"arm-orion1","uid":"2f65608b-ee9b-4f28-b081-a9ba12e45dbd","labels":{"kubernetes_io/hostname":"arm-orion1","kubernetes_io/os":"linux","node-role_kubernetes_io/control-plane":"","node_kubernetes_io/exclude-from-external-load-balancers":"","beta_kubernetes_io/arch":"arm64","beta_kubernetes_io/os":"linux","kubernetes_io/arch":"arm64"},"hostname":"arm-orion1"}},"host":{"name":"imp-filebeat-9pwmn"},"log":{"offset":330004,"file":{"path":"/var/log/containers/kube-apiserver-arm-orion1_kube-system_kube-apiserver-dffcbb96bc57e6118624094c3b6e7f3b14fd9b133c4f98461002d9ef7f8c4290.log"}},"message":"Trace[745652572]: [1.033300555s] [1.033300555s] END","input":{"type":"container"},"container":{"image":{"name":"registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.6"},"id":"dffcbb96bc57e6118624094c3b6e7f3b14fd9b133c4f98461002d9ef7f8c4290","runtime":"containerd"},"ecs":{"version":"8.0.0"},"agent":{"version":"8.7.1","ephemeral_id":"99a25386-419c-42d2-9da5-8b3f7fd58423","id":"0c8c8586-9278-4c4f-b2d2-24cf02deba37","name":"imp-filebeat-9pwmn","type":"filebeat"}}

KevinLiangX commented 1 year ago

more logs

kubernetes log: [root@arm-orion1 ~]# grep "I0703 14:44:52.599925 3294735 connection.go:184] GRPC error: " /var/log/containers/* /var/log/containers/csi-xsky-iscsi-driver-xr88q_xsky-iscsi_driver-registrar-8e467d01b9dcb1dc1a5d2c0cc659a448bde92a4147a3f793f16adebe31f6425e.log:2023-07-03T14:44:52.600012926+08:00 stderr F I0703 14:44:52.599925 3294735 connection.go:184] GRPC error:

kafka logs

{"@timestamp":"2023-07-03T06:44:52.600Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.7.1","_id":"TXt_GokBQktRn7ZLOSnC"},"input":{"type":"container"},"kubernetes":{"cluster":"local-1"},"agent":{"name":"imp-filebeat-9pwmn","type":"filebeat","version":"8.7.1","ephemeral_id":"efd0cf26-3982-4a11-8267-aab641e1ed5b","id":"0c8c8586-9278-4c4f-b2d2-24cf02deba37"},"ecs":{"version":"8.0.0"},"host":{"name":"imp-filebeat-9pwmn"},"log":{"file":{"path":"/var/log/containers/csi-xsky-iscsi-driver-xr88q_xsky-iscsi_driver-registrar-8e467d01b9dcb1dc1a5d2c0cc659a448bde92a4147a3f793f16adebe31f6425e.log"},"offset":940},"stream":"stderr","message":"I0703 14:44:52.599925 3294735 connection.go:184] GRPC error: "}

there are no k8s metadata

botelastic[bot] commented 1 year ago

This issue doesn't have a Team:<team> label.

KevinLiangX commented 1 year ago

Hi how to take the label , i have no idea to attach the label

botelastic[bot] commented 3 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

elastic / beats

Filebeat can't collect the k8s metadata #35982