Filebeat is not creating indices for each pod

[Nandeeshbb]

Please help on this and stuck in this from last 5 days and not able to create index for each pod in elastic search using filebeat.

Elastic search version: 7.10.1 AWS Filebeat version: 7.12.1

Filebeat configuration file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: filebeat
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/lib/kubelet/pods/*_communication-service_*
      tags: ["communication-service"]
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/lib/kubelet/pods/"
                resource_type: "pod"
            default_indexers.enabled: false
            default_matchers.enabled: false
            indexers:
              - pod_name:
    - type: container
      paths:
        - /var/lib/kubelet/pods/*_bgv-dl-verification_*
      tags: ["bgv-dl-verification"]
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/lib/kubelet/pods/"
                resource_type: "pod"
            default_indexers.enabled: false
            default_matchers.enabled: false
            indexers:
              - pod_name:

 output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:443}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      protocol: "https"
      indices:
      - index: "logs-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"
        when.contains:
          tags: "communication-service"
      - index: "logs-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"
        when.contains:
          tags: "bgv-dl-verification"
      - index: "logs-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"
        when.contains:
          tags: "reports-generator"
      - index: "logs-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"
        when.contains:

Help would be very much appreciated..

And the file beat logs says index is created but dont see any index with name of the pod

DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: logging/fluentd-8cvhz {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.712Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index logging/fluentd-8cvhz for pod logging/fluentd-8cvhz {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.712Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/ebs-csi-node-fkj9g {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.712Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index kube-system/ebs-csi-node-fkj9g for pod kube-system/ebs-csi-node-fkj9g {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.712Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: dev/insurance-service-dev-deployment-649c79c86d-nbrvw {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.718Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index dev/insurance-service-dev-deployment-649c79c86d-nbrvw for pod dev/insurance-service-dev-deployment-649c79c86d-nbrvw {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.718Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/csi-secrets-store-provider-aws-tst2r {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.718Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index kube-system/csi-secrets-store-provider-aws-tst2r for pod kube-system/csi-secrets-store-provider-aws-tst2r {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.719Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: uat/bgv-voter-verification-bot-uat-deployment-d9fc798f-8r4rt {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.722Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index uat/bgv-voter-verification-bot-uat-deployment-d9fc798f-8r4rt for pod uat/bgv-voter-verification-bot-uat-deployment-d9fc798f-8r4rt {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.722Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: dev/bgv-aadhaar-verification-bot-dev-deployment-669c5b8c58-x2mgr {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.725Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index dev/bgv-aadhaar-verification-bot-dev-deployment-669c5b8c58-x2mgr for pod dev/bgv-aadhaar-verification-bot-dev-deployment-669c5b8c58-x2mgr {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.725Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: fluentd-bit/fluent-bit-kd4f2 {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.726Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index fluentd-bit/fluent-bit-kd4f2 for pod fluentd-bit/fluent-bit-kd4f2 {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.726Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: filebeat/filebeat-5g84z {"libbeat.processor": "add_kubernetes_metadata"} 2024-07-05T13:45:46.726Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index filebeat/filebeat-5g84z for pod filebeat/filebeat-5g84z {"libbeat.processor": "add_kubernetes_metadata"}

[botelastic[bot]]

This issue doesn't have a Team:<team> label.

[VihasMakwana]

@Nandeeshbb Hi! This is more related to add_kubernetes_metadata processor.

If I'm not wrong, this wouldn't create a new index in ElasticSearch but a new index (i.e. identifier) in local memory cache. The concept of index in terms of add_kubernetes_metadata can be found here.

The logs Created index xxx means that it has internally created an index which will be used to extract kubernetes pod metadata.

[Nandeeshbb]

Thanks @VihasMakwana ,,but I have followed the same , but not able to create index per pod name .. Do you have any idea?

[Nandeeshbb]

Any help here?