fluent / fluentd-kubernetes-daemonset

Fluentd daemonset for Kubernetes and it Docker image
Apache License 2.0
1.26k stars 978 forks source link

Dynamic aliases for indexes #1301

Closed gfrntz closed 3 years ago

gfrntz commented 3 years ago

Hello, I'm creating indexes via record_modifier

    # Use tag as index name prefix by default
    <filter **>
      @type record_modifier
      <record>
        _es_index fluentd-kube-dev-${tag_parts[0]}-${Time.at(time).strftime('%F')}
      </record>
    </filter>

    <filter kubernetes.**>
      @type record_modifier
      <record>
        _es_index fluentd-kube-dev-${record['kubernetes']['namespace_name'] or 'kubernetes'}-${Time.at(time).strftime('%F')}
      </record>
    </filter>

In elasticsearch i have some indexes after:

green  open   fluentd-kube-dev-ingress-nginx-2021-09-15        kgS07gNxTliUydUutNK7jw   1   1      20048            0     15.5mb          7.8mb
green  open   fluentd-kube-dev-kube-oidc-2021-09-15            5ZcQKIFXQCWMD_HLGpBxhQ   1   1      14052            0      4.5mb          2.2mb
green  open   fluentd-kube-dev-kube-system-2021-09-15          ECoRsvJcRNe--g9arVxnyA   1   1     181013            0     97.8mb         48.8mb

After that i'm trying to create ILM for all of those indexes but can't understand how to create this indexes with dynamic field aliases: {"fluentd-kube-dev-kube-system-<<here should be date format>>": "is_write_index: true}} for ilm rollover.

Does anyone have any ideas about this? Or i should create only one index from fluentd with all records and single ilm policy and create separate predefined searches in kibana?

gfrntz commented 3 years ago

This issue isn't related for this project. Close.