uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 310 forks source link

Dynamic aliases field for indexes #918

Open gfrntz opened 3 years ago

gfrntz commented 3 years ago

(check apply)

Problem

Hello, I'm creating indexes via record_modifier

    # Use tag as index name prefix by default
    <filter **>
      @type record_modifier
      <record>
        _es_index fluentd-kube-dev-${tag_parts[0]}-${Time.at(time).strftime('%F')}
      </record>
    </filter>

    <filter kubernetes.**>
      @type record_modifier
      <record>
        _es_index fluentd-kube-dev-${record['kubernetes']['namespace_name'] or 'kubernetes'}-${Time.at(time).strftime('%F')}
      </record>
    </filter>

In elasticsearch i have some indexes after:

green  open   fluentd-kube-dev-ingress-nginx-2021-09-15        kgS07gNxTliUydUutNK7jw   1   1      20048            0     15.5mb          7.8mb
green  open   fluentd-kube-dev-kube-oidc-2021-09-15            5ZcQKIFXQCWMD_HLGpBxhQ   1   1      14052            0      4.5mb          2.2mb
green  open   fluentd-kube-dev-kube-system-2021-09-15          ECoRsvJcRNe--g9arVxnyA   1   1     181013            0     97.8mb         48.8mb

After that i'm trying to create ILM for all of those indexes but can't understand how to create this indexes with dynamic field aliases: {"fluentd-kube-dev-kube-system-<>": "is_write_index: true}} for ilm rollover.

Does anyone have any ideas about this? Or i should create only one index from fluentd with all records and single ilm policy and create separate predefined searches in kibana? ...

Expected Behavior or What you need to ask

Index creation with dynamic aliases with index name. ...