Fluentd elasticsearch - ILM Policy

[ghost]

Is this a request for help?: yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes: not of interest

Which chart in which version: v7.9.0 - https://github.com/elastic/helm-charts/tree/master/elasticsearch v3.0.4 - https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch

What happened: i have tried to use ilm and im getting an error that ilm_policy and ilm_policies dont work together. 2020-08-20T09:52:13.21790686Z 2020-08-20 09:52:13 +0000 [error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="You can't specify ilm_policy and ilm_policies at the same tim │ │ e"

What you expected to happen: When i leave one of the both empty it should not be honored in the daemonset

How to reproduce it (as minimally and precisely as possible):

elasticsearch:
  hosts: ["<yourServiceNameHere>"]
  indexName: "fluentd"
  logstash:
    enabled: false
  ilm:
    enabled: true
    policy_id: fluentd-30d-delete-policy
    policy:
      phases:
        delete:
          min_age: 30d
          actions:
            delete:
              delete_searchable_snapshot: true
    policy_overwrite: true

Anything else we need to know: I can fix that problem when i comment out the lines in the daemonSet:

#        - name: ILM_POLICIES
#          value: map[]

So this is a simple fix in the helm chart (if not empty) :)

[ghost]

It works also if i overwrite policies to: policies: instead of policies: {} maybe that is an even easier fix. To make it as default.

[ghost]

In general the policy generation dont work. There are no errors but in elasticsearch the policy is missing.

[monotek]

The ilm stuff was a third party contribution in https://github.com/kiwigrid/helm-charts/pull/380

Maybe ask the original contributor @niklaskhf for help.

[niklaskhf]

@monotek I remember that it was necessary to add a template file as ConfigMap. Since I didn’t have time to implement this in a configurable way, I forked this repo and added a default fluentd.json ConfigMap and the required environment variables https://github.com/niklaskhf/helm-charts/tree/master/charts/fluentd-elasticsearch/templates

[monotek]

@niklaskhf So the ilm part of this chart does not work at all?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[monotek]

Chart was deprecated and moved to: https://github.com/kokuwaio/helm-charts/tree/main/charts/fluentd-elasticsearch