template_file on ElasticSearch Issues

[jaimehrubiks]

UPDATED: Check my comment bellow

Describe the bug: The documentation says the following:

template_file	*secret.Secret	No	-	The path to the file containing the template to install.Secret

But instead of requesting a string, it requests a secret. My understanding here is that I should save the template as a json file within a secret, so I configured it this way:

kubectl create secret generic index-template --from-file=index-template=./logs/logging-operator/index-template.json  --dry-run -o yaml |  kubectl -n logging  apply -f -

    ilm_policy_id: logs-delete
    ilm_policy: '{ "policy": { "phases": { "hot": { "actions": {} }, "delete": { "min_age": "31d", "actions": { "delete": {} } } } } }'
    ilm_policy_overwrite: true
    template_file: 
      valueFrom:
        secretKeyRef:
          name: index-template
          key: index-template
    template_name: index-template
    template_overwrite: true

Expected behaviour: Given that I am asked to put a secret, I expected the file to be saved in the system and that template_file variable would be translated into the fluend configuration a path to that file.

Instead, I see that the content of the secret is added to the variable as follows:

      template_file "{\"order\":0,\"index_patterns\":[\"logstash-*\"],\"settings\":{\"index\":{\"lifecycle\":{\"name\":\"logs-delete\"},\"number_of_shards\":\"1\",\"number_of_replicas\":\"1\"}},\"mappings\":{\"_doc\":{\"_routing\":{\"required\":false},\"numeric_detection\":false,\"dynamic_date_formats\":[\"strict_date_optional_time\",\"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z\"],\"_meta\":{},\"dynamic\":true,\"_source\":{\"excludes\":[],\"includes\":[],\"enabled\":true},\"dynamic_templates\":[],\"date_detection\":true,\"properties\":{}}}}"

Which I guess it is wrong, as it doesn't work for me, and also, it doesn't follow the definition (it is not a path)

Additional context: For such a case, I would expect that either:

a) template_file be a path, and then mount the file somehow (Although, I can't find any way to mount a file on fluentd, I'm checking the "Loggings" fluentd spec, and don't see anything)

b) template_file be a secret/configmap, and then add the actual file as the content of the secret, as I was trying.

In any case, probably I am missing something, so it would be very helpful if someone could explain how to make it work.

Thanks for your time.

[tarokkk]

To use secret as files you should use the mountFrom as described here https://banzaicloud.com/docs/one-eye/logging-operator/plugins/outputs/secret/#define-secret-mount instead of valueFrom.

[jaimehrubiks]

I had read the link before but as mountFrom in secret does not exist in regular kubernetes, my mind missed it. Thanks for pointing it out! I had automated curl-ing to ES instead meanwhile.

[jaimehrubiks]

I will reopen as I still have some issues.

First issue (just documentation issue), in the guide (https://banzaicloud.com/docs/one-eye/logging-operator/plugins/outputs/secret/#define-secret-mount) we see the following:

tls_cert_path:
  valueFrom:
    mountFrom:
      name: <kubernetes-secret-name>
      key: <kubernetes-secret-key>

However, I think it is wrong, and the proper syntax according to the CRD is:

tls_cert_path:
  mountFrom:
    secretKeyRef:
      name: <kubernetes-secret-name>
      key: <kubernetes-secret-key>

The second and more important issue, is that, whenever I apply that change, the log shipping stops working. There are no errors in the logs from fluentbit or fluentd. I also checked the logs on /fluentd/log/out (with debug mode enabled for logging fluentd) and there are no errors there either.

Changes applied to ClusterOutput that make it not work:

    ilm_policy_id: ccc-index-lifecycle-policy
    ilm_policy: '{ "policy": { "phases": { "hot": { "actions": { "set_priority": { "priority": 50 } } }, "warm": { "min_age": "15d", "actions": { "allocate": { "number_of_replicas": 0 }, "readonly": {}, "set_priority": { "priority": 25 } } }, "delete": { "min_age": "30d", "actions": { "delete": {} } } } }}'
    ilm_policy_overwrite: true
    template_file: 
      mountFrom:
        secretKeyRef:
          name: ccc-index-template
          key: ccc-index-template
    template_name: ccc-index-template
    template_overwrite: true

Changes that show in /fluentd/app-config/fluentd.conf

<label @87ad0b6cb7e2bdeff9bf1>
  <match **>
    @type elasticsearch
    @id clusterflow:logging:clusterflow-namespaces-elasticsearch:clusteroutput:logging:clusteroutput-elasticsearch
    exception_backup true
    fail_on_putting_template_retry_exceed true
    host elasticsearch-es-http.logging.svc.cluster.local
    ilm_policy { "policy": { "phases": { "hot": { "actions": { "set_priority": { "priority": 50 } } }, "warm": { "min_age": "15d", "actions": { "allocate": { "number_of_replicas": 0 }, "readonly": {}, "set_priority": { "priority": 25 } } }, "delete": { "min_age": "30d", "actions": { "delete": {} } } } }}
    ilm_policy_id ccc-index-lifecycle-policy
    ilm_policy_overwrite true
    logstash_format true
    password .....
    port 9200
    reconnect_on_error true
    reload_connections false
    reload_on_failure true
    request_timeout 15s
    scheme https
    ssl_verify false
    ssl_version TLSv1_2
    template_file /fluentd/secret/logging-ccc-index-template-ccc-index-template
    template_name ccc-index-template
    template_overwrite true
    user elastic
    utc_index true
    verify_es_version_at_startup true
    <buffer tag,time>
      @type file
      chunk_limit_size 16MB
      flush_interval 5s
      flush_mode interval
      flush_thread_count 8
      path /buffers/clusterflow:logging:clusterflow-namespaces-elasticsearch:clusteroutput:logging:clusteroutput-elasticsearch.*.buffer
      retry_forever true
      timekey 1m
      timekey_use_utc true
      timekey_wait 30s
    </buffer>
  </match>
</label>

I can confirm that /fluentd/secret/logging-ccc-index-template-ccc-index-template is properly mounted on the pod with the expected content

Also, I manually checked that the ilm and index-template are correct (I did a manual PUT to ES and it works fine with my other configuration of logging-operator without those)

[maxisam]

Why this is close? I have exactly the same issue. I can see it in /fluentd/app-config/fluentd.conf

And I have no log

[joao-dantas]

I am facing the same issue here too.