FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT not working

[dbaumgarten]

Hello there,

I am using the v1.9 fluentd-elasticsearch7 daemonset and encountered a problem with the name of the index the logs are written to.

Per default the logs are written to an index called "logstash\<date>. But I want all logs to go to an index of a specific name.

According to the docs of the fluentd-elasticsearch plugin I have to set logstash_format to false and index_name to the name I want.

The docker image provides env-vars for these config values ('FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT and FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME), but when I set these env vars as follows:

- env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: 3ef313bba04646089393a373a4922070.kibana.mlaas.prod.mls.projects.de-wob-3.cloud.vwgroup.com
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9243"
        - name: FLUENT_ELASTICSEARCH_SCHEME
          value: https
        - name: FLUENT_ELASTICSEARCH_SSL_VERIFY
          value: "true"
        - name: FLUENT_ELASTICSEARCH_SSL_VERSION
          value: TLSv1_2
        - name: FLUENT_ELASTICSEARCH_USER
          value: caas_beat_writer
        - name: FLUENT_ELASTICSEARCH_PASSWORD
          value: ********************
        - name: FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME
          value: caas-gks-dev
        - name: SSL_CERT_DIR
          value: /etc/ssl/certs
        - name: FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT
          value: "false"

the logs still go to logstash-\<date>.

Interestingly the startup log shows the used config and it contains the lines:

 reload_on_failure true
 log_es_400_reason false
 logstash_prefix "logstash"
 logstash_format false
 index_name "caas-gks-dev"
 type_name "fluentd"

It seems the env vars are correctly inserted into the config, but the logs still go to the wrong index. I am absolutely out of ideas and would appreciate some help.

[dbaumgarten]

I found the problem. If logstash_format is false and include_timestamp is false (the default) no timestamps are included in the log-data and the logs therefore do not show up in Kibana. Once setting include_timestamp to true everything works fine.

[cotjoey]

@dbaumgarten - I still can't get this working with the values you specified to use above. How did you find the startup logs?