fluent / fluentd-kubernetes-daemonset

Fluentd daemonset for Kubernetes and it Docker image
Apache License 2.0
1.26k stars 978 forks source link

Set indices format on monthly basis? #1322

Closed riupie closed 2 years ago

riupie commented 2 years ago

I have fluentd deployment on kubernetes using following config:

          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
            value: cri
          - name: FLUENTD_PROMETHEUS_CONF
            value: disable
          - name: FLUENTD_SYSTEMD_CONF
            value: disable
          - name: FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMAT
            value: "%Y-%m-%dT%H:%M:%S.%N%:z"
          - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
            value: "/var/log/containers/fluent*"
          - name: FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME
            value: k8s-staging
          - name: FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX
            value: k8s-staging
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName

Using above config, it will generate indices on daily basis like below:

k8s-staging-2021.11.10
k8s-staging-2021.11.12
k8s-staging-2021.11.13
...

How to set my indices set on monthly basis?

fujimotos commented 2 years ago

You just need to specify your own format to index_name, instead of relying on logstash_format.

Replace the following section:

 - name: FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME
   value: k8s-staging
 - name: FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX
   value: k8s-staging

with this:

 - name: FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT
   value: false
 - name: FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME
   value: "k8s-staging-%Y.%m"
github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days