[BUG] Variable in Index-Name

[sevenval-admins]

Describe the bug Hi everybody, I know that this theme was already discussed many times, but I cannot find something relating to my issue. What happen to me is that every first Index of the day the second variable (kubernetes.namespace) is taken as literal and not her real value. All subsequently created indices correctly report the exact value of the variable.

To Reproduce

My config:
apiVersion: v1
data:
  pipelines.yml: |-
    - pipeline.id: beats-server
      config.string: |
        input { beats {  port => 5044  }}
        output {
          if [kubernetes][cluster_name] == "my-cluster {
            pipeline { send_to => ["opensearch-stack"] ensure_delivery => false }
          }
        }

    - pipeline.id: opensearch
      pipeline.batch.size: 35
      pipeline.batch.delay: 10
      config.string: |
        input  { pipeline { address => "opensearch-stack" }}
        output {
          opensearch { hosts => ["https://my-opensearch:443"]
                       index => "[kubernetes][cluster_name]-%{[kubernetes][namespace]}-%{+yyyy.MM.dd}"
                       user => "logstash"
                       password => "pwd" }

Expected behavior Index: my-cluster-mynamespace-2022-05-12

Instead is Index: my-cluster-%{[kubernetes][namespace]}-2022-05-12 As I already told it, this happen just for the first index of the day, all the subsequently has the right namespace.

Host/Environment (please complete the following information): Logstash run as a k8s statefulset on a CentOS7 VM. image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.3

Thanks in advance to everyone who will have a look into it.

[acrispim]

I went through the same difficulties. The solution I found was first still in the filter create the field that will receive the dynamic name that will compose the index. I feel that there are still some problems in the interpretation of variables, mainly in the logical operators. Below is an example of how it works for me, where I wanted to create a different index name depending on the name of the applications.

input { tcp { port => 5514 }}

filter {
    mutate { add_field => { "target_index" => "logstash-base" } }

    if [metadata_app_name] =~ /.+/ {
        if [metadata_app_name] =~ /^(app_name_1|app_name_2|app_name_2)/ {
            mutate { update => { "target_index" => "logstash-context-1" }}
        }
    } else {
        mutate { update => {"target_index" => "logstash-notracking" }}
    }
}
output {
    opensearch {
        hosts => ["https://opensearch-node1:9200"]
        index => "%{target_index}-%{+YYYY.MM.dd.HH}"
        user => "logstash"
        password => "${LOGSTASH_PASSWORD}"
        ssl => true
        ssl_certificate_verification => false
    }
}