Issue with customConfig Vector Templating

[csongpaxos]

I see the below customConfig example in the README. However, for some reason when I try to use that same syntax, I am getting the issue that the s3 tag is malformed, because it thinks the tag itself is {{ container_name }}, not the actual value of the container_name (which is a field in the log). I've tried {{ .container_name }} and some others but to no avail. Not sure if I am missing something here.

README example:

customConfig:
  #...
  sinks:
    loki:
      #...
      labels:
        foo: bar
        host: |-
          {{ print "{{ host }}" }}
        source: |-
          {{ print "{{ source_type }}" }}

My values file contains:

  customConfig:
    data_dir: /var/lib/vector
    sources:
      kubernetes_logs:
        type: kubernetes_logs
    sinks:
      s3:
        type: aws_s3
       ...
       ...
        tags:
          splunk_index: test
          splunk_sourcetype: |-
            {{ print "{{ container_name }}" }}

[jszwedko]

Hi @csongpaxos !

Unfortunately the aws_s3 sink tags configuration doesn't support templating (https://vector.dev/docs/reference/configuration/sinks/aws_s3/#tags). Only fields that are marked as "template" in the documentation do. For example the labels option on the loki sink: https://vector.dev/docs/reference/configuration/sinks/loki/#labels.

You can open a feature request to support templating on the tags option if you like: https://github.com/vectordotdev/vector/issues/new?assignees=&labels=type%3A+feature&projects=&template=feature.yml

Hope this helps!

[autokilla47]

I had a similar situation and problem. I form the name of the index from the label specified for the pod.

customConfig:
  sources:
    kubernetes:
      type: kubernetes_logs
  transforms:
    kubernetes_transform:
      type: remap
      inputs:
        - kubernetes
      source: |
        if ! exists(.kubernetes.pod_labels.es_index_name) {
          .kubernetes.pod_labels.es_index_name = "other"
        }
        if exists(.kubernetes.pod_labels.es_index_period) && downcase!(.kubernetes.pod_labels.es_index_period) == "day" {
          .kubernetes.pod_labels.es_index_period_format = format_timestamp!(now(),"%Y.%m.%d")
        } else if exists(.kubernetes.pod_labels.es_index_period) && downcase!(.kubernetes.pod_labels.es_index_period) == "month" {
          .kubernetes.pod_labels.es_index_period_format = format_timestamp!(now(),"%Y.%m")
        } else {
          .kubernetes.pod_labels.es_index_period_format = format_timestamp!(now(),"%Y.%m")
        }
  sinks:
    opensearch:
      type: elasticsearch
      inputs:
        - kubernetes_transform
      endpoints: [ "https://10.10.10.10:9200" ]
      bulk:   <--- tried this
        index: "{{ .kubernetes.pod_labels.es_index_name }}-{{ .kubernetes.pod_labels.es_index_period_format }}"
      bulk:   <--- but work like this
        index: |-
           {{ print "{{ .es_index_name }}-{{ .es_index_period_format }}" }}