Cannot build Data Stream Name dynamically using placeholder templates

[marcin-brzozowski]

I'm trying to build data_stream_name dynamically using Kubernetes metadata. I'm observing the following behavior, fluentd creates datastream with name fluentd.${$.kubernetes.container_name} instead of substituting values for the {$.kubernetes.container_name} placeholder.

Supposedly, it should be working (see #869 ). Is something off in my configuration or there's a bug?

I've also pulled out one of the log messages to the stdout, to make sure that the kubernetes labels are there, and so they are:

{
    "stream": "stdout",
    "logtag": "F",
    "message": "I removed this msg...",
    "time": "2022-01-18T12:59:48.0980824Z",
    "docker": {
        "container_id": "60d7f5efa369b90aa5d816f20db2099fa1aa11b65ee448ad649fd56992b80fc1"
    },
    "kubernetes": {
        "container_name": "kibana",
        "namespace_name": "default",
        "pod_name": "kibana-kibana-79855ccc65-qxx57",
        "container_image": "docker.elastic.co/kibana/kibana:7.14.0",
        "container_image_id": "docker.elastic.co/kibana/kibana@sha256:a1c80a2b22f6c9a93a089c8b983078d482e6dad5e693c64e84b491afd0e90f53",
        "pod_id": "6c2c78ed-3bf0-49be-9760-d135b65038ac",
        "pod_ip": "10.42.0.10",
        "host": "k3d-hello-server-0",
        "labels": {
            "app": "kibana",
            "pod-template-hash": "79855ccc65",
            "release": "kibana"
        },
        "master_url": "https://10.43.0.1:443/api",
        "namespace_id": "ecb2a711-800c-4a72-bfbe-a97bb0ebf936",
        "namespace_labels": {
            "kubernetes_io/metadata_name": "default"
        }
    }
}

Using Fluentd and ES plugin versions

• Kubernetes, using fluent/fluentd-kubernetes-daemonset:v1.14.3-debian-elasticsearch7-1.0
• Fluentd: fluentd 1.14.3
• ES 7.14.0

• Fluentd Configuration

  fileConfigs:
  01_sources.conf: |-
  <source>
    @type tail
    @id in_tail_container_logs
    @label @KUBERNETES
    path /var/log/containers/*.log
    pos_file /var/log/fluentd-containers.log.pos
    tag kubernetes.*
    read_from_head true
    <parse>
      @type cri
    </parse>
    emit_unmatched_lines true
  </source>
  
  02_filters.conf: |-
  <label @KUBERNETES>
    <match kubernetes.var.log.containers.fluentd**>
      @type relabel
      @label @FLUENT_LOG
    </match>
  
    # <match kubernetes.var.log.containers.**_kube-system_**>
    #   @type null
    #   @id ignore_kube_system_logs
    # </match>
  
    <filter kubernetes.**>
      @type kubernetes_metadata
      @id filter_kube_metadata
      skip_labels false
      skip_container_metadata false
      skip_namespace_metadata false
      skip_master_url false
    </filter>
  
    <match **>
      @type relabel
      @label @DISPATCH
    </match>
  </label>
  
  03_dispatch.conf: |-
  <label @DISPATCH>
    <filter **>
      @type prometheus
      <metric>
        name fluentd_input_status_num_records_total
        type counter
        desc The total number of incoming records
        <labels>
          tag ${tag}
          hostname ${hostname}
        </labels>
      </metric>
    </filter>
  
    <match **>
      @type relabel
      @label @OUTPUT
    </match>
  </label>
  
  04_outputs.conf: |-
  <label @OUTPUT>
    <match **>
      @type                elasticsearch_data_stream
      host                 elasticsearch-master
      port                 9200
      suppress_type_name   true
  
      data_stream_name     fluentd.${$.kubernetes.container_name}
      data_stream_ilm_name foo
  
      <buffer _index, $.kubernetes.container_name>
        flush_thread_count 5
        @type              file
        path               /var/log/fluentd/buffer/
        timekey            10m
        chunk_limit_size   10m
        total_limit_size   10g
        flush_mode         interval
        flush_interval     1m
        overflow_action    drop_oldest_chunk
        retry_type         exponential_backoff
        retry_wait         5s
        retry_max_interval 60s
        retry_randomize    true
        retry_forever      true
      </buffer>
    </match>
  </label>

• Gems:

  
  *** LOCAL GEMS ***

addressable (2.8.0) bigdecimal (default: 1.4.1) bundler (2.2.24, default: 1.17.2) cmath (default: 1.0.0) concurrent-ruby (1.1.9) cool.io (1.7.1) csv (default: 3.0.9) date (default: 2.0.2) dbm (default: 1.0.0) domain_name (0.5.20190701) e2mmap (default: 0.1.0) elasticsearch (7.15.0) elasticsearch-api (7.15.0) elasticsearch-transport (7.15.0) elasticsearch-xpack (7.15.0) etc (default: 1.0.1) excon (0.88.0) faraday (1.8.0) faraday-em_http (1.0.0) faraday-em_synchrony (1.0.0) faraday-excon (1.1.0) faraday-httpclient (1.0.1) faraday-net_http (1.0.1) faraday-net_http_persistent (1.2.0) faraday-patron (1.0.0) faraday-rack (1.0.0) fcntl (default: 1.0.0) ffi (1.15.4) ffi-compiler (1.0.1) fiddle (default: 1.0.0) fileutils (default: 1.1.0) fluent-config-regexp-type (1.0.0) fluent-plugin-concat (2.5.0) fluent-plugin-dedot_filter (1.0.0) fluent-plugin-detect-exceptions (0.0.14) fluent-plugin-elasticsearch (5.1.4) fluent-plugin-grok-parser (2.6.2) fluent-plugin-json-in-json-2 (1.0.2) fluent-plugin-kubernetes_metadata_filter (2.9.2) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-parser-cri (0.1.1) fluent-plugin-prometheus (2.0.2) fluent-plugin-record-modifier (2.1.0) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-systemd (1.0.5) fluentd (1.14.3) forwardable (default: 1.2.0) gdbm (default: 2.0.0) http (4.4.1) http-accept (1.7.0) http-cookie (1.0.4) http-form_data (2.3.0) http-parser (1.2.3) http_parser.rb (0.8.0) io-console (default: 0.4.7) ipaddr (default: 1.2.2) irb (default: 1.0.0) json (default: 2.1.0) jsonpath (1.1.0) kubeclient (4.9.2) logger (default: 1.3.0) lru_redux (1.1.0) matrix (default: 0.1.0) mime-types (3.4.1) mime-types-data (3.2021.1115) msgpack (1.4.2) multi_json (1.15.0) multipart-post (2.1.1) mutex_m (default: 0.1.0) netrc (0.11.0) oj (3.11.0) openssl (default: 2.1.2) ostruct (default: 0.1.0) prime (default: 0.1.0) prometheus-client (2.1.0) psych (default: 3.1.0) public_suffix (4.0.6) rake (13.0.6) rdoc (default: 6.1.2.1) recursive-open-struct (1.1.3) rest-client (2.1.0) rexml (default: 3.1.9.1) rss (default: 0.2.7) ruby2_keywords (0.0.5) scanf (default: 1.0.0) sdbm (default: 1.0.0) serverengine (2.2.4) shell (default: 0.7) sigdump (0.2.4) stringio (default: 0.0.2) strptime (0.2.5) strscan (default: 1.0.0) sync (default: 0.5.0) systemd-journal (1.4.2) thwait (default: 0.1.0) tracer (default: 0.1.0) tzinfo (2.0.4) tzinfo-data (1.2021.5) unf (0.1.4) unf_ext (0.0.8) webrick (1.7.0, default: 1.4.4) yajl-ruby (1.4.1) zlib (default: 1.0.0)

[ayaranga]

@marcin-brzozowski I had got it working with the similar configuration except that I used <buffer tag, $.kubernetes.container_name>

[aarongorka]

I was not able to get it working even with the $.myfield syntax in the buffer; however I did find that $tag interpolates just fine 🤷