jti-input: Fluentd.conf tag (indexes) more fields than expected

[henriklb]

Hi,

I'm running latest open-nti with JTI. Issuing a show tag keys in InfluxDB outputs a lot of tags, more than expected by looking at input-jti's fluent.conf file (see below).

However, I cannot find/understand the code that turn certain "fields" into indexed tags. According to the the fluent.conf file only a few of these fields should be turned into tags.

For instance, where is the code (in fluent.conf?) that specifies that interface_stats.if_name should be a tag? What am I missing?

Thank you very much

name: juniperNetworks.cpu_memory_util_ext tagKey

_seq device utilization.application_utilization.name utilization.name

name: juniperNetworks.fabricMessageExt tagKey

device edges.class_stats.priority edges.destination_pfe edges.destination_slot edges.destination_type edges.source_pfe edges.source_slot edges.source_type location

name: juniperNetworks.inline_jflow_stats_ext tagKey

_seq device

name: juniperNetworks.jnprLogicalInterfaceExt tagKey

device interface_info.if_name interface_info.parent_ae_name interface_info.snmp_if_index

name: juniperNetworks.jnpr_firewall_ext tagKey

device firewall_stats.counter_stats.name firewall_stats.filter_name firewall_stats.memory_usage.name firewall_stats.policer_stats.name

name: juniperNetworks.jnpr_interface_ext tagKey

device interface_stats.egress_queue_info.queue_number interface_stats.if_name

name: juniperNetworks.jnpr_optics_ext tagKey

Optics_diag.if_name Optics_diag.optics_diag_stats.optics_lane_diag_stats.lane_number device

[psagrera]

Hi, Fluentd UDP sensor plugin retrieve key information embedded in proto files and create data with these information. In case you want to (re-)define those keys you have to specifically define it into the fluent.conf

i.e (In that case it will take keys from .proto files + defined manually)

    <match juniperNetworks.cpu_memory_util_ext>
        [.....]
            tag_keys ["device","utilization.application_utilization.name","utilization.name"]
            tag_keys_field key_fields
        [.....]
    </match>

or (in that case it will take key values only from the proto file)

    <match juniperNetworks.**>

    {% if OUTPUT_INFLUXDB == 'true' %}
            tag_keys ["device"]
            tag_keys_field key_fields
    {% endif %}

For example if you take a look at .proto file of jnpr_interface_ext , you will see a couple of key fields:

  //
 // Interface information
//
 message InterfaceInfos {
    // Interface name, e.g., xe-0/0/0
       required string if_name                        = 1 [(telemetry_options).is_key = true]; 

 //
 // Interface queue statistics
//
  message QueueStats {
   // Queue number
      optional uint32 queue_number                  = 1 [(telemetry_options).is_key = true];   

Regards

[henriklb]

Thanks very much for the quick and informative reply.

I solved the issue. I figured I cloned the wrong branch of the fluentd-plugin repo. I had to specify the "with_keys" branch.