splunk / splunk-connect-for-kubernetes

Helm charts associated with kubernetes plug-ins
Apache License 2.0
344 stars 270 forks source link

Fluentd metrics plugin issue - Splunk does not show actual value but just "metric" string for all metrics #786

Closed f6fss closed 2 years ago

f6fss commented 2 years ago

We are using Fluentd metrics plugin to send data into our Splunk environment. I wonder if there is any config/settings issue from our end? Please see our settings at bottom.

What happened: Metric data shows "metric" word (string) instead of actual metrics

What you expected to happen: Metric data shows actual meaningful metric

How to reproduce it (as minimally and precisely as possible): N/A

Anything else we need to know?:

image

Environment:

<match kube.**> @type splunk_hec data_type metric metric_name_key "metric_name" metric_value_key "value" protocol https hec_host "XXXXXXX" hec_port 8088 consume_chunk_on_4xx_errors true hec_token xxxxxx host "ose-dev45-kxzgv-master-2" index "rd_openshift_metrics" source "${tag}" sourcetype "kube:metrics" insecure_ssl true app_name "splunk-kubernetes-metrics" app_version "1.4.15"

hvaghani221 commented 2 years ago

You need to create metric index. Select Metrics index data type.

Refer following image for creating metric index.

image

Provide the metric index in config. For example:

splunk-kubernetes-metrics:
  enabled: true
  splunk:
    hec:
      indexName: metric_index