search

emiller42 / splunk-statsd-backend

A backend plugin for Statsd to output metrics to the Splunk HTTP Event Collector (HEC)
MIT License
11 stars 2 forks source link

Support splunk metrics #10

Closed emiller42 closed 3 years ago

emiller42 commented 3 years ago

@lewismc This PR adds support for Splunk Metrics. Can you review and see if it suits your purposes?

emiller42 commented 3 years ago

Some screenshots of this in action:

Counter data

Counter Data

Gauge data

Gauge data

Set data

Set data

Timer data

Timer data

Example filtering on metric_type

Searching by metric_type
lewismc commented 3 years ago

Testing today @emiller42

lewismc commented 3 years ago

An example of some metrics we are trying to send

3 Jun 18:51:05 - DEBUG: airflow.scheduler_heartbeat:1|c
3 Jun 18:51:05 - DEBUG: airflow.scheduler.critical_section_duration:6.958220|ms
3 Jun 18:51:06 - DEBUG: airflow.scheduler.critical_section_duration:6.989922|ms
3 Jun 18:51:08 - DEBUG: airflow.scheduler.critical_section_duration:7.694453|ms
3 Jun 18:51:09 - DEBUG: airflow.scheduler.critical_section_duration:7.632654|ms
3 Jun 18:51:10 - DEBUG: airflow.scheduler.critical_section_duration:7.189173|ms
3 Jun 18:51:10 - DEBUG: airflow.scheduler.critical_section_duration:6.864361|ms
emiller42 commented 3 years ago

@lewismc How is testing going?

lewismc commented 3 years ago

Hi @emiller42 testing is ongoing. We have been unsuccessful so far but we think it has something to do with SSL configuration. Still investigating and I will provide an update here once we get further. Thanks for checking in...

lewismc commented 3 years ago

@emiller42 I confirm that this PR allows us to send events into Splunk Enterprise v8.0.6 with the following configuration

{
  debug: true,
  dumpMessages: true,
  flushInterval: 10000,
  backends: ['splunk-statsd-backend'],
  splunk: {
    splunkHost: '...',
    splunkPort: 8088,
    useSSL: true,
    strictSSL: false,
    splunkToken: '...',
    index: '...',
    source: 'statsd',
    sourcetype: '_json',
    useMetrics: false
  },
  servers: [
    {
      server: './servers/tcp'
    },
    {
      server: './servers/udp'
    }
  ]
}

This is exactly what we were looking for. Thank you for working so proactively on this PR.

lewismc commented 3 years ago

Some more info, the above config results in metricName instead of metric_name. We are experimenting setting useMetrics: true

emiller42 commented 3 years ago

That sounds right. Behavior should be unchanged when useMetrics: false. The difference between metricName and metric_name only matters when you're sending to a Metrics index. (and thus should have useMetrics: true)

lewismc commented 3 years ago

Everything appears to be in good order @emiller42