lukemonahan / splunk_modinput_prometheus

A Splunk modular input for ingesting Prometheus metrics
Apache License 2.0
46 stars 23 forks source link

Splunk Base App #31

Open rafeeqsid opened 1 year ago

rafeeqsid commented 1 year ago

Working on to integrate metric forwarding via Prometheus remote-write to Splunk HF.

Current Setup:- Splunk HF is hosted on HTTPS. As below document suggest for HTTP, Than how to use HTTPS for remote-write?

Using Below configuration in Splunk-HF as reference to https://splunkbase.splunk.com/app/4077

[prometheusrw] port = 8098 maxClients = 10 disabled = 0

[prometheusrw://testing] bearerToken = ABC123 index = prometheus whitelist = * sourcetype = prometheus:metric disabled = 0

And not able to justify the team that how to configure bearer-token in prometheus, As for this no document is available. And also how to use HTTPS in url as splunk hosted over HTTPS

remote_write:

lukemonahantnt commented 1 year ago

To enable HTTPs on the Splunk side, please use the "enableTLS" and other related options in inputs.conf.

https://github.com/lukemonahan/splunk_modinput_prometheus/blob/master/modinput_prometheus/README/inputs.conf.spec#L11-L18

On the Prometheus side it should be just using a "https" URL, and ensuring the tls_config is set to trust the cert you have provided within Splunk.

It appears like Prometheus has changed config for bearer tokens in more recent versions. From reading docs, it seems this might be correct, but I haven't tested it. Will get onto this and update the Splunk add-on docs when I can.

authorization:
    type: "Bearer"
    credentials: "ABC123"

tls_config and new Bearer token options described here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

rafeeqsid commented 1 year ago

Yes above suggestion was very helpful.

One more query for below inputs.conf for remote-write integration the index(prometheus) will be metric type or Event type?

[prometheusrw://testing] bearerToken = ABC123 index = prometheus whitelist = * sourcetype = prometheus:metric disabled = 0

lukemonahantnt commented 1 year ago

Use a metrics type index in all cases.