search

lukemonahan / splunk_modinput_prometheus

A Splunk modular input for ingesting Prometheus metrics
Apache License 2.0
44 stars 23 forks source link

Cannot Ingest Prometheus Data: inputs.conf - recieving errors: btool does not list the stanza [prometheusrw]) #33

Open Codex787 opened 3 months ago

Codex787 commented 3 months ago

I'm encountering an issue with ingesting data from a Prometheus remote_write_agent into Splunk Enterprise – this solution utilises the ‘Prometheus Metrics for Splunk and is within a Test Environment.

Problem Summary: Despite ensuring that the 'inputs.conf' file matches the configuration specifications defined in the 'inputs.conf.spec' file, the Prometheus data is not being ingested and I am receiving errors, e.g port: Not found in "btool" output (btool does not list the stanza [prometheusrw]) when viewing the inputs.conf file in the config explorer application.

Details:

Splunk Version: Splunk Enterprise 9.2 (Trial License)

Operating System: Ubuntu 22.04

Splunk Application: Prometheus Metrics for Splunk (Latest Version 1.0.1)

inputs.conf.spec

/opt/splunk/etc/apps/modinput_prometheus/README/inputs.conf.spec

Screenshot 2024-06-25 at 11 30 03 am

As seen in image, the inputs.conf.spec file states there is a port and maxClients configuration parameters.

In the inputs.conf I updated the /opt/splunk/etc/apps/modinput_prometheus/local/inputs.conf file to include the details below which meet the required formatting above:

Screenshot 2024-06-25 at 11 32 41 am

The inputs.conf file was saved, and the Splunk Server rebooted. After rebooting the input.conf was checked to ensure the config specification where being accepted using the Config Explorer App –

These errors where received for the following configuration parameters:

Screenshot 2024-06-25 at 11 31 20 am

However, other configuration parameters such as index, sourcetype & whitelist Returned: 'Found in "btool" output. Exists in spec file (Stanza=[prometheusrw]) - and were accepted by Splunk.

Screenshot 2024-06-25 at 11 33 13 am

For some unknown reason, Splunk is not recognising some of the configuration parameters above that are listed within the inputs.conf.spec file, even when formatted accordingly.

Other Information:

Prometheus remote-write-exporter details:

Screenshot 2024-06-25 at 11 33 34 am

Splunk Index: skyline_prometheus_metrics

Network007_8-1718959016503.pn

lukemonahan commented 3 months ago

Hi:

This looks like a limitation of btool to be able to show the global configuration ([prometheusrw]) properly. At a test, it can show the specific input configuration ([prometheusrw://x]) correctly.

Btool can properly show system global specs (e.g. [http]) properly, and it has the same inputs.conf.spec structure as prometheusrw. So at the moment I'm unsure what is wrong.

This should not, however, affect metrics being received properly.

Can you see port 8098 being opened on your host with netstat?

Codex787 commented 3 months ago

Hi @lukemonahan thanks for assisting so quickly.

Ran then sudo ss -tulpn command and confirmed that port 8098 is In State Listen.

Is there trouble shooting steps you can advise to determine the cause of the data not being ingested?

For some context I having utilising:https://community.checkpoint.com/t5/OpenTelemetry-Skyline/how-to-ingest-skyline-data-into-splunk/td-p/181936 to set up configuration.