search

lukemonahan / splunk_modinput_prometheus

A Splunk modular input for ingesting Prometheus metrics
Apache License 2.0
46 stars 23 forks source link

After upgrading to V1.0.1 receiving client timeout error #34

Open siddharthajuprod07 opened 2 months ago

siddharthajuprod07 commented 2 months ago

Hi, We have seen an issue where the older version of prometheus app is working fine but the CPU usage was very high (80%). So we upgarded the app to recent version but after the metrices stopped flowing to splunk and we are receiving client timeout error. Can someone please throw some light on this.

lukemonahantnt commented 2 months ago

I don't believe any changes in this time period should be increasing or decreasing CPU usage in any manner.

For the rest of it, I'm not sure if I can help without more logs or configuration details. Is there anything you can provide here?

siddharthajuprod07 commented 2 months ago

@lukemonahantnt , Thank you for your reply, please find the log errors which we are seeing, 8/22/24 3:33:30.562 PM
08-22-2024 15:33:30.562 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:33:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:32:30.601 PM
08-22-2024 15:32:30.601 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:32:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:31:30.486 PM
08-22-2024 15:31:30.486 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:31:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:30:30.503 PM
08-22-2024 15:30:30.503 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:30:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:29:30.501 PM
08-22-2024 15:29:30.501 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:29:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:28:30.455 PM
08-22-2024 15:28:30.455 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:28:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:27:34.982 PM
08-22-2024 15:27:34.982 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheusrw" 2024/08/22 14:27:34 Prometheus input globally disabled host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:27:31.005 PM
08-22-2024 15:27:31.005 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:27:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

again many thanks for your time.

siddharthajuprod07 commented 2 months ago

This is the current config we have

[prometheus://test] URI = http://host:port/federate match = {name=~"..*"} index = sourcetype = interval = 60 disabled = 0

end of change