Hello.
We are running blackbox_exporter as a docker container (official image) to probe some of our infrastructure with icmp module.
For some reason after few days/weeks of constant probing blackbox exporter starting to returned probe_success metric with value 0 for some targets.
If I curl blackbox_exporter with this target and &debug=true parameter I'm getting following output:
curl 'http://192.168.10.2:9115/probe?module=icmp&target=${DST_IP}&debug=true'
Logs for the probe:
ts=2024-05-30T08:08:07.411598137Z caller=main.go:181 module=icmp target=${DST_IP} level=info msg="Beginning probe" probe=icmp timeout_seconds=3
ts=2024-05-30T08:08:07.411699339Z caller=icmp.go:91 module=icmp target=${DST_IP} level=info msg="Resolving target address" target=${DST_IP} ip_protocol=ip6
ts=2024-05-30T08:08:07.411718022Z caller=icmp.go:91 module=icmp target=${DST_IP} level=info msg="Resolved target address" target=${DST_IP} ip=${DST_IP}
ts=2024-05-30T08:08:07.411733126Z caller=handler.go:120 module=icmp target=${DST_IP} level=info msg="Creating socket"
ts=2024-05-30T08:08:07.411798973Z caller=handler.go:120 module=icmp target=${DST_IP} level=info msg="Creating ICMP packet" seq=26515 id=60301
ts=2024-05-30T08:08:07.41186266Z caller=handler.go:120 module=icmp target=${DST_IP} level=info msg="Writing out packet"
ts=2024-05-30T08:08:07.411891262Z caller=handler.go:120 module=icmp target=${DST_IP} level=debug msg="Overriding TTL (raw IPv4)" ttl=64
ts=2024-05-30T08:08:07.412139679Z caller=handler.go:120 module=icmp target=${DST_IP} level=warn msg="Error writing to socket" err="write ip 0.0.0.0->${DST_IP}: sendmsg: message too long"
ts=2024-05-30T08:08:07.412223981Z caller=main.go:181 module=icmp target=${DST_IP} level=error msg="Probe failed" duration_seconds=0.000571897
Metrics that would have been returned:
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 1.7268e-05
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.000571897
# HELP probe_icmp_duration_seconds Duration of icmp request by phase
# TYPE probe_icmp_duration_seconds gauge
probe_icmp_duration_seconds{phase="resolve"} 1.7268e-05
probe_icmp_duration_seconds{phase="rtt"} 0
probe_icmp_duration_seconds{phase="setup"} 0.000128335
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 2.164536847e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 0
Module configuration:
prober: icmp
timeout: 3s
http:
ip_protocol_fallback: true
follow_redirects: true
enable_http2: true
tcp:
ip_protocol_fallback: true
icmp:
ip_protocol_fallback: true
payload_size: 1472
dont_fragment: true
ttl: 64
dns:
ip_protocol_fallback: true
recursion_desired: true
Problem resolves after blackbox_exporter container has been restarted, but will resurfaces again after few days/weeks.
Worth mention that other destinations on same blackbox_exporter with using same module are working fine, only few specific ones returning sendmsg: message too long.
Host operating system: output of uname -a
Linux host-1 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
blackbox_exporter version: output of blackbox_exporter --version
blackbox_exporter, version 0.24.0 (branch: HEAD, revision: 0b0467473916fd9e8526e2635c2a0b1c56011dff)
build user: root@e5bbfcc8184e
build date: 20230516-11:07:25
go version: go1.20.4
platform: linux/amd64
tags: netgo
Hello. We are running blackbox_exporter as a docker container (official image) to probe some of our infrastructure with icmp module. For some reason after few days/weeks of constant probing blackbox exporter starting to returned probe_success metric with value 0 for some targets. If I curl blackbox_exporter with this target and
&debug=trueparameter I'm getting following output:Problem resolves after blackbox_exporter container has been restarted, but will resurfaces again after few days/weeks. Worth mention that other destinations on same blackbox_exporter with using same module are working fine, only few specific ones returning
sendmsg: message too long.Host operating system: output of
uname -ablackbox_exporter version: output of
blackbox_exporter --versionWhat is the blackbox.yml module config.