search

prometheus / statsd_exporter

StatsD to Prometheus metrics exporter
Apache License 2.0
921 stars 231 forks source link

panic in dogstatsd extended aggregation parsing #572

Closed matthiasr closed 3 weeks ago

matthiasr commented 1 month ago

https://github.com/prometheus/statsd_exporter/pull/558#pullrequestreview-2259676259 – breaking this out into an issue by itself.

From @m-barthelemy:

After updating to 0.27.1, this line is triggering a panic: runtime error: index out of range [1] with length 1 for us.

Could you run the exporter with --log.level=debug and give us an example of a line that's triggering this? Ideally, add it as a test case and see if that also fails.

@GrgDev it seems that we need to check whether the splitting actually resulted in enough parts, do you have capacity to look into that?

twskipper commented 1 month ago

we got the same error, while using the 0.27.1, fixed by using 0.26.2

m-barthelemy commented 1 month ago

@matthiasr sorry for the delay! Here you go:

ts=2024-09-13T07:09:04.461Z caller=listener.go:97 level=debug msg="Incoming line" proto=udp line=karafka.consumer.consume.cpu_usage_second:0.12528835100000002|h|#consumer:Kafka::SharedConfigurationConsumer,topic:shared_configuration_update,partition:1,consumer_group:tc_rc_us
ts=2024-09-13T07:09:04.461Z caller=listener.go:97 level=debug msg="Incoming line" proto=udp line="karafka.consumer.consume.cpu_idle_second:  0.111090  -0.055903  -0.195390 (  2.419002)"
ts=2024-09-13T07:09:04.461Z caller=line.go:270 level=debug msg="Bad component" line="karafka.consumer.consume.cpu_idle_second:  0.111090  -0.055903  -0.195390 (  2.419002)"
ts=2024-09-13T07:09:04.461Z caller=listener.go:97 level=debug msg="Incoming line" proto=udp line=|h|#consumer:Kafka::SharedConfigurationConsumer,topic:shared_configuration_update,partition:1,consumer_group:tc_rc_us
ts=2024-09-13T07:09:04.461Z caller=line.go:124 level=debug msg="Malformed name tag" tag=consumer component=consumer
panic: runtime error: index out of range [1] with length 1

goroutine 35 [running]:
github.com/prometheus/statsd_exporter/pkg/line.(*Parser).LineToEvents(0xc000288174, {0xc0008306d7, 0x75}, {0xc00028c4b0}, {0xaf60c0, 0xc00007e540}, {0xaf60c0, 0xc0002867e0}, {0xaf60c0, 0xc0002866c0}, ...)
    /app/pkg/line/line.go:235 +0x1785
github.com/prometheus/statsd_exporter/pkg/listener.(*StatsDUDPListener).HandlePacket(0xc0002e80c0, {0xc0008300d0?, 0xc0002873e0?, 0xc0001ca7d0?})
    /app/pkg/listener/listener.go:102 +0xb3
github.com/prometheus/statsd_exporter/pkg/listener.(*StatsDUDPListener).ProcessUdpPacketQueue(...)
    /app/pkg/listener/listener.go:90
created by github.com/prometheus/statsd_exporter/pkg/listener.(*StatsDUDPListener).Listen in goroutine 67
    /app/pkg/listener/listener.go:58 +0x85

(there is no such panic triggered with 0.26.1 and older releases)

matthiasr commented 1 month ago

Thank you! This seems like there is a linebreak in the stats line. That's, uh, tricky because we assume that this is a line-based format. Which client are you using? As a first step, we should not panic, but it may be a bigger lift to actually support this.

matthiasr commented 1 month ago

I'm trying to find a reference for this format but the DogStatsD datagram spec only mentions :-separated packing. I suspect something odd is going on on the sending side here, and we should refuse these lines (but we should not crash).