Metricbeat - panic while fetching 'zookeeper/server'

[gservat]

• Version: 6.8.6
• Operating System: Linux / Ubuntu 16.04 (xenial)

Metricbeat stopped collecting zookeeper/server metrics presumably when it had issues talking to the ZK client port (2191). We got around it by restarting metricbeat. Logs:

2021-08-25T03:24:05.159Z ERROR runtime/panic.go:28 recovered from panic while fetching 'zookeeper/server' for host '172.17.0.1:2191'. Recovering, but please report this. {"panic": "runtime error: index out of range", "stack": "github.com/elastic/beats/libbeat/logp.Recover
    /go/src/github.com/elastic/beats/libbeat/logp/global.go:105
runtime.call32
    /usr/local/go/src/runtime/asm_amd64.s:573
runtime.gopanic
    /usr/local/go/src/runtime/panic.go:502
runtime.panicindex
    /usr/local/go/src/runtime/panic.go:28
github.com/elastic/beats/metricbeat/module/zookeeper/server.parseSrvr
    /go/src/github.com/elastic/beats/metricbeat/module/zookeeper/server/data.go:54
github.com/elastic/beats/metricbeat/module/zookeeper/server.(*MetricSet).Fetch
    /go/src/github.com/elastic/beats/metricbeat/module/zookeeper/server/server.go:84
github.com/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).fetch
    /go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:238
github.com/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).startPeriodicFetching
    /go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:219
github.com/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).run
    /go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:196
github.com/elastic/beats/metricbeat/mb/module.(*Wrapper).Start.func1
    /go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:137"}

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[wandergeek]

saw this issue occur today https://elastic.slack.com/archives/CGM2U8Q84/p1634771410257600 and restarted metricbeat with docker restart frc-beats-runners-beats-runner

[lucasmoore]

Hi @sayden, just bumping this, as it keeps happining in cloud and is creating extra work for the on-call engineer.

[sayden]

Thanks for bumping. Let me try to find the correct people for this :slightly_smiling_face:

[lucasmoore]

Thanks for bumping. Let me try to find the correct people for this 🙂

Hi @sayden, sorry for the confusion, but I think this may actually be happening because of a (mis)configuration on our end. I'll be taking a look at it this week, but in the meantime there may not be anything for you to look into.

[sayden]

Still, we must be more resilient to all stuff that a user might reproduce at some point (and give proper feedback about the error). :slightly_smiling_face: I'll try to take a look to improve this behaviour too

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!