search

redpanda-data / kminion

KMinion is a feature-rich Prometheus exporter for Apache Kafka written in Go. It is lightweight and highly configurable so that it will meet your requirements.
MIT License
610 stars 122 forks source link

Kminion 2.2.1 panics time after time #190

Closed mikekamornikov closed 1 year ago

mikekamornikov commented 1 year ago

I have several kminion v2.2.1 deployments and time after time some of them panic:

panic: interface conversion: kmsg.Request is *kmsg.OffsetFetchRequest, not *kgo.pinReq

goroutine 21872 [running]:
github.com/twmb/franz-go/pkg/kgo.(*Client).handleShardedReq.func2({0x0, {0xcf55e8, 0xc00069a0c0}, {0x0, 0x0}})
    /go/pkg/mod/github.com/twmb/franz-go@v1.10.0/pkg/kgo/client.go:1766 +0x974
github.com/twmb/franz-go/pkg/kgo.(*Client).handleShardedReq(0xc0000ce000, {0xcf0750?, 0xc00054ca80}, {0xcf55e8?, 0xc00069a0c0})
    /go/pkg/mod/github.com/twmb/franz-go@v1.10.0/pkg/kgo/client.go:1821 +0x9f9
github.com/twmb/franz-go/pkg/kgo.(*Client).shardedRequest(0xc0000ce000, {0xcf07f8?, 0xc000a9d8c0?}, {0xcf55e8?, 0xc00069a0c0})
    /go/pkg/mod/github.com/twmb/franz-go@v1.10.0/pkg/kgo/client.go:978 +0x691
github.com/twmb/franz-go/pkg/kgo.(*Client).Request(0xc00011f8a0?, {0xcf07f8?, 0xc000a9d8c0?}, {0xcf55e8?, 0xc00069a0c0?})
    /go/pkg/mod/github.com/twmb/franz-go@v1.10.0/pkg/kgo/client.go:778 +0x2d
github.com/twmb/franz-go/pkg/kmsg.(*OffsetFetchRequest).RequestWith(...)
    /go/pkg/mod/github.com/twmb/franz-go/pkg/kmsg@v1.2.0/generated.go:10894
github.com/cloudhut/kminion/v2/minion.(*Service).listConsumerGroupOffsets(0xc0004b6000, {0xcf07f8, 0xc000a9d8c0}, {0xc000041c00, 0x20})
    /app/minion/consumer_group_offsets.go:73 +0xf0
github.com/cloudhut/kminion/v2/minion.(*Service).listConsumerGroupOffsetsBulk.func1.1()
    /app/minion/consumer_group_offsets.go:42 +0x65
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa

I use the following settings on top of kminion defaults:

configMapGenerator:
- name: kafka-minion
  literals:
  - EXPORTER_NAMESPACE=minion
  - LOGGER_LEVEL=error
  - KAFKA_TLS_ENABLED=true
  - KAFKA_TLS_INSECURE_SKIP_TLS_VERIFY=true
    # See https://github.com/redpanda-data/kminion/blob/v2.2.1/docs/reference-config.yaml#L66-L74
  - MINION_CONSUMERGROUPS_SCRAPEMODE=adminApi

I use AWS MSK with Kafka 3.3.1. Any ideas what could it be?

twmb commented 1 year ago

https://github.com/twmb/franz-go/blob/master/CHANGELOG.md#v1113

franz-go dep needs to be bumped

weeco commented 1 year ago

Updated the deps and released v2.2.3