twmb / franz-go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.7+. Producing, consuming, transacting, administrating, etc.
BSD 3-Clause "New" or "Revised" License
1.78k stars 182 forks source link

kprom: Support for multiple clients #815

Open lukasrynt opened 1 month ago

lukasrynt commented 1 month ago

Hi,

I'm trying to use the kprom plugin with two producers running in the same program to expose separate metrics for each producer. I expected the WithClientLabel configuration option to allow this, but when I register both producers via the hooks option to the same metric, only the most recent producer gets registered. Here's an example:

metrics := kprom.NewMetrics("test", kprom.WithClientLabel())

// Prepare producers
primary, err := kgo.NewClient(
        kgo.SeedBrokers(brokers...),
        kgo.ClientID("primary"),
        kgo.WithHooks(metrics),
    )
...
secondary, err := kgo.NewClient(
        kgo.SeedBrokers(brokers...),
        kgo.ClientID("secondary"),
        kgo.WithHooks(metrics),
    )
...
mux := nethttp.NewServeMux()
mux.Handle("/metrics", metrics.Handler())

// Run the actual HTTP server with metrics
...
// Only metrics about secondary client are seen (although I suspect it's sum of both)

However, I believe that supporting separate metrics for multiple consumers/producers within a single program is a valid use case for this plugin, especially since kgo allows multiple client instances in the same application. I would love to see this use case supported, I can even create a pull request for it if you'll find this issue valid.

twmb commented 2 weeks ago

I'm open to this, what's the proposed API?

lukasrynt commented 1 week ago

Thanks for the swift response. I created a PR for this. I would retain the API, feel free to correct me if you don't like it.