search

karafka / rdkafka-ruby

Modern and performant Kafka client library for Ruby based on librdkafka
https://karafka.io
Other
354 stars 120 forks source link

Non global logger and statistics callback #82

Open mensfeld opened 5 years ago

mensfeld commented 5 years ago

Majority of the settings of this lib are per instance based.

Is there any reason why logger and statistics_callback aren't? It is really problematic for systems where you may have several instances of rdkafka running for producing and consuming with different policies around instrumentation.

thijsc commented 5 years ago

Looking at the docs it does seem like both are scoped on the client. So it should be possible to move them to a config's instance. I have some other issues with higher priority, will take a look at this after those. Also open to a pull of course.

thijsc commented 3 years ago

Closing this one since we don't currently need this. If anybody needs this do reopen.

mensfeld commented 3 years ago

If anybody needs this do reopen.

I do not have a way to reopen :(

thijsc commented 3 years ago

Are you interested in working on this?

mensfeld commented 3 years ago

@thijsc yes. Just leave it open for 2-3 more weeks. I'm finishing Karafka 2.0 and then I will have some time to adjust things on rdkafka side :)

mensfeld commented 2 years ago

Closing as I was able to bypass this limitation without changing the design. If anyone needs this, just ping me.

trickleup commented 2 years ago

@mensfeld - running into the same issue that you describe, of needing to have multiple instances of producers/consumers, and to be able to isolate the statistics_callbacks between those instances.

I was able to bypass this limitation without changing the design. If anyone needs this, just ping me.

Do you mind sharing the gist of what you did to achieve this?

mensfeld commented 2 years ago

@trickleup I'm on a holiday and I plan to backport if from karafka and waterdrop to rdkafka once back (2 weeks).

If you are looking for multiple instances I would recommend going with the wrapper I did waterdrop: https://github.com/karafka/waterdrop as it tackles those cases. The code for this is also somewhere there. Writing from mobile so can't locate the exact files sorry :pray:

jychen7 commented 1 year ago

I plan to backport if from karafka and waterdrop to rdkafka once back (2 weeks).

👍 we have use case where one process contains multiple rdkafka consumer. They share same client_id, so it would be great if we can separate RebalanceCallback for each (e.g. attach name / group_instance_id for statistics metrics).

mensfeld commented 1 year ago

@jychen7 ok clearly it is taking me more than that but my original solution still stands and I still have plans to merge this and other things to the upstream