search

dashbitco / broadway_kafka

A Broadway connector for Kafka
233 stars 53 forks source link

Implementing offset lag telemetry #145

Closed David-Klemenc closed 4 months ago

David-Klemenc commented 4 months ago

Would it be possible to get lag offset in telemetry something like kolt ?

v0idpwn commented 4 months ago

I think you should be able to use kolt with broadway_kafka as it uses brod

David-Klemenc commented 4 months ago

I can get producer names from Broadway:

Broadway.producer_names(:kafka_test)
[:"kafka_test.Broadway.Producer_0"]

And from the logs I know that the client names look like

:"Elixir.kafka_test.Broadway.Producer_0.Client"

But I have not found a way to get the client name from a Broadway api ? (I need it to call :brod)

:brod.fetch_committed_offsets(:"Elixir.kafka_test.Broadway.Producer_0.Client", "test_group_id")
josevalim commented 4 months ago

I don’t believe there is one. A PR is welcome!

GregorGrasselli commented 4 months ago

Based on the debate in the PR, @David-Klemenc also agrees we can close this.

josevalim commented 4 months ago

Thanks! A PR to expose the client is still welcome (if there isn't one).