search

tensorflow / io

Dataset, streaming, and file system extensions maintained by TensorFlow SIG-IO
Apache License 2.0
706 stars 286 forks source link

[Kafka-io - Suggestion] Adding monitoring options for KafkaDataset and KafkaOutputSequence #1137

Open tungnat97 opened 4 years ago

tungnat97 commented 4 years ago

It would be helpful for monitoring Kafka-io consuming and producing performance.

nmatare commented 4 years ago

Check out:

https://github.com/lightbend/kafka-lag-exporter https://github.com/cloudworkz/kafka-minion

kvignesh1420 commented 4 years ago

@tungnat97 can you let us know what exactly you are looking for? Thanks.

tungnat97 commented 4 years ago

@tungnat97 can you let us know what exactly you are looking for? Thanks.

I want to measure latency of KafkaDataset when comsuming messages from topics and KafkaOutputSequence when producing messages to topics.

kvignesh1420 commented 3 years ago

@tungnat97 I see, so here is what you can do as of now. In the configurations parameter, you can pass `statistics.interval.ms" value to log the stats on the terminal.

For example:

configuration=[
            "session.timeout.ms=7000",
            "max.poll.interval.ms=8000",
            "statistics.interval.ms=1000",
            "auto.offset.reset=earliest"
     ]

Increase or decrease the value as per need.