koeninger
commented
7 years ago You should be able to get a rough idea from logs, without really changing anything other than perhaps logging levels.
KafkaRDDIterator logs at info level when it starts consuming a particular offset range
logInfo(s"Computing topic ${part.topic}, partition ${part.partition} " +
s"offsets ${part.fromOffset} -> ${part.untilOffset}")CachedKafkaConsumer logs at debug level when it actually gets a particular offset.
logDebug(s"Get $groupId $topic $partition nextOffset $nextOffsetrequested $offset")
On Mon, Jun 26, 2017 at 11:35 AM, igor mazor notifications@github.com wrote:
Hi,
I would like to verify small thing. In case my stream processing is bigger then the micro-batch interval, I see queuing up tasks with how much records would be consumed. The consumption would be done actually only when that task started ? (so the tasks in the queue are not consuming from kafka until they actually executed)
My workflow is quite simple, I read from kafka, do some simple processing then reshuffle the data for some further processing (reduceByKey). Usually the job is completed fast enough without causing tasks to be queued. But from time to time I am seeing quite large delay, usually only on 1-2 workers, in the stage that consume data from kafka. I start to suspicious that the bottleneck is consuming from kafka and hence I would like to somehow measure how long it is actually take to consume each topic/partition before the data processing part starts. Any idea how to do it? maybe I can access somehow more consumers metrics ?
Thanks,
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/koeninger/kafka-exactly-once/issues/13, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGAB58uT-OATSsDaiUDPzg4onHhCdPkks5sH93CgaJpZM4OFjpw .
mazorigal
Hi,
I would like to verify small thing. In case my stream processing is bigger then the micro-batch interval, I see queuing up tasks with how much records would be consumed. The consumption would be done actually only when that task started ? (so the tasks in the queue are not consuming from kafka until they actually executed)
My workflow is quite simple, I read from kafka, do some simple processing then reshuffle the data for some further processing (reduceByKey). Usually the job is completed fast enough without causing tasks to be queued. But from time to time I am seeing quite large delay, usually only on 1-2 workers, in the stage that consume data from kafka. I start to suspicious that the bottleneck is consuming from kafka and hence I would like to somehow measure how long it is actually take to consume each topic/partition before the data processing part starts. Any idea how to do it? maybe I can access somehow more consumers metrics ?
Thanks,