Kafka consumer Offset from beginning

Loxnol commented 9 months ago

Hello,

I'm currently working on a simple pipeline to ingest kafka messages inside a log file.

I'm trying to consume all the data from the beginning of a topic but i'm only getting newer data added to this topic. Once consumed, the previous topic messages are not accessible anymore.

I've already test all the different "Auto Offset Reset" properties. Same for simple and multi topic consumers. In the official documentation docs.streamsets.com : auto.commit.interval.ms bootstrap.servers enable.auto.commit group.id max.poll.records

If I understand correctly all those parameters are locked so I can't disable the offset management and process all the data from the beginning of a topic.

Is there an additionnal Kafka configuration property to use or do I need to configure the topic directly via kafka CLI ??

StreamSets Data Collector version : 3.14.0 Kafka Consumer version : 2.0.0

Regards.

xverges commented 9 months ago

Hi Loxnol, thanks for your interest in StreamSets.

it will be easier to get support in https://community.streamsets.com, so I kindly ask you to move your question there. I will be closing this issue.
SDC 3.14.0 is not supported any more by StreamSets. Given that you are using an Open Source version, this ols announcement might be relevant: https://community.streamsets.com/faq-53/streamsets-data-collector-and-transformer-4-0-overview-faq-254

Loxnol commented 9 months ago

Hi @xverges, I've created this article : https://community.streamsets.com/community-articles-and-got-a-question-7/kafka-consumer-offset-from-beginning-every-batch-2301

Indeed I'm using the Open Source version.

Thank you for your help.

You can close the issue.

streamsets / tutorials

Kafka consumer Offset from beginning #146