search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.8k stars 562 forks source link

connector: allow customization on the prefix of `group.id` for Kafka readers #13386

Open BugenZhao opened 10 months ago

BugenZhao commented 10 months ago

Currently we hardcode it to be consumer-. Users may want to customize it to be consumer_ or something else.

https://github.com/risingwavelabs/risingwave/blob/e723f6a66ea9b02b79efed40306bb6317006c1ae/src/connector/src/source/kafka/source/reader.rs#L79-L90

Note that specifying the whole group id is not expected as described in https://github.com/risingwavelabs/risingwave/pull/13223#issuecomment-1791798953. So maybe we should remove consumer_group from properties to avoid confusion.

cc @shanicky @tabVersion

tabVersion commented 10 months ago

So maybe we should remove consumer_group from properties to avoid confusion.

We've removed the group.id from the dev doc to avoid unexpected behavior.

Users may want to customize it to be consumer_ or something else.

I am afraid of bringing up more confusion for normal users that we have to explain how RW uses the low-level API and how splits are distributed to different actors. group.id is required by the kafka client but we don't necessarily need it.

github-actions[bot] commented 3 months ago

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.