search

databricks / iceberg-kafka-connect

Apache License 2.0
219 stars 49 forks source link

Add ability to override consumer group IcebergSinkConfig.DEFAULT_CONTROL_GROUP_PREFIX #178

Open drewtech opened 10 months ago

drewtech commented 10 months ago

Background

At my company, our kafka consumer groups are restricted to a specific prefix, so the ability to override would be useful.

The issue

Currently there is a hardcoded group IcebergSinkConfig.DEFAULT_CONTROL_GROUP_PREFIX + UUID.randomUUID() that is raising an error on our cluster. The error is 

[2023-12-21 03:46:15,972] ERROR [iceberg-cdc-datalake|task-0] WorkerSinkTask{id=iceberg-cdc-datalake-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:210)
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: cg-control-0d46c80d-19c8-497d-9211-e6c17f1f9f66

Suggested fix

Add a configuration override for IcebergSinkConfig.DEFAULT_CONTROL_GROUP_PREFIX to allow for customization.