search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
118 stars 1.04k forks source link

Unnecessary unclean.leader.election.enable override #8479

Open michalstutzmann opened 2 years ago

michalstutzmann commented 2 years ago

Describe the bug In environments with strict polices on topic creation setting unclean.leader.election.enable is often not allowed.

ksqlDB sets unclean.leader.election.enable to false on topic creation: https://github.com/confluentinc/ksql/blob/0.22.0-ksqldb/ksqldb-rest-app/src/main/java/io/confluent/ksql/rest/util/KsqlInternalTopicUtils.java#L44

This setting is already false by default: https://kafka.apache.org/documentation/#brokerconfigs_unclean.leader.election.enable, thus setting it is unnecessary.

To Reproduce N/A

Expected behavior unclean.leader.election.enable is not set.

Actual behaviour ksqlDB tries to set unclean.leader.election.enable to default value on topic creation.

Additional context N/A

colinhicks commented 2 years ago

@michalstutzmann would you be able to clarify the impact when ksqlDB's Kafka client applies the config?

In environments with strict polices on topic creation setting unclean.leader.election.enable is often not allowed.

Is the effect that the topic creation request fails, for example?

As for why ksqlDB is setting the same value as the broker default, I think there is some defensiveness involved: Older versions of Kafka enabled unclean leader election by default.

michalstutzmann commented 2 years ago

@colinhicks yes, exactly, topic creation fails with information about breaking a policy. This is of course an internal issue in my organisation but I can imagine the problem might appear elsewhere.

sugarcrm-jgminder commented 1 year ago

We've actually had Amazon MSK contact us about topics that have unclean.leader.election.enable=false.

We have detected an issue in your Amazon MSK "cluster that could lead to cluster downtime if unaddressed. This cluster has 2 topics configured with unclean.leader.election.enable=false. This effectively stops leadership elections for a partition when the only brokers which are up and running are not in-sync replicas

It would be great to resolve this and allow internal Kafka Streams topics to respect the environment defaults (MSK in our case).