search

twmb / franz-go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.6+. Producing, consuming, transacting, administrating, etc.
BSD 3-Clause "New" or "Revised" License
1.6k stars 158 forks source link

Confluent: Unable to create a topic due to policy violation #751

Open Amogh-Bharadwaj opened 1 month ago

Amogh-Bharadwaj commented 1 month ago

This issue is related to an attempt to create a topic (using the kadm package) in a Confluent cluster.

Implementation

Admin client is created with the following configurations:

        kgo.SeedBrokers(config.Servers...),
        kgo.AllowAutoTopicCreation(),
        kgo.WithLogger(kslog.New(slog.Default()))

and with kadm.NewOptClient(aboveOptions...)

The request is as follows:

// 5 partitions, replicationFactor of 1, no other configs
adminClient.CreateTopics(ctx, 5, 1, nil, "mytopic")

Both the .CreateTopic() and the.CreateTopics() methods fail with:

POLICY_VIOLATION: Request parameters do not satisfy the configured policy.

I am aware that the issue could be something related to Confluent technicalities, but just wanted to know if there is something configurable from the franz-go side to get through this. Does kadm have support for Confluent Cloud?

twmb commented 1 month ago

kgo works with Confluent cloud -- it's all client protocol stuff, and you can make this client look like any other by changing the client ID. Do you have any broker logs? It sounds like you have too many topics or partitions in the cluster already?

Amogh-Bharadwaj commented 1 month ago

Turns out Confluent has a policy which mandatorily requires replication factor be equal to 3. The error message conveying that was revealed when I tried the same thing using confluent-kafka-go. I then switched back to franz-go and it's working now :)

twmb commented 1 month ago

Where did you see the error? The same information should be visible from kadm... I think the problem is that kadm doesn't include a CreateTopicsResponse.ErrorMessage field right now, which it should. I'm going to reopen this to track adding that field as an enhancement, and potentially audit other result structs at the same time.