Kafka topic is not created when provisioning the Kafka Cluster

[j-tim]

I try to run the example to provision a Kafka cluster in the Confluent Cloud and create a topic. The environment and cluster are successfully created but the topic is not created. It looks like the Kafka broker is not yet ready to create the topic.

I'm using:

• Terraform v0.13.4
• provider registry.terraform.io/mongey/confluentcloud v0.0.5
• provider registry.terraform.io/mongey/kafka v0.2.10

terraform apply output:

confluentcloud_environment.environment: Creating...
confluentcloud_service_account.test: Creating...
confluentcloud_environment.environment: Creation complete after 1s [id=env-9pk7v]
confluentcloud_kafka_cluster.test: Creating...
confluentcloud_kafka_cluster.test: Creation complete after 0s [id=lkc-9g3ky]
confluentcloud_api_key.provider_test: Creating...
confluentcloud_api_key.provider_test: Creation complete after 1s [id=145974]
kafka_topic.syslog: Creating...

Error: Provider produced inconsistent result after apply

When applying changes to confluentcloud_service_account.test, provider
"registry.terraform.io/mongey/confluentcloud" produced an unexpected new
value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

  on main.tf line 54, in resource "kafka_topic" "syslog":
  54: resource "kafka_topic" "syslog" {

When I run terraform apply for the second time the topic will be created. Is there a way to wait until the Kafka cluster is ready to create a topic?

[Mongey]

This is a know issue (#18) ... I have started to look at it in #37

[j-tim]

Thanks @Mongey

[ryandutton]

Hey I'm getting the same error when trying to create a an API key with a service account

[Mongey]

@ryandutton Yeah I believe there's a similar issue with API Keys.... they aren't ready to be used immediately after creation. I'm experimenting with having the provider wait until the key is active -- by having the provider repeatedly attempt to connect to the cluster. That's going on in #37, and if successful there, I can port that to API Key creation.

[ryandutton]

It might be a different issue but I was getting similar results where I would create an api key using the confluentcloud provider and then use that to authenticate the Kafka provider. What I found was that there was insufficient permissions on the service account key to allow the creation of a topic (or an act to allow topic creation). Once I manually assigned cluster and topic create permissions in the UI, I could successfully run the terraform to create the topics.

To summarise, in my case it wasn't that the key didn't exist, it was insufficient permissions on creation.

[ryandutton]

Not sure if possible in the confluentcloud provider but if you can create acl for cluster and topic creation and read then that will allow the Kafka provider to have sufficient permissions to create topic.

[prabhakarank87]

Not sure if possible in the confluentcloud provider but if you can create acl for cluster and topic creation and read then that will allow the Kafka provider to have sufficient permissions to create topic.

I'm also looking for this to be integrated in the terraform plugin

[estao17]

Does anybody know if this is resolved after a year since last post? I still get the same error. confluent kafka topic produce topic-filing-alert Error: failed to obtain topics from client: API key may not be provisioned yet

[ckapucu]

I am getting the same error. Any solution yet?

[ckapucu]

Edit: I asked my network admin to open port :9092 then I could connect and did not get the “API key may not be provisioned yet” error.

[wjduck]

You need to have access to port 9092 to produce and consume messages with the Kafka CLI.