confluent_kafka_topic Resource: optional rest_endpoint and credentials attributes are actually mandatory

[alukic]

Using https://registry.terraform.io/providers/confluentinc/confluent/1.9.0 I create a topic by

resource "confluent_kafka_topic" "bogus_topic" {
  kafka_cluster {
    id = var.ccloud_cluster_id
  }

 topic_name       = "bogus-deleteme"
  partitions_count = 1
}

The plan confirms the topic creation

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # confluent_kafka_topic.bogus_topic will be created
  + resource "confluent_kafka_topic" "bogus_topic" {
      + config           = (known after apply)
      + id               = (known after apply)
      + partitions_count = 1
      + topic_name       = "bogus-deleteme"

      + kafka_cluster {
          + id = "lkc-123"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

But terraform apply failed to create

│ Error: error creating Kafka Topic: one of provider.kafka_rest_endpoint (defaults to KAFKA_REST_ENDPOINT environment variable) or resource.rest_endpoint must be set
│
│   with confluent_kafka_topic.bogus_topic,
│   on main.tf line 68, in resource "confluent_kafka_topic" "bogus_topic":
│   68: resource "confluent_kafka_topic" "bogus_topic" {

Documentation https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_kafka_topic says that rest_endpoint and credentials are optional attributes

My provider config use environment vars as authentication

terraform {
  required_providers {
    confluent = {
      source  = "confluentinc/confluent"
      version = "~> 1.9.0"
    }
  }
}

provider "confluent" {
  #cloud_api_key    = "???" # optionally use CONFLUENT_CLOUD_API_KEY env-var
  #cloud_api_secret = "???" # optionally use CONFLUENT_CLOUD_API_SECRET env-var
}

Expected behaviour

rest_endpoint and credentials attributes should not be mandatory as per documentation. Also, the confluent provider has succeeded to authenticate to Confluent and had computed the plan correctly. Otherwise, terraform plan should inform about the missing/mandatory attributes.

[linouk23]

👋 thanks for creating the issue!

Could you double check that your configuration block looks like ⬇️ (also you might want to take a look at managing-single-cluster for more details):

# https://registry.terraform.io/providers/confluentinc/confluent/latest/docs

# Option #2 when managing a single cluster in the same Terraform workspace
# See https://github.com/confluentinc/terraform-provider-confluent/tree/master/examples/configurations/managing-single-cluster for more details
provider "confluent" {
  cloud_api_key       = var.confluent_cloud_api_key    # optionally use CONFLUENT_CLOUD_API_KEY env var
  cloud_api_secret    = var.confluent_cloud_api_secret # optionally use CONFLUENT_CLOUD_API_SECRET env var

  kafka_rest_endpoint = var.kafka_rest_endpoint        # optionally use KAFKA_REST_ENDPOINT env var 
  kafka_api_key       = var.kafka_api_key              # optionally use KAFKA_API_KEY env var
  kafka_api_secret    = var.kafka_api_secret           # optionally use KAFKA_API_SECRET env var
}

[linouk23]

cc @alukic

[tringuyen-yw]

@linouk23 The documentation Confluent Provider, Provider Authentication mentions

Option #1 when managing multiple clusters in the same Terraform workspace
$ export CONFLUENT_CLOUD_API_KEY="<cloud_api_key>"
$ export CONFLUENT_CLOUD_API_SECRET="<cloud_api_secret>"

Which is closest to our use case, ie. managing multiple clusters. However we don't use Terraform workspace. Furthermore, using the Confluent CLI we can query Kafka cluster objects without using the REST endpoint

confluent kafka topic describe lkc-333aaa --environment env-xx456
confluent kafka topic describe --cluster lkc-333aaa --environment env-xx456 mytopic1

Q1. why does the Confluent provider need kafka_rest_endpoint to manage a cluster? But the Confluent CLI doesn't not.

Q2. Let's assume the confluent_kafka_topic needs a rest_endpoint and credentials. How come terraform plan doesn't inform about those missing properties? terraform plan was successful with the TF code mentioned in the first post (ie. WITHOUT rest_endpoint and credentials) this indicates that TF was able to query Confluent to get the topic properties. Why does then terraform apply fail on the same code?

Q3. Setting the kafka_rest_endpoint, kafka_api_key, kafka_api_secret in the provider config. Does NOT address the original question. Which is that confluent_kafka_topic documentation mentions that those properties are OPTIONAL:

rest_endpoint (Optional String) The REST endpoint of the Kafka cluster credentials (Optional Configuration Block) supports the following

[linouk23]

Q1: CLI uses some fancy optimizations that we decided not to add to TF Provider at the moment (for example, consider someone managing 100s of topics and 100s of ACLs -- calling getCluster() to infer REST endpoint would create quite a load for Confluent APIs). We might consider adding it in a future though.

Q2:

this indicates that TF was able to query Confluent to get the topic properties

I'm not sure that's the case.

How come terraform plan doesn't inform about those missing properties?

Sounds like our input validation is not strict enough - we'll consider improving it 👌

Q3: We figured docs should reflect both Option #1 and Option #2 (since both having and not having this attribute is valid -- we decided to mark it optional to make it valid for both options).

[linouk23]

I'm going to resolve this issue but feel free to reopen if you have other questions.