Stack trace creating confluentcloud_kafka_acl resource

Hey Folks,

I'm getting the following stack trace trying to create a Kafka ACL using Confluent Terraform provider v0.5.0:

confluentcloud_kafka_acl.describe_cluster: Creating...
╷
│ Error: Request cancelled
│ 
│   with confluentcloud_kafka_acl.describe_cluster,
│   on stack_trace.tf line 22, in resource "confluentcloud_kafka_acl" "describe_cluster":
│   22: resource "confluentcloud_kafka_acl" "describe_cluster" {
│ 
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵

Stack trace from the terraform-provider-confluentcloud_0.5.0 plugin:

panic: reflect: call of reflect.Value.FieldByName on zero Value

goroutine 66 [running]:
reflect.flag.mustBe(...)
    /usr/local/golang/1.16/go/src/reflect/value.go:221
reflect.Value.FieldByName(0x0, 0x0, 0x0, 0xdd983e, 0x6, 0x0, 0x140, 0x12e)
    /usr/local/golang/1.16/go/src/reflect/value.go:903 +0x25a
github.com/confluentinc/terraform-provider-ccloud/internal/provider.createDiagnosticsWithDetails(0xeda2e0, 0xc000596140, 0xc000267470, 0x3, 0x3)
    src/github.com/confluentinc/terraform-provider-confluentcloud/internal/provider/utils.go:304 +0x2c5
github.com/confluentinc/terraform-provider-ccloud/internal/provider.kafkaAclCreate(0xee8fe8, 0xc00060cc60, 0xc0001cae80, 0xd25360, 0xc00022e930, 0xc0001fe3e0, 0x868caa, 0xc0001cad00)
    src/github.com/confluentinc/terraform-provider-confluentcloud/internal/provider/resource_kafka_acl.go:179 +0x547
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0xc0003b10a0, 0xee8f78, 0xc00009e4c0, 0xc0001cae80, 0xd25360, 0xc00022e930, 0x0, 0x0, 0x0)
    pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:341 +0x17f
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc0003b10a0, 0xee8f78, 0xc00009e4c0, 0xc0001d2680, 0xc0001cad00, 0xd25360, 0xc00022e930, 0x0, 0x0, 0x0, ...)
    pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:467 +0x67b
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc000304120, 0xee8f78, 0xc00009e4c0, 0xc00014b090, 0xde2957, 0x12, 0x0)
    pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/grpc_provider.go:977 +0xacf
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc000788080, 0xee9020, 0xc00009e4c0, 0xc000242000, 0x0, 0x0, 0x0)
    pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/tf5server/server.go:603 +0x465
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler(0xda0580, 0xc000788080, 0xee9020, 0xc0001181b0, 0xc00060c360, 0x0, 0xee9020, 0xc0001181b0, 0xc0001be600, 0x2f8)
    pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:380 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002f6540, 0xef0618, 0xc0004b4a80, 0xc0001ba000, 0xc0002a25a0, 0x1396a80, 0x0, 0x0, 0x0)
    pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1210 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0002f6540, 0xef0618, 0xc0004b4a80, 0xc0001ba000, 0x0)
    pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1533 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0003101d0, 0xc0002f6540, 0xef0618, 0xc0004b4a80, 0xc0001ba000)
    pkg/mod/google.golang.org/grpc@v1.33.2/server.go:871 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    pkg/mod/google.golang.org/grpc@v1.33.2/server.go:869 +0x1fd

Error: The terraform-provider-confluentcloud_0.5.0 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

How to reproduce?

Create a Terraform file with the following content:

terraform {
  required_providers {
    confluentcloud = {
      source  = "confluentinc/confluentcloud"
      version = "0.5.0"
    }
  }
}

provider "confluentcloud" {}

resource "confluentcloud_environment" "stack_trace" {
  display_name = "stack_trace"
}

resource "confluentcloud_service_account" "stack_trace" {
  display_name = "stack-trace"
  description  = "Service account for stack trace reproduction"
}

resource "confluentcloud_kafka_cluster" "stack_trace" {
  display_name = "default"
  availability = "SINGLE_ZONE"
  cloud        = "GCP"
  region       = "us-west4"
  basic {}

  environment {
    id = confluentcloud_environment.stack_trace.id
  }
}

output "environment_id" {
  value = confluentcloud_environment.stack_trace.id
}

output "cluster_id" {
  value = confluentcloud_kafka_cluster.stack_trace.id
}

output "service_account_id" {
  value = confluentcloud_service_account.stack_trace.id
}

Run terraform apply:

$ terraform apply
confluentcloud_environment.default: Refreshing state... [id=env-3y5do]
confluentcloud_service_account.tessitura_integration: Refreshing state... [id=sa-22g2dq]
confluentcloud_kafka_cluster.default: Refreshing state... [id=lkc-w7769j]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # confluentcloud_environment.stack_trace will be created
  + resource "confluentcloud_environment" "stack_trace" {
      + display_name = "stack_trace"
      + id           = (known after apply)
    }

  # confluentcloud_kafka_cluster.stack_trace will be created
  + resource "confluentcloud_kafka_cluster" "stack_trace" {
      + api_version        = (known after apply)
      + availability       = "SINGLE_ZONE"
      + bootstrap_endpoint = (known after apply)
      + cloud              = "GCP"
      + display_name       = "default"
      + http_endpoint      = (known after apply)
      + id                 = (known after apply)
      + kind               = (known after apply)
      + rbac_crn           = (known after apply)
      + region             = "us-west4"

      + basic {}

      + environment {
          + id = (known after apply)
        }
    }

  # confluentcloud_service_account.stack_trace will be created
  + resource "confluentcloud_service_account" "stack_trace" {
      + api_version  = (known after apply)
      + description  = "Service account for stack trace reproduction"
      + display_name = "stack-trace"
      + id           = (known after apply)
      + kind         = (known after apply)
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + cluster_id         = (known after apply)
  + environment_id     = (known after apply)
  + service_account_id = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

confluentcloud_service_account.stack_trace: Creating...
confluentcloud_environment.stack_trace: Creating...
confluentcloud_environment.stack_trace: Creation complete after 2s [id=env-zoy90]
confluentcloud_kafka_cluster.stack_trace: Creating...
confluentcloud_service_account.stack_trace: Creation complete after 2s [id=sa-rrm5k0]
confluentcloud_kafka_cluster.stack_trace: Creation complete after 8s [id=lkc-w77kkg]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

cluster_id = "lkc-w77kkg"
environment_id = "env-zoy90"
service_account_id = "sa-rrm5k0"

Create API Key:

confluent api-key create \
    --service-account $(terraform output -raw service_account_id) \
    --environment $(terraform output -raw environment_id) \
    --resource $(terraform output -raw cluster_id)

It may take a couple of minutes for the API key to be ready.
Save the API key and secret. The secret is not retrievable later.
+---------+------------------------------------------------------------------+
| API Key | XXX_api_key_here_XXX                                             |
| Secret  | XXX_secret_here_XXX                                              |
+---------+------------------------------------------------------------------+

Using the API Key and Secret from previous command output, add the following code to the Terraform file created in point 1:

resource "confluentcloud_kafka_acl" "describe_cluster" {
kafka_cluster = confluentcloud_kafka_cluster.stack_trace.id
http_endpoint = confluentcloud_kafka_cluster.stack_trace.http_endpoint
resource_type = "CLUSTER"
resource_name = "kafka-cluster"
pattern_type  = "LITERAL"
principal     = "User:${confluentcloud_service_account.stack_trace.id}"
host          = "*"
operation     = "DESCRIBE"
permission    = "ALLOW"

credentials {
key    = "XXX_api_key_here_XXX"
secret = "XXX_secret_here_XXX"
}
}

Finally, run terraform apply again to get the stack trace:

$ terraform apply
confluentcloud_service_account.tessitura_integration: Refreshing state... [id=sa-22g2dq]
confluentcloud_environment.stack_trace: Refreshing state... [id=env-zoy90]
confluentcloud_environment.default: Refreshing state... [id=env-3y5do]
confluentcloud_service_account.stack_trace: Refreshing state... [id=sa-rrm5k0]
confluentcloud_kafka_cluster.stack_trace: Refreshing state... [id=lkc-w77kkg]
confluentcloud_kafka_cluster.default: Refreshing state... [id=lkc-w7769j]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # confluentcloud_kafka_acl.describe_cluster will be created
  + resource "confluentcloud_kafka_acl" "describe_cluster" {
      + host          = "*"
      + http_endpoint = "https://pkc-6ojv2.us-west4.gcp.confluent.cloud:443"
      + id            = (known after apply)
      + kafka_cluster = "lkc-w77kkg"
      + operation     = "DESCRIBE"
      + pattern_type  = "LITERAL"
      + permission    = "ALLOW"
      + principal     = "User:sa-rrm5k0"
      + resource_name = "kafka-cluster"
      + resource_type = "CLUSTER"

      + credentials {
          + key    = (sensitive value)
          + secret = (sensitive value)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

confluentcloud_kafka_acl.describe_cluster: Creating...
╷
│ Error: Request cancelled
│ 
│   with confluentcloud_kafka_acl.describe_cluster,
│   on stack_trace.tf line 22, in resource "confluentcloud_kafka_acl" "describe_cluster":
│   22: resource "confluentcloud_kafka_acl" "describe_cluster" {
│ 
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵

Stack trace from the terraform-provider-confluentcloud_0.5.0 plugin:

panic: reflect: call of reflect.Value.FieldByName on zero Value

goroutine 66 [running]:
reflect.flag.mustBe(...)
    /usr/local/golang/1.16/go/src/reflect/value.go:221
reflect.Value.FieldByName(0x0, 0x0, 0x0, 0xdd983e, 0x6, 0x0, 0x140, 0x12e)
    /usr/local/golang/1.16/go/src/reflect/value.go:903 +0x25a
github.com/confluentinc/terraform-provider-ccloud/internal/provider.createDiagnosticsWithDetails(0xeda2e0, 0xc000596140, 0xc000267470, 0x3, 0x3)
    src/github.com/confluentinc/terraform-provider-confluentcloud/internal/provider/utils.go:304 +0x2c5
github.com/confluentinc/terraform-provider-ccloud/internal/provider.kafkaAclCreate(0xee8fe8, 0xc00060cc60, 0xc0001cae80, 0xd25360, 0xc00022e930, 0xc0001fe3e0, 0x868caa, 0xc0001cad00)
    src/github.com/confluentinc/terraform-provider-confluentcloud/internal/provider/resource_kafka_acl.go:179 +0x547
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0xc0003b10a0, 0xee8f78, 0xc00009e4c0, 0xc0001cae80, 0xd25360, 0xc00022e930, 0x0, 0x0, 0x0)
    pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:341 +0x17f
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc0003b10a0, 0xee8f78, 0xc00009e4c0, 0xc0001d2680, 0xc0001cad00, 0xd25360, 0xc00022e930, 0x0, 0x0, 0x0, ...)
    pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:467 +0x67b
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc000304120, 0xee8f78, 0xc00009e4c0, 0xc00014b090, 0xde2957, 0x12, 0x0)
    pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/grpc_provider.go:977 +0xacf
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc000788080, 0xee9020, 0xc00009e4c0, 0xc000242000, 0x0, 0x0, 0x0)
    pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/tf5server/server.go:603 +0x465
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler(0xda0580, 0xc000788080, 0xee9020, 0xc0001181b0, 0xc00060c360, 0x0, 0xee9020, 0xc0001181b0, 0xc0001be600, 0x2f8)
    pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:380 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002f6540, 0xef0618, 0xc0004b4a80, 0xc0001ba000, 0xc0002a25a0, 0x1396a80, 0x0, 0x0, 0x0)
    pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1210 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0002f6540, 0xef0618, 0xc0004b4a80, 0xc0001ba000, 0x0)
    pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1533 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0003101d0, 0xc0002f6540, 0xef0618, 0xc0004b4a80, 0xc0001ba000)
    pkg/mod/google.golang.org/grpc@v1.33.2/server.go:871 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    pkg/mod/google.golang.org/grpc@v1.33.2/server.go:869 +0x1fd

Error: The terraform-provider-confluentcloud_0.5.0 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Additional information

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ terraform -v
Terraform v1.1.3
on linux_amd64
+ provider registry.terraform.io/confluentinc/confluentcloud v0.5.0

Your version of Terraform is out of date! The latest version
is 1.1.7. You can update by downloading from https://www.terraform.io/downloads.html

Thanks for reporting the issue @ggmoy-tt and providing so many details to help us debug the issue!

Could you switch to 0.4.0 version of TF Provider for Confluent Cloud and reapply your configuration? This way you should be able to see a more descriptive error message (vs just stack trace) and share with us the findings?

Could you double check you set the following env variables:

export CONFLUENT_CLOUD_API_KEY="<cloud_api_key>"
export CONFLUENT_CLOUD_API_SECRET="<cloud_api_secret>"

I re-created the environment, cluster and service account using v0.4.0. Now, the error is the following:

$ terraform apply
confluentcloud_environment.default: Refreshing state... [id=env-3y5do]
confluentcloud_environment.stack_trace: Refreshing state... [id=env-qm5g7]
confluentcloud_service_account.tessitura_integration: Refreshing state... [id=sa-22g2dq]
confluentcloud_service_account.stack_trace: Refreshing state... [id=sa-nvmyx3]
confluentcloud_kafka_cluster.default: Refreshing state... [id=lkc-w7769j]
confluentcloud_kafka_cluster.stack_trace: Refreshing state... [id=lkc-yoowmk]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # confluentcloud_kafka_acl.describe_cluster will be created
  + resource "confluentcloud_kafka_acl" "describe_cluster" {
      + host          = "*"
      + http_endpoint = "https://pkc-6ojv2.us-west4.gcp.confluent.cloud:443"
      + id            = (known after apply)
      + kafka_cluster = "lkc-yoowmk"
      + operation     = "DESCRIBE"
      + pattern_type  = "LITERAL"
      + permission    = "ALLOW"
      + principal     = "User:sa-nvmyx3"
      + resource_name = "kafka-cluster"
      + resource_type = "CLUSTER"

      + credentials {
          + key    = (sensitive value)
          + secret = (sensitive value)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

confluentcloud_kafka_acl.describe_cluster: Creating...
╷
│ Error: 403 Forbidden
│ 
│   with confluentcloud_kafka_acl.describe_cluster,
│   on stack_trace.tf line 22, in resource "confluentcloud_kafka_acl" "describe_cluster":
│   22: resource "confluentcloud_kafka_acl" "describe_cluster" {
│ 
╵

And, as to the environment variables CONFLUENT_CLOUD_API_KEYand CONFLUENT_CLOUD_API_SECRET, the are in place. Otherwise, the Terraform provider would not be able to create any resource.

Thanks for sending these details @ggmoy-tt!

So the issue is related to step #3:

confluent api-key create \
    --service-account $(terraform output -raw service_account_id) \
    --environment $(terraform output -raw environment_id) \
    --resource $(terraform output -raw cluster_id)

As of right now, even if the owner of Kafka API Key is a service account (and not a user) and even if its role is OrganizationAdmin its permissions doesn't "apply" on a cluster level (which is why you're getting 403 for ACL / topic creation) so there're 2 simple options:

Wait around 1-2 weeks 🤞 for us to release a feature / fix such that ⬆️ will start working.
Alternatively, you can use the updated command for creating a Global access Kafka API Key:
```
confluent api-key create \
--environment $(terraform output -raw environment_id) \
--resource $(terraform output -raw cluster_id)
```
that will be linked to your account (instead of a service account). Later on, you'll be able to rotate Kafka API Keys by following this note. As a matter of fact, we're working on api_key TF resource and we'll release a corresponding tutorial / sample TF configs to make things easier.

Let me know if it helps at all.

Thanks for your time, @linouk23!

I think we'll wait for the fix. We don't like the idea of using user accounts because, as stated in the Access Control List documentation, anyone with access to a particular cluster while using the Cloud Console browser has full access to all resources in that cluster (which is the same as having super user access).

Sounds good, I'll keep this issue open and update you as soon as Kafka RBAC goes GA such that your Kafka API Key created with:

confluent api-key create \
    --environment $(terraform output -raw environment_id) \
    --resource $(terraform output -raw cluster_id)

will stop returning 403 👌

Actually, the command used to create the API Key is the following:

confluent api-key create \
    --service-account $(terraform output -raw service_account_id) \
    --environment $(terraform output -raw environment_id) \
    --resource $(terraform output -raw cluster_id)

Oh right, that's a typo (I meant Kafka API Key owned by a service account with CloudClusterAdmin role assigned), sorry about that! Here's the updated message:

Sounds good, I'll keep this issue open and update you as soon as Kafka RBAC goes GA such that your Kafka API Key created with:
confluent api-key create \
    --service-account $(terraform output -raw service_account_id) \
    --environment $(terraform output -raw environment_id) \
    --resource $(terraform output -raw cluster_id)
will stop returning 403 👌

@ggmoy-tt fyi Granular role-based access control (RBAC) for Kafka resources has been promoted to General Availability today (see here for more details).

Let me know if it works now.

@linouk23, I'm still receiving a 403 trying to create confluentcloud_kafka_acl.

@ggmoy-tt, right, what's happening is the owner of your Kafka API Key:

confluent api-key create \
    --service-account $(terraform output -raw service_account_id) \
    --environment $(terraform output -raw environment_id) \
    --resource $(terraform output -raw cluster_id)

is a "fresh" service account without any permissions (let alone "create ACLs" permission).

The expected design would be to have

a app-manager service account that has an admin permissions for a cluster and its Kafka API Key will be used for topic / ACL creation for app-consumer and app-producer service accounts.
app-consumer service account -- use its Kafka API Key to consume data from a specific topic
app-producer service account -- use its Kafka API Key to produce data to a specific topic

So first you need to add a CloudClusterAdmin role to it (and use it as an app-manager service account or create a separate app-manager):

// 'app-manager' service account is required in this configuration to create 'orders' topic and grant ACLs
// to 'app-producer' and 'app-consumer' service accounts.
resource "confluentcloud_service_account" "app-manager" {
  display_name = "app-manager"
  description  = "Service account to manage 'inventory' Kafka cluster"
}

resource "confluentcloud_role_binding" "app-manager-kafka-cluster-admin" {
  principal   = "User:${confluentcloud_service_account.app-manager.id}"
  role_name   = "CloudClusterAdmin"
  crn_pattern = confluentcloud_kafka_cluster.stack_trace.rbac_crn
}

and then

confluent api-key create \
    # or use app-manager's service account ID
    --service-account $(terraform output -raw service_account_id) \
    --environment $(terraform output -raw environment_id) \
    --resource $(terraform output -raw cluster_id)

and then you might need to create a separate app-consumer service account:

resource "confluentcloud_service_account" "app-consumer" {
  display_name = "app-consumer"
  description  = "Service account to consume from 'X' topic of 'Y' Kafka cluster"
}

resource "confluentcloud_kafka_acl" "describe_cluster" {
  kafka_cluster = confluentcloud_kafka_cluster.stack_trace.id
  http_endpoint = confluentcloud_kafka_cluster.stack_trace.http_endpoint
  resource_type = "CLUSTER"
  resource_name = "kafka-cluster"
  pattern_type  = "LITERAL"
  principal     = "User:${confluentcloud_service_account.app-consumer.id}"
  host          = "*"
  operation     = "DESCRIBE"
  permission    = "ALLOW"

  credentials {
    # app-manager's credentials
    key    = "XXX_api_key_here_XXX"
    secret = "XXX_secret_here_XXX"
  }
}

Unfortunately api_key resource isn't ready yet but will be available soon 🤞 so you can't avoid this manual step at the moment. Later on, you'll be able to do it all in a single terraform apply.

Btw not sure whether it's helpful but just in case here's a complete set of ACLs you might need to create:

// Note that in order to consume from a topic, the principal of the consumer ('app-consumer' service account)
// needs to be authorized to perform 'READ' operation on both Topic and Group resources:
// confluentcloud_kafka_acl.app-consumer-read-on-topic, confluentcloud_kafka_acl.app-consumer-read-on-group.
// https://docs.confluent.io/platform/current/kafka/authorization.html#using-acls
resource "confluentcloud_kafka_acl" "app-consumer-read-on-topic" {
  kafka_cluster = confluentcloud_kafka_cluster.basic.id
  resource_type = "TOPIC"
  resource_name = confluentcloud_kafka_topic.orders.topic_name
  pattern_type  = "LITERAL"
  principal     = "User:${confluentcloud_service_account.app-consumer.id}"
  operation     = "READ"
  permission    = "ALLOW"
  http_endpoint = confluentcloud_kafka_cluster.basic.http_endpoint
  credentials {
    key    = confluentcloud_api_key.app-manager-kafka-api-key.id
    secret = confluentcloud_api_key.app-manager-kafka-api-key.secret
  }
}

resource "confluentcloud_kafka_acl" "app-consumer-read-on-group" {
  kafka_cluster = confluentcloud_kafka_cluster.basic.id
  resource_type = "GROUP"
  // The existing values of resource_name, pattern_type attributes are set up to match Confluent CLI's default consumer group ID ("confluent_cli_consumer_<uuid>").
  // https://docs.confluent.io/confluent-cli/current/command-reference/kafka/topic/confluent_kafka_topic_consume.html
  // Update the values of resource_name, pattern_type attributes to match your target consumer group ID.
  // https://docs.confluent.io/platform/current/kafka/authorization.html#prefixed-acls
  resource_name = "confluent_cli_consumer_"
  pattern_type  = "PREFIXED"
  principal     = "User:${confluentcloud_service_account.app-consumer.id}"
  operation     = "READ"
  permission    = "ALLOW"
  http_endpoint = confluentcloud_kafka_cluster.basic.http_endpoint
  credentials {
    key    = confluentcloud_api_key.app-manager-kafka-api-key.id
    secret = confluentcloud_api_key.app-manager-kafka-api-key.secret
  }
}

resource "confluentcloud_kafka_acl" "app-producer-write-to-topic" {
  kafka_cluster = confluentcloud_kafka_cluster.basic.id
  resource_type = "TOPIC"
  resource_name = confluentcloud_kafka_topic.orders.topic_name
  pattern_type  = "LITERAL"
  principal     = "User:${confluentcloud_service_account.app-producer.id}"
  operation     = "WRITE"
  permission    = "ALLOW"
  http_endpoint = confluentcloud_kafka_cluster.basic.http_endpoint
  credentials {
    key    = confluentcloud_api_key.app-manager-kafka-api-key.id
    secret = confluentcloud_api_key.app-manager-kafka-api-key.secret
  }
}

@ggmoy-tt did you have a chance to test it out again?

Yes, I could test it. I confirm that, following your instructions, it is working.

Thanks so much for your help!

I'm closing the issue because the underlying problem is resolved, NPE exception itself will be fixed in the next release 👌

@ggmoy-tt we're very excited to let you know we've just published a new version of TF Provider that includes a fix for this issue among other very exciting improvements: it enables fully automated provisioning of our key Kafka workflows (see the demo) with no more manual intervention and makes it our biggest and most impactful release.

The only gotcha we've renamed it from confluentinc/confluentcloud to confluentinc/confluent but we published a migration guide so it should be fairly straightforward. The existing confluentinc/confluentcloud will be deprecated soon so we'd recommend switching as soon as possible.

New confluentinc/confluent provider also includes a lot of sample configurations so you won't need to write them from scratch. You can find them here, find a full list of changes here.

confluentinc / terraform-provider-confluentcloud

Stack trace creating confluentcloud_kafka_acl resource #55

How to reproduce?

Additional information