search

confluentinc / terraform-provider-confluent

Terraform Provider for Confluent
Apache License 2.0
118 stars 61 forks source link

Unable to create confluent_role_binding without cloud api key #314

Closed Bjego closed 9 months ago

Bjego commented 9 months ago

Hey Confluent Team, we've purchased ConfluentCloud on Azure for our organisation. Due to the fact that development teams should create and grant kafkatopics on their own, we are facing an issue when terraform is trying to create a rolebinding.

I'm using the version 1.53 of the confluent provider.

terraform {
  required_providers {
    confluent = {
      source  = "confluentinc/confluent"
      version = "~> 1.0"
    }
  }
}

provider "confluent" {
  kafka_id            = var.kafka_id
  kafka_rest_endpoint = var.kafka_restapi
  kafka_api_key       = var.kafka_api_key  
  kafka_api_secret    = var.kafka_api_secret 
}

resource "confluent_kafka_topic" "topic" {
  topic_name       = "test.sometopic"
  partitions_count = 1
}

resource "confluent_role_binding" "topic_writer" {
  principal   = "User:sa-95XXXX"
  role_name   = "DeveloperWrite"
  crn_pattern = "crn://confluent.cloud/organization=${var.kafka_organisation_id}/environment=${var.kafka_environment_id}/cloud-cluster=${var.kafka_id}/kafka=${var.kafka_id}/topic=${confluent_kafka_topic.topic.topic_name}"
}

I'm always facing an issue "401 unauthorized", but the service account I'm using is RessourceOwner for topics and is already able to create the topic, only the rbac role is throwing issues.

module.topic_schadenereignis.confluent_role_binding.topic_writer: Creating...
╷
│ Error: error creating Role Binding: 401 Unauthorized: Unauthorized

Thank you!

Bjego commented 9 months ago

Here are my debug logs:

2023-09-25T15:38:14.858+0200 [INFO]  backend/local: apply calling Apply
2023-09-25T15:38:14.859+0200 [DEBUG] Building and walking apply graph for NormalMode plan
2023-09-25T15:38:14.860+0200 [DEBUG] Resource state not found for node "module.topic_schadenereignis.confluent_role_binding.topic_writer", instance module.topic_schadenereignis.confluent_role_binding.topic_writer
2023-09-25T15:38:14.861+0200 [DEBUG] ProviderTransformer: "module.topic_schadenereignis.confluent_role_binding.topic_writer (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/confluentinc/confluent"]
2023-09-25T15:38:14.862+0200 [DEBUG] ProviderTransformer: "module.topic_schadenereignis.confluent_role_binding.topic_writer" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/confluentinc/confluent"]
2023-09-25T15:38:14.862+0200 [DEBUG] ProviderTransformer: "module.topic_schadenereignis.confluent_kafka_topic.topic (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/confluentinc/confluent"]
2023-09-25T15:38:14.863+0200 [DEBUG] ReferenceTransformer: "var.kafka_restapi" references: []
2023-09-25T15:38:14.864+0200 [DEBUG] ReferenceTransformer: "var.kafka_id" references: []
2023-09-25T15:38:14.864+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis.confluent_role_binding.topic_writer" references: [module.topic_schadenereignis.var.kafka_organisation_id (expand) module.topic_schadenereignis.var.kafka_environment_id (expand) module.topic_schadenereignis.var.kafka_id (expand) module.topic_schadenereignis.var.kafka_id (expand) module.topic_schadenereignis.confluent_kafka_topic.topic (expand)]
2023-09-25T15:38:14.865+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis.confluent_role_binding.topic_writer (expand)" references: [] 
2023-09-25T15:38:14.866+0200 [DEBUG] ReferenceTransformer: "var.funk_environment" references: []
2023-09-25T15:38:14.866+0200 [DEBUG] ReferenceTransformer: "var.kafka_api_secret" references: []
2023-09-25T15:38:14.866+0200 [DEBUG] ReferenceTransformer: "var.kafka_cloud_api_secret" references: []
2023-09-25T15:38:14.867+0200 [DEBUG] ReferenceTransformer: "var.kafka_organisation_id" references: []
2023-09-25T15:38:14.867+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis.var.partition (expand)" references: []
2023-09-25T15:38:14.868+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis.confluent_kafka_topic.topic (expand)" references: []
2023-09-25T15:38:14.868+0200 [DEBUG] ReferenceTransformer: "var.kafka_environment_id" references: []
2023-09-25T15:38:14.869+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis.var.kafka_organisation_id (expand)" references: [var.kafka_organisation_id]
2023-09-25T15:38:14.870+0200 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/confluentinc/confluent\"]" references: [var.kafka_restapi var.kafka_api_key var.kafka_api_secret var.kafka_id]
2023-09-25T15:38:14.871+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis (expand)" references: []
2023-09-25T15:38:14.871+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis.var.topicname (expand)" references: []
2023-09-25T15:38:14.871+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis.var.kafka_environment_id (expand)" references: [var.kafka_environment_id]
2023-09-25T15:38:14.872+0200 [DEBUG] ReferenceTransformer: "var.kafka_api_key" references: []
2023-09-25T15:38:14.872+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis.var.kafka_id (expand)" references: [var.kafka_id]
2023-09-25T15:38:14.872+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis.var.funk_environment (expand)" references: [var.funk_environment]
2023-09-25T15:38:14.873+0200 [DEBUG] ReferenceTransformer: "module.topic_schadenereignis (close)" references: []
2023-09-25T15:38:14.874+0200 [DEBUG] ReferenceTransformer: "var.kafka_cloud_api_key" references: []
2023-09-25T15:38:14.874+0200 [DEBUG] ReferenceTransformer: "var.kafka_schema_registry_id" references: []
2023-09-25T15:38:14.878+0200 [DEBUG] pruneUnusedNodes: module.topic_schadenereignis.var.topicname (expand) is no longer needed, removing
2023-09-25T15:38:14.878+0200 [DEBUG] pruneUnusedNodes: module.topic_schadenereignis.var.funk_environment (expand) is no longer needed, removing       
2023-09-25T15:38:14.879+0200 [DEBUG] pruneUnusedNodes: module.topic_schadenereignis.var.partition (expand) is no longer needed, removing
2023-09-25T15:38:14.881+0200 [DEBUG] Starting graph walk: walkApply
2023-09-25T15:38:14.882+0200 [DEBUG] created provider logger: level=debug
2023-09-25T15:38:14.882+0200 [INFO]  provider: configuring client automatic mTLS
2023-09-25T15:38:14.892+0200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/confluentinc/confluent/1.53.0/windows_amd64/terraform-provider-confluent_1.53.0.exe args=[.terraform/providers/registry.terraform.io/confluentinc/confluent/1.53.0/windows_amd64/terraform-provider-confluent_1.53.0.exe]
2023-09-25T15:38:14.905+0200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/confluentinc/confluent/1.53.0/windows_amd64/terraform-provider-confluent_1.53.0.exe pid=29964
2023-09-25T15:38:14.906+0200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/confluentinc/confluent/1.53.0/windows_amd64/terraform-provider-confluent_1.53.0.exe
2023-09-25T15:38:15.188+0200 [INFO]  provider.terraform-provider-confluent_1.53.0.exe: configuring server automatic mTLS: timestamp=2023-09-25T15:38:15.187+0200
2023-09-25T15:38:15.197+0200 [DEBUG] provider.terraform-provider-confluent_1.53.0.exe: plugin address: address=127.0.0.1:10000 network=tcp timestamp=2023-09-25T15:38:15.197+0200
2023-09-25T15:38:15.197+0200 [DEBUG] provider: using plugin: version=5
2023-09-25T15:38:15.213+0200 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/confluentinc/confluent\"]" changed the config value, but that value is unused
2023-09-25T15:38:15.214+0200 [INFO]  provider.terraform-provider-confluent_1.53.0.exe: Initializing Terraform Provider for Confluent Cloud: @caller=src/github.com/confluentinc/terraform-provider-confluent/internal/provider/provider.go:322 @module=provider tf_provider_addr=provider tf_req_id=fcb43da9-d5ed-f8a7-687c-c18222aef83f tf_rpc=Configure timestamp=2023-09-25T15:38:15.214+0200
module.topic_schadenereignis.confluent_role_binding.topic_writer: Creating...
2023-09-25T15:38:15.216+0200 [INFO]  Starting apply for module.topic_schadenereignis.confluent_role_binding.topic_writer
2023-09-25T15:38:15.216+0200 [DEBUG] module.topic_schadenereignis.confluent_role_binding.topic_writer: applying the planned Create change
2023-09-25T15:38:15.217+0200 [DEBUG] provider.terraform-provider-confluent_1.53.0.exe: Creating new Role Binding: {"crn_pattern":"crn://confluent.cloud/organization=415596b0-1115-40a0-b5b7-43856ee1fbc1/environment=env-rvp1k/cloud-cluster=lkc-nw759z/kafka=lkc-nw759z/topic=nightly.kim_schaden.schadenereignis","principal":"User:sa-95mkz0","role_name":"DeveloperWrite"}: tf_provider_addr=provider tf_rpc=ApplyResourceChange @caller=src/github.com/confluentinc/terraform-provider-confluent/internal/provider/resource_role_binding.go:86 tf_req_id=5daeff54-23e8-70fa-eab9-17e1b177d87e tf_resource_type=confluent_role_binding @module=provider timestamp=2023-09-25T15:38:15.217+0200
2023-09-25T15:38:15.218+0200 [WARN]  provider.terraform-provider-confluent_1.53.0.exe: Could not find Cloud API Key: tf_provider_addr=provider tf_req_id=5daeff54-23e8-70fa-eab9-17e1b177d87e tf_resource_type=confluent_role_binding tf_rpc=ApplyResourceChange @caller=src/github.com/confluentinc/terraform-provider-confluent/internal/provider/utils.go:167 @module=provider timestamp=2023-09-25T15:38:15.217+0200
2023-09-25T15:38:15.219+0200 [WARN]  provider.terraform-provider-confluent_1.53.0.exe: Could not find Cloud API Key: @caller=src/github.com/confluentinc/terraform-provider-confluent/internal/provider/utils.go:167 @module=provider tf_resource_type=confluent_role_binding tf_provider_addr=provider tf_req_id=5daeff54-23e8-70fa-eab9-17e1b177d87e tf_rpc=ApplyResourceChange timestamp=2023-09-25T15:38:15.217+0200
2023-09-25T15:38:15.219+0200 [DEBUG] provider.terraform-provider-confluent_1.53.0.exe: 2023/09/25 15:38:15 [DEBUG] POST https://api.confluent.cloud/iam/v2/role-bindings
2023-09-25T15:38:16.040+0200 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2023-09-25T15:38:16.040+0200 [ERROR] vertex "module.topic_schadenereignis.confluent_role_binding.topic_writer" error: error creating Role Binding: 401 Unauthorized: Unauthorized
╷
│ Error: error creating Role Binding: 401 Unauthorized: Unauthorized
│
│   with module.topic_schadenereignis.confluent_role_binding.topic_writer,
│   on modules\topic\topic.tf line 6, in resource "confluent_role_binding" "topic_writer":
│    6: resource "confluent_role_binding" "topic_writer" {
│
╵
2023-09-25T15:38:16.050+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-09-25T15:38:16.068+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/confluentinc/confluent/1.53.0/windows_amd64/terraform-provider-confluent_1.53.0.exe pid=29964
2023-09-25T15:38:16.068+0200 [DEBUG] provider: plugin exited
linouk23 commented 9 months ago

@Bjego could you try

provider "confluent" {
  kafka_id            = var.kafka_id
  kafka_rest_endpoint = var.kafka_restapi
  kafka_api_key       = var.kafka_api_key  
  kafka_api_secret    = var.kafka_api_secret 
}

resource "confluent_kafka_topic" "topic" {
  topic_name       = "test.sometopic"
  partitions_count = 1
}

# Reference: https://developer.hashicorp.com/terraform/language/providers/configuration
provider "confluent" {
  alias  = "cloud"
  cloud_api_key = var.cloud_api_key
  cloud_api_secret = var.cloud_api_secret
}

resource "confluent_role_binding" "topic_writer" {
  provider = confluent.cloud
  principal   = "User:sa-95XXXX"
  role_name   = "DeveloperWrite"
  crn_pattern = "crn://confluent.cloud/organization=${var.kafka_organisation_id}/environment=${var.kafka_environment_id}/cloud-cluster=${var.kafka_id}/kafka=${var.kafka_id}/topic=${confluent_kafka_topic.topic.topic_name}"
}

and let us know whether it works?

Alternatively, you could do

provider "confluent" {
  kafka_id            = var.kafka_id
  kafka_rest_endpoint = var.kafka_restapi
  kafka_api_key       = var.kafka_api_key  
  kafka_api_secret    = var.kafka_api_secret 
  cloud_api_key = var.cloud_api_key
  cloud_api_secret = var.cloud_api_secret
}
Bjego commented 9 months ago

Hi @linouk23 , that works (I've tested option 2 on Friday already). But it still requires the cloud API keys. Is there any chance to set the roles without the keys? Best regards

linouk23 commented 9 months ago

Unfortunately Cloud API Key is a requirement for now @Bjego.