confluentinc / terraform-provider-confluent

Terraform Provider for Confluent
Apache License 2.0
118 stars 61 forks source link

How to select schema registry region on 2.0.0? #399

Closed Israphel closed 5 days ago

Israphel commented 5 days ago

I was reading this guide: https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/guides/version-2-upgrade

and with the confluent_schema_registry_cluster going away, I'm not sure how to create new environments with an specific config of schema registry cluster config (package, region).

This is part of my module:

# Schema registry region
data "confluent_schema_registry_region" "schema_registry_region" {
  cloud   = var.schema_registry.cloud
  region  = var.schema_registry.region
  package = var.schema_registry.package
}

# Schema registry cluster
resource "confluent_schema_registry_cluster" "schema_registry_cluster" {
  package = data.confluent_schema_registry_region.schema_registry_region.package

  environment {
    id = confluent_environment.environment.id
  }

  region {
    id = data.confluent_schema_registry_region.schema_registry_region.id
  }
}

Without the resource, how can I tell confluent in which region to create the schema registry?

Having only the data means that we can't decide wether to create the schema registry cluster or not? and where?

linouk23 commented 5 days ago

@Israphel thanks for creating the issue!

The short answer is

The Stream Governance cluster will be enabled in the same region as your first Kafka cluster within the environment.

We'll be releasing a new version of TF Provider with an updated code for confluent_kafka_cluster resource soon and we'll update docs & examples accordingly and we expect it to look as follows:

resource "confluent_environment" "prod" {
  display_name = "Production"

  stream_governance {
    package = "ESSENTIALS"
  }
}

resource "confluent_kafka_cluster" "prod" {
  ...
  region = "us-west-2"
}

data "confluent_schema_registry_cluster" "essentials" {
  environment {
    id = confluent_environment.prod.id
  }
  # region is going to be "us-west-2" as well
}
Israphel commented 5 days ago

but if I don't want that? my kafka clusters are us-east-1 and us-west-2, but my schema registry cluster is us-east-2

ConfluentSpencer commented 5 days ago

@Israphel , this is part of our Data Governance team moving customers away from managing Schema Registry Clusters in general. @akhileshm1 or @GunalKupta can comment more on details.

Israphel commented 5 days ago

we had to start with the schema registry in us-east-2 because us-east-1 had extra cost. What's the deal now? any region without extra cost?

akhileshm1 commented 4 days ago

Schema Registry (SR) cluster will always be provisioned in the same region as the first Kafka cluster in the environment. given this, you can choose to deploy the Kafka cluster in the region that you prefer your SR cluster to be in. you can always choose to delete your Kafka cluster if you don't need it in that region. this will not delete your SR cluster. in the event you already created the first Kafka cluster in a region you don't want your SR to be in and you don't want to re-create your environment, please reach out to Confluent support and we'll delete and re-provision the SR cluster in the region of your choosing. hope that helps.

Israphel commented 4 days ago

but about my previous question: In the Essentials package, only some regions had the SR without extra cost. That remains? or now any region is without extra cost?

Contacting support for a change seems to be adding extra steps to something that was easily done via IaC, the whole point of this provider.

akhileshm1 commented 4 days ago

both Essentials and Advanced packages are available in all Confluent Cloud regions.

Israphel commented 4 days ago

what I'm talking about is:

before this change, you created an environment and then a schema registry cluster. You used to get a list of regions to chose from and only some of them were without additional cost (like us-east-2), but us-east-1 had extra cost. Did that change? How do I know where to create the schema registry cluster without getting charged extra?

the only reason we have clusters in us-east-1 and us-west-2 but the schema registry cluster in us-east-2 is because of that very reason.

akhileshm1 commented 4 days ago

if you select Essentials package in your environment resource, your SR cluster will always be free (for first 100 schemas) regardless of the region. so, yes, Essentials package in us-east-1 is free. https://docs.confluent.io/cloud/current/stream-governance/packages.html#governance-package-types https://docs.confluent.io/cloud/current/stream-governance/packages.html#cloud-providers-and-region-support