confluent_cluster_link provider settings

[pixie79]

Hi,

It would appear that the confluent_cluster_link & confluent_kafka_mirror_topic are not respecting the provider settings giving the kafka_rest_endpoint; instead, they appear to be getting this set for themselves.

This is not true for the topics endpoint correctly using the kafka_rest_endpoint in the provider settings.

This creates a problem as we must proxy the kafka_rest_endpoint via API GW so that Github's CI/CD can provision and maintain the private clusters. Hopefully, it is a quick and simple fix.

Thanks

Mark

[linouk23]

👋 @pixie79, thanks for reporting the issue!

It would appear that the confluent_cluster_link & confluent_kafka_mirror_topic are not respecting the provider settings giving the kafka_rest_endpoint; instead, they appear to be getting this set for themselves. This is not true for the topics endpoint correctly using the kafka_rest_endpoint in the provider settings.

This is accurate. The reason is confluent_cluster_link operates on 2 Kafka clusters so if kafka_rest_endpoint is set globally, it's unclear whether it corresponds to a source or a sink Kafka cluster.

Let me know if that makes sense!

[pixie79]

That makes sense but how do I over ride it to use the settings I give it, as GitHub can not talk to it directly on the internal IP and needs to go via a proxy. This works for configuring other reaources

[linouk23]

@pixie79 one thing that might help is to use a data source for Kafka cluster to infer its Kafka REST endpoint and then use it for proxy.

[pixie79]

@linouk23 I think you will see that is exactly what I am doing. The issue is that the Kafka REST endpoint can not reach Github's CI/CD runners. The route to there from Github is via an AWS API Gw Proxy. This is configured in the providers and overrides the endpoint correctly for all other resources. The same needs to be done for these two to enable me to correctly and securely provision the resource. I tried to set the destination cluster manually as an override pointing to the API Gw but that also failed. If not how can we override this so that for provisioning it can be overridden ?

resource "confluent_cluster_link" "this" {
  link_name = local.cluster_link_name
  source_kafka_cluster {
    id            = local.public_cluster_id
    rest_endpoint = local.public_kafka_rest_endpoint
    credentials {
      key    = local.public_cluster_api_key
      secret = local.public_cluster_api_secret
    }
  }

  destination_kafka_cluster {
    id                 = local.private_cluster_id
    bootstrap_endpoint = local.private_kafka_bootstrap_endpoint
    credentials {
      key    = local.private_cluster_api_key
      secret = local.private_cluster_api_secret
    }
  }
}

resource "confluent_kafka_mirror_topic" "this" {
  for_each = toset(local.confluent_kafka_mirror_topic_pp)
  source_kafka_topic {
    topic_name = each.value
  }
  cluster_link {
    link_name = local.cluster_link_name
  }
  kafka_cluster {
    id = local.private_cluster_id
    rest_endpoint = local.private_kafka_rest_endpoint
    credentials {
      key    = local.private_cluster_api_key
      secret = local.private_cluster_api_secret
    }
  }
}

data "confluent_schema_registry_cluster" "private" {
  id = local.schema_registry_id
  environment {
    id = local.environment_id
  }
}

locals {
  private_cluster_api_key          = data.aws_ssm_parameter.private_cluster_api_key_name.value
  private_cluster_api_secret       = data.aws_ssm_parameter.private_cluster_api_key_secret.value
  private_cluster_id               = data.aws_ssm_parameter.private_cluster_id.value
  private_kafka_rest_endpoint      = data.confluent_kafka_cluster.private.rest_endpoint
  private_kafka_bootstrap_endpoint = data.confluent_kafka_cluster.private.bootstrap_endpoint

  public_cluster_api_key          = data.aws_ssm_parameter.public_cluster_api_key_name.value
  public_cluster_api_secret       = data.aws_ssm_parameter.public_cluster_api_key_secret.value
  public_cluster_id               = data.aws_ssm_parameter.public_cluster_id.value
  public_kafka_rest_endpoint      = data.aws_ssm_parameter.public_cluster_rest_endpoint.value
}

provider "confluent" {
  cloud_api_key    = data.aws_ssm_parameter.cloud_api_key.value
  cloud_api_secret = data.aws_ssm_parameter.cloud_api_secret.value

  #  kafka_rest_endpoint = data.aws_ssm_parameter.private_cluster_endpoint.value
  kafka_rest_endpoint = var.private_rest_endpoint
  kafka_api_key       = data.aws_ssm_parameter.private_cluster_api_key_name.value
  kafka_api_secret    = data.aws_ssm_parameter.private_cluster_api_key_secret.value

  schema_registry_id            = data.aws_ssm_parameter.schema_registry_id.value
  schema_registry_rest_endpoint = data.aws_ssm_parameter.schema_registry_endpoint.value
  schema_registry_api_key       = local.schema_registry_key
  schema_registry_api_secret    = local.schema_registry_secret

}

[linouk23]

Oh I see yeah that's a challenging issue to address with our current design.

One solution that could help is that we could introduce another rest_endpoint as a oneOf(id, rest_endpoint). And then make TF Provider look at rest_endpoint first and use id if rest_endpoint is not available.

What do you think about it?

[pixie79]

Hi,

Yes I think that makes sence and would solve the issue with the both the cluster_link and mirror TF recourses.

Thanks

Mark

Sent with Proton Mail secure email.

------- Original Message ------- On Monday, March 6th, 2023 at 00:12, Kostya Linou @.***> wrote:

Oh I see yeah that's a challenging issue to address with our current design.

One solution that could help is that we could introduce another rest_endpoint as a oneOf(id, rest_endpoint). And then make TF Provider look at rest_endpoint first and use id if rest_endpoint is not available.

What do you think about it?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.[https://mail.proton.me/api/core/v4/images?Url=https%3A%2F%2Fgithub.com%2Fnotifications%2Fbeacon%2FAAECPECNGF3GXUMC2FXMVKTW2UTYHA5CNFSM6AAAAAAVNH55P2WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSWXV63S.gif&DryRun=0&UID=sdo7epine6zzgnggbowaqgeniopc7ijb]

[linouk23]

@pixie79 my team synced about this issue and our thinking is it might be a challenging patch to add to this already GA'ed resource that could cause new bugs or something so we might be looking at the next major version as a timeline here and we were wondering whether it's possible to white list URLs instead of using private / public approach instead.

[pixie79]

Hi

I don’t see how whitelisting would help as the issue is that that the call is being made from outside and needs to be proxied to be able to hit the private confluent resource. The call from GitHub CI/Cd can not talk directly to the private endpoint whitelisting will not help in hitting a private IP.

Regards

[linouk23]

Hmm interesting, is it essentially a CL between private source and private destination clusters?

[pixie79]

No one cluster is public the other is private. 

[pixie79]

Do you have an estimated timeline of when you might be able to update this resource currently? It is the only way to set up cluster-linking between a private and public cluster in two different environments. The UI does not appear to support this option. Each time I try after selecting my source private cluster, I then get told there are no destinations available. I know it works as the current TF can, but only if it runs from within a node that has access directly to the private cluster. Unfortunately, we are 100% serverless, so unable to have internal ec2 type nodes.

[linouk23]

It is the only way to set up cluster-linking between a private and public cluster in two different environments. The UI does not appear to support this option.

Just to confirm, have you tried using Confluent CLI client with this workaround?

[pixie79]

I can do this via the API if that helps

[linouk23]

@pixie79 by the way regarding

Do you have an estimated timeline of when you might be able to update this resource currently?

I'm not sure I'm in a position to share a timeline right now 😕

[DavidGamba]

I just ran into this same issue.

It is a big problem since it can't dynamically use a proxy. My workaround is to take the rest endpoint, put it in the /etc/hosts file pointing to 127.0.0.1 and then have the proxy listening on 127.0.0.1:443 then the proxy passes that single endpoint through the internal VPC.

This has a few limitations:

• I can't use custom ports, it can only be 443 which limits my ability to create/manage more than 1 cluster at a time.
• Since I can't set the proxy endpoint in the provider or the resource, I have to have super user access to modify the /etc/hosts file (tried dnsmasq but that also required super user access).

I am using provider aliases so I don't see why you need to worry about the provider affecting more than one resource:

provider "aws" {
  region = var.defaults["aws_region"]
}

data "aws_secretsmanager_secret" "confluent_api_key_pair" {
  name = "dev/confluent/aws-dev-svc-terraform-aws-dev-api-key-pair"
}

data "aws_secretsmanager_secret_version" "confluent_api_key_pair" {
  secret_id = data.aws_secretsmanager_secret.confluent_api_key_pair.id
}

provider "confluent" {
  cloud_api_key    = jsondecode(data.aws_secretsmanager_secret_version.confluent_api_key_pair.secret_string)["api_key"]
  cloud_api_secret = jsondecode(data.aws_secretsmanager_secret_version.confluent_api_key_pair.secret_string)["api_secret"]
}

provider "confluent" {
  alias = "kafka"

  cloud_api_key    = jsondecode(data.aws_secretsmanager_secret_version.confluent_api_key_pair.secret_string)["api_key"]
  cloud_api_secret = jsondecode(data.aws_secretsmanager_secret_version.confluent_api_key_pair.secret_string)["api_secret"]

  kafka_id            = data.confluent_kafka_cluster.this.id
  kafka_rest_endpoint = data.confluent_kafka_cluster.this.rest_endpoint
  # kafka_rest_endpoint = "127.0.0.1:8443"
  kafka_api_key    = jsondecode(data.aws_secretsmanager_secret_version.confluent_api_key_pair.secret_string)["api_key"]
  kafka_api_secret = jsondecode(data.aws_secretsmanager_secret_version.confluent_api_key_pair.secret_string)["api_secret"]
}

data "confluent_service_account" "this" {
  display_name = var.defaults["owner_sa"]
}

data "confluent_kafka_cluster" "this" {
  display_name = var.cluster_name
  environment {
    id = var.defaults["env_id"]
  }
}

resource "confluent_api_key" "ingestion" {
  provider = confluent.kafka

  display_name = "ingestion"
  description  = "Topic ingestion API key"

  owner {
    id          = data.confluent_service_account.this.id
    api_version = data.confluent_service_account.this.api_version
    kind        = data.confluent_service_account.this.kind
  }

  managed_resource {
    id          = data.confluent_kafka_cluster.this.id
    api_version = data.confluent_kafka_cluster.this.api_version
    kind        = data.confluent_kafka_cluster.this.kind

    environment {
      id = var.defaults["env_id"]
    }
  }
}