confluentinc / terraform-provider-confluent

Terraform Provider for Confluent
Apache License 2.0
118 stars 61 forks source link

Not able to create PrivateLink access for AWS account #395

Open VipulSharma1999 opened 1 week ago

VipulSharma1999 commented 1 week ago

Problem statement: Trying to create a private link access from the different VPC but from the AWS account, but it's failing with the error.

╷
│ Error: error creating Private Link Access "": 409 Conflict: There are privatelinks with same realm and network, cannot create another private link with same attributes.
│
│   with confluent_private_link_access.aws,
│   on [main.tf](http://main.tf/) line 12, in resource "confluent_private_link_access" "aws":
│   12: resource "confluent_private_link_access" "aws" {
│
╵
╷
│ Error: creating Route53 Hosted Zone (**.us-west-2.aws.confluent.cloud): operation error Route 53: CreateHostedZone, https response error StatusCode: 400, RequestID: ****, ConflictingDomainExists: The VPC ** in region us-west-2 has already been associated with the hosted zone XYZ with the same domain name.
│
│   with aws_route53_zone.privatelink,
│   on [main.tf](http://main.tf/) line 84, in resource "aws_route53_zone" "privatelink":
│   84: resource "aws_route53_zone" "privatelink" {
│

Please help me with the same. Just to give some context: the dedicated cluster in my case has already a private link connectivity from one of the VPCs in the same AWS account and working properly, but while creating the AWS private link access from the different VPC under the same AWS account it's failing with this error.

Other piece of code that's used for creating private link.

data "confluent_environment" "env_id" {
  id = var.kafka_env_id
}

data "confluent_network" "private_link" {
  id = var.network_id
  environment {
    id = var.kafka_env_id
  }
}

resource "confluent_private_link_access" "aws" {
  display_name = "AWS-Private-Link-Access"
  aws {
    account = var.account_id
  }
  environment {
    id = data.confluent_environment.env_id.id
  }
  network {
    id = data.confluent_network.private_link.id
  }
}

# https://docs.confluent.io/cloud/current/networking/private-links/aws-privatelink.html
# Set up the VPC Endpoint for AWS PrivateLink in your AWS account
# Set up DNS records to use AWS VPC endpoints

data "aws_vpc" "privatelink" {
  id = var.vpc_id
}

data "aws_availability_zone" "privatelink" {
  for_each = var.subnets_to_privatelink
  zone_id  = each.key
}

resource "aws_security_group" "privatelink" {
  # Ensure that SG is unique, so that this module can be used multiple times within a single VPC
  name        = "cloud-privatelink_${var.env_name}_${var.vpc_id}"
  description = "Confluent Cloud Private Link minimal security group for ${var.account_id} in ${var.vpc_id}"
  vpc_id      = data.aws_vpc.privatelink.id

  ingress {
    description = "Allow all traffic"
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["[0.0.0.0/0](http://0.0.0.0/0)"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["[0.0.0.0/0](http://0.0.0.0/0)"]
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_vpc_endpoint" "privatelink" {
  vpc_id            = data.aws_vpc.privatelink.id
  service_name      = data.confluent_network.private_link.aws[0].private_link_endpoint_service
  vpc_endpoint_type = "Interface"

  security_group_ids = [
    aws_security_group.privatelink.id,
  ]

  subnet_ids          = [for zone, subnet_id in var.subnets_to_privatelink : subnet_id]
  private_dns_enabled = false

  depends_on = [
    confluent_private_link_access.aws,
  ]
}

resource "aws_route53_zone" "privatelink" {
  name = var.hosted_zone

  vpc {
    vpc_id = data.aws_vpc.privatelink.id
  }
}

resource "aws_route53_record" "privatelink" {
  count   = length(var.subnets_to_privatelink) == 1 ? 0 : 1
  zone_id = aws_route53_zone.privatelink.zone_id
  name    = "*.${[aws_route53_zone.privatelink.name](http://aws_route53_zone.privatelink.name/)}"
  type    = "CNAME"
  ttl     = "60"
  records = [
    aws_vpc_endpoint.privatelink.dns_entry[0]["dns_name"]
  ]
}

locals {
  endpoint_prefix = split(".", aws_vpc_endpoint.privatelink.dns_entry[0]["dns_name"])[0]
}

resource "aws_route53_record" "privatelink_zonal" {
  for_each = var.subnets_to_privatelink

  zone_id = aws_route53_zone.privatelink.zone_id
  name    = length(var.subnets_to_privatelink) == 1 ? "*" : "*.${each.key}"
  type    = "CNAME"
  ttl     = "60"
  records = [
    format("%s-%s%s",
      local.endpoint_prefix,
      data.aws_availability_zone.privatelink[each.key].name,
      replace(aws_vpc_endpoint.privatelink.dns_entry[0]["dns_name"], local.endpoint_prefix, "")
    )
  ]
}

terraform {
  backend "s3" {}
}

resource "aws_instance" "ec2_instance" {
  count                = var.create_ec2 ? 1 : 0
  ami                  = var.ami
  instance_type        = "t2.micro"
  monitoring           = true
  iam_instance_profile = var.iam_role

  root_block_device {
    volume_type           = "gp2"
    volume_size           = 30
    delete_on_termination = false
    encrypted             = false
  }

  vpc_security_group_ids = [aws_security_group.privatelink.id]
  subnet_id              = lookup(var.subnets_to_privatelink, "usw2-az1", "")

  lifecycle {
    create_before_destroy = true
  }

  user_data = file("${path.module}/userdata.sh")

  tags = {
    Name = "PROD_Confluent_Script_Runner"
  }
}

All the vars are passed via terragrunt as inputs and used in this module to run this terraform. @linouk23 please help me with the same.

VipulSharma1999 commented 1 week ago

@confluent-alec-kendall @ConfluentSpencer ^

linouk23 commented 1 week ago

👋 @VipulSharma1999 did you get a chance to look at dedicated-privatelink-aws-kafka-acls already?