Add Dedicated Cluster type with networking options

rjudin commented 3 years ago

Please add deployment.sku="DEDICATED" with networking"Internet", "PrivateLink", "VPCPeering"

resource "confluentcloud_kafka_cluster" "test" {
  name           = "provider-test"
  environment_id = confluentcloud_environment.env0.id
  #bootstrap_servers = string
  service_provider = "aws" # AWS/GCP
  region           = "ap-southeast-1"
  availability     = "LOW" # LOW(single-zone) or HIGH(multi-zone)
  storage          = 5000  # Storage limit(GB)
  network_ingress  = 100   #Network ingress limit(MBps)
  network_egress   = 100   #Network egress limit(MBps)
  deployment = {           #"Deployment settings.  Currently only `sku` is supported."
    sku = "DEDICATED"      #"BASIC"/"STANDARD"; "DEDICATED" - not supported yet

    # For sku="DEDICATED" only:
    networking       = "VPCPeering" # "Internet", "PrivateLink", "VPCPeering"
    cku = 1
    ## For sku="DEDICATED" && networking="VPCPeering"only:
    cidr_for_confluentcloud = "10.12.0.0/16"
  }
}

UPDATE 2020.10.08: seems go-client has it already - we need just port this arguments into tf-provider :)

rjudin commented 3 years ago

There is a related issue about missed cku: https://github.com/cgroschupp/go-client-confluent-cloud/issues/6

SimonBirds commented 3 years ago

+1 for both the ability to configure Dedicated Cluster networking and adding 'cku'.

grrywlsn commented 3 years ago

+1 for this. Trying to set up a dedicated cluster with VPC peering but can't do it with the provider yet.

Mongey commented 3 years ago

I think this is actually supported now 🤔

resource "confluentcloud_kafka_cluster" "test" {
  name             = "provider-test"
  service_provider = "aws"
  region           = "eu-west-1"
  availability     = "LOW"
  environment_id   = confluentcloud_environment.environment.id
  network_egress  = 100
  network_ingress = 100
  storage         = 5000

  deployment = {
    sku = "DEDICATED"
  }
  cku = 1
}

grrywlsn commented 3 years ago

I can provide sku = "DEDICATED" and that works, but the cluster is set to be public and that is an immutable property once the cluster is created. I want to provision with VPC peering only, which (at least from what I have tried) doesn't seem to be supported yet.

grrywlsn commented 3 years ago

I guess the TF provider isn't yet implementing the ClusterDeploymentNetworkAccess struct? https://github.com/cgroschupp/go-client-confluent-cloud/blob/master/confluentcloud/cluster.go#L36-L41

Marcus-James-Adams commented 3 years ago

Whilst I can't help on the code side of adding the networking options, I can assist on the testing debugging or Azure-based Private Link and peering if it helps

onobc commented 3 years ago

In response to the initial comment of this ticket... the Go client actually does not have support for this during create. It returns the networking info in the response from the create request but does not allow it to be specified on the create request.

A step further, the Confluent CLI does not support creating clusters w/ PrivateLink. Does the API even support this?

Marcus-James-Adams commented 3 years ago

@bono007 please excuse my ignorance as i am an infra guy rather than a dev - but does not this https://github.com/cgroschupp/go-client-confluent-cloud/blob/master/confluentcloud/cluster.go#L39 suggest that it is supported?

onobc commented 3 years ago

@Marcus-James-Adams 👋🏻

So at first glance I thought the same. However, that struct is not on the ClusterCreateConfig (aka the struct that is used for creation of the cluster) but rather in the return payload.

Marcus-James-Adams commented 3 years ago

Not sure if it will help get things moving but I've put a request in against the go client - https://github.com/cgroschupp/go-client-confluent-cloud/issues/14 If people could add their votes to it.

Marcus-James-Adams commented 3 years ago

Once again not being a dev but an infra guy i notice that confluent provide their own client https://github.com/confluentinc/confluent-kafka-go - Would it not make sense to wrap the terraform provider around the actual client, or is there a technical reason for using somebody elses?

amitkgupta commented 3 years ago

@Marcus-James-Adams https://github.com/confluentinc/confluent-kafka-go is a client for Kafka APIs, it enables doing things like producing messages to and consuming messages from a topic in a Kafka cluster. It works against any Kafka cluster, it is not specific in any way to Confluent Cloud.

https://github.com/cgroschupp/go-client-confluent-cloud/ is a client for Confluent Cloud APIs, it enables doing things like creating and deleting Kafka clusters on demand in Confluent Cloud. It is not for reading/writing data to a particular Kafka cluster.

The difference is analogous to using the aws CLI or SDK to create RDS PostgreSQL clusters, but using the psql CLI or SDK to SELECT or INSERT rows of data into a PostgreSQL cluster.

emmaLP commented 3 years ago

Need private link support. Although you can set dedicated, the cluster can only be public dedicated

emmaroberts-nbs commented 3 years ago

Any update on this please?

Marcus-James-Adams commented 2 years ago

Just as its been some time, is there any movement on this?

Mongey / terraform-provider-confluentcloud

Add Dedicated Cluster type with networking options #33