Deploy ksql queries using Terraform

[bluedog13]

It would be nice for Terraform to support the deployment of ksql queries without any manual steps involved. Opening this ticket to keep track for the requested feature.

Thanks.

[linouk23]

cc @bluedog13 @justrp @mtranter @agoel2 @oilegor1029 @serhiimakogon @sahu0817 @ssemyonov

Do you have a TF configuration snippet in mind how UX could look like? (see https://github.com/confluentinc/terraform-provider-confluent/issues/174#issuecomment-1387631565). Will support of confluent_ksql_stream, confluent_ksql_table be sufficient?

[burkematthew]

@linouk23 This is a rough snippet, but I think an example for a ksql Stream could look something like this:

resource "confluent_ksql_stream" "test-stream" {
  environment {
    id = confluent_environment.test.id
  }
  kafka_cluster {
    id = confluent_kafka_cluster.basic.id
  }
  credentials {
    key    = confluent_api_key.api-key.id
    secret = confluent_api_key.api-key.secret
  }

  // Options for source could be TOPIC vs. VIEW to differentiate between CREATE STREAM and CREATE STREAM AS SELECT
  source            = ""
  columns           = []
  serialization     = ""
  rowtime           = ""
  partitioning      = ""
  row_partitioning  = ""
  row_offset        = ""
  source_table      = true/false
  format            = ""
  kafka_topic       = ""
  key_format        = ""
  key_schema {
    id = confluent_key_schema.test_schema.id
  }
  partitions        = ""
  replicas          = ""
  timestamp         = ""
  timestamp_format  = ""
  value_delimiter   = ""
  value_format      = ""
  value_schema {
    id = confluent_value_schema.test_schema.id
  }
  wrap_single_value = true/false
  emit              = "changes"/"final"

  // For CREATE STREAM AS SELECT, maybe include some sort of a view_definition that allows for including the raw query definition.
  view_definition = "select users.name from users where users.disabled_at is false parition by users.id"
}

ksql tables would have a similar layout.

This feature would be immensely helpful for our team, especially as we try to distribute the workload of our Confluent implementation, so that we can better train on deploying new features and better control the deployment of them.

[S1M0NM]

Is there an update on this? We are also very interested in the feature

[linouk23]

Update: this feature hasn't been prioritized yet.

[burkematthew]

@linouk23 just wanted to follow-up here again, especially as I see the blocked label added a couple months ago.

[burkematthew]

Hi @linouk23 , just following up again. Is there anything that I can do to help keep this feature request moving?

[linouk23]

Update: this feature hasn't been prioritized yet.

[stfclv]

please let us know once available - another interested customer is looking for the same feature to maintain the ksql queries via TF.

[vmnaidu2018]

Please confirm when it would be available, ksql stream and ksql table through terraform

[linouk23]

Unfortunately, there are no new updates, @vmnaidu2018.

[Zico56]

Hello, In our company, we would be very interested by this feature too.

By the way, waiting for this to be implemented, does anyone know if there is an alternative TF provider or solution in order to manage ksql queries with TF ?

Thanks a lot.

[burkematthew]

Hi @linouk23 - any update on this request?

[linouk23]

Unfortunately, there are no new updates, @burkematthew.