Kafka integration with AVRO format

[loicdelecroix]

Hello,

I'm trying to create a Kafka Integration (Confluent) with Avro as Data Format, but I'm keep having the same error. My Terraform Rockset integration is working fine, I already created a Kafka Integration without specifying the kafka_data_format.

Working resource

resource "rockset_kafka_integration" "kafka_cluster" {
  description = "Confluent integration (managed by terraform)"
  name        = "kafka-cluster"

  bootstrap_servers = split("SASL_SSL://", confluent_kafka_cluster.prod_europe_west1.bootstrap_endpoint)[1]

  schema_registry_config = {
    "key"    = var.rockset_kafka_schema_registry_key
    "secret" = var.rockset_kafka_schema_registry_secret
    "url"    = confluent_schema_registry_cluster.prod.rest_endpoint
  }

  security_config = {
    "api_key" = confluent_api_key.rockset_api_key.id
    "secret"  = confluent_api_key.rockset_api_key.secret
  }

  wait_for_integration = false
  use_v3               = true

  lifecycle {
    prevent_destroy = true
} 

Not working resource

resource "rockset_kafka_integration" "kafka_cluster_avro" {
  description = "Confluent integration (managed by terraform)"
  name        = "kafka-cluster-avro"
  kafka_data_format = "AVRO"
  bootstrap_servers = split("SASL_SSL://", confluent_kafka_cluster.prod_europe_west1.bootstrap_endpoint)[1]

  schema_registry_config = {
    "key"    = var.rockset_kafka_schema_registry_key
    "secret" = var.rockset_kafka_schema_registry_secret
    "url"    = confluent_schema_registry_cluster.prod.rest_endpoint
  }

  security_config = {
    "api_key" = confluent_api_key.rockset_api_key.id
    "secret"  = confluent_api_key.rockset_api_key.secret
  }

  wait_for_integration = false
  use_v3               = true

  lifecycle {
    prevent_destroy = true
} 

Terraform plan output

│ Error: Conflicting configuration arguments
│
│   with rockset_kafka_integration.kafka_cluster_avro,
│   on rockset_integration.tf line 66, in resource "rockset_kafka_integration" "kafka_cluster_avro":
│   66:   kafka_data_format = "AVRO"
│
│ "kafka_data_format": conflicts with security_config

[loicdelecroix]

FYI, the problem also happens if you try to specify topics :

│ Error: Conflicting configuration arguments
│
│   with rockset_kafka_integration. kafka_cluster_avro,
│   on rockset_integration.tf line 56, in resource "rockset_kafka_integration" "kafka_cluster_avro":
│   56:   kafka_topic_names = ["avro-topic"]
│
│ "kafka_topic_names": conflicts with security_config

[loicdelecroix]

Hello there,

I just had a meet with one of your co-worker (Kiril), and we saw that the problem wasn't coming from the integration, but from the collection creation.

When creating a collection in the UI, the payload contains the value sources[].format_params[].avro that could not be given in Terraform

{
  "name": "test",
  "retention_secs": null,
  "source_download_soft_limit_bytes": null,
  "field_mapping_query": {
    "sql": "SELECT *\nEXCEPT (_meta)\nFROM _input"
  },
  "sources": [
    {
      "integration_name": "kafka",
      "kafka": {
        "kafka_topic_name": "topic_name",
        "use_v3": true,
        "offset_reset_policy": "LATEST"
      },
      "format_params": {
        "avro": {}
      }
    }
  ],
  "storage_compression_type": "LZ4"
}

Do you want me to re-create a new issue, or do you prefer to use this one ?

[sbaldwin-rs]

Thanks for filing the issue, there is no need to re-create. We do intend on supporting this at some point, however I'm unsure of exact timelines.