Add Support for Custom Partitioning in Kafka Sinks via `PARTITION BY`

[bobbyiliev]

Feature request

Materialize has recently added support for custom partitioning in Kafka sinks using the PARTITION BY clause. This allows users to partition data based on specific columns, such as customer ID, ensuring that related data (e.g., orders for the same customer) are grouped into the same partition. This is particularly useful when working with upsert semantics to retain only the latest state of each record in a Kafka topic.

Proposed Solution

• Extend the materialize_sink_kafka resource in the Terraform provider to include a new argument for partition_by.
• The partition_by argument should accept a list of column names that will be used for partitioning Kafka sink data.
• Ensure that this feature integrates seamlessly with the existing upsert support in Kafka sinks, so that users can specify partitioning columns without disrupting current functionalities.
• Update the provider documentation to reflect the addition of the partition_by option, including examples of its usage.
• Add tests

resource "materialize_sink_kafka" "orders_kafka_sink" {
  name         = "orders_sink"
  kafka_connection {
    name = "kafka_connection"
  }
  topic        = "orders_topic"

  partition_by = ["customer_id"]  # New feature request

  # Additional configuration...
}

References

• Materialize Documentation on Custom Partitioning

[benesch]

One quick note: PARTITION BY accepts any SQL expression, not just column names. I don’t think it changes things for you too much, as you’ll still want to take the expression as a string, but you won’t want to escape it as an identifier like you normally would.