MaterializeInc / terraform-provider-materialize

A Terraform provider for Materialize
https://registry.terraform.io/providers/MaterializeInc/materialize
Mozilla Public License 2.0
11 stars 8 forks source link

`materialize_source_table_kafka`: Include `envelope_type`, `key_format` and `value_format` in read func #665

Open bobbyiliev opened 1 month ago

bobbyiliev commented 1 month ago

Feature request

@rjobanp has added envelope, key_format, and value_format columns to the mz_internal.mz_kafka_source_tables table in this PR.

We need to extend the logic in the read function within resource_source_table_kafka.go to introduce parsing and mapping logic for the complex envelope, key, and value Terraform schema structures.

The new table values look like:

materialize=> select * from mz_internal.mz_kafka_source_tables;
  id  | topic  |      envelope_type      | key_format | value_format 
------+--------+-------------------------+------------+--------------
 u333 | topic1 | none                    |            | 
 u529 | topic1 | upsert-value-err-inline | text       | text

We need to:

  1. Parse the envelope, key_format, and value_format columns from the mz_internal.mz_kafka_source_tables.
  2. Map these parsed values to the appropriate Terraform schema.
  3. Ensure the complex structures are accurately reflected in the resource state, especially for envelope_type, which may impact the resource behavior in Terraform.