Open bobbyiliev opened 1 month ago
@morsapaes @bobbyiliev let's discuss this PR at the sources & sinks meeting this week - we should decide when it makes sense to merge this - my thinking is we should do so whenever we move into private preview for the source versioning feature. But if we want to merge sooner and just have a disclaimer that the things mentioned as 'deprecated' here are not actually yet deprecated, that could work too
One thing that we can consider here as per this old tracking issue: https://github.com/MaterializeInc/terraform-provider-materialize/issues/391 is take the chance and decide if we still want to rename some of the attributes in the new source table resources:
Lists For attributes that use list, we have more cases of singular than plural. | Attribute | Resource | Type | Plural | Comment |
---|---|---|---|---|---|
start_offset | materialize_source_kafka | List of Strings | In Materialize the attribute is singular START OFFSET even though it is a list of strings |
||
header | materialize_source_kafka | List of Strings | In Materialize the attribute is singular HEADER even though it is a list of strings |
||
text_columns | materialize_source_postgres | List of Strings | X |
Blocks We had decided should be singular. There are some blocks that use plural so this could be a good chance to rename those attributes in the new source table load gen resource: | Attribute | Resource | Type | Plural |
---|---|---|---|---|
auction_options | materialize_source_load_generator | Block | X | |
counter_options | materialize_source_load_generator | Block | X | |
marketing_options | materialize_source_load_generator | Block | X | |
tpch_options | materialize_source_load_generator | Block | X | |
check_options | materialize_webhook | Block | X |
Initial implementation for the source versioning refactor as per #646
The main changes to consider:
table
attribute as optional and deprecated for both the MySQL and Postgres sourcesall_tables
bool attribute for the MySQL and the Loadgen sources, as in the past this was defaulting always usingFOR ALL TABLES
in the load gen sources (auction, marketing, tpch) and in the MySQL case whenever notable
blocks were defined, we defaulted toFOR ALL TABLES
. Thisall_tables
bool attribute allows us to create sources without any tables defined as per the source versioning workmaterialize_source_table_{mysql|postgres|load_generator}
resource which allows us to doCREATE TABLE ... FROM SOURCE ...
Things that are still pending: https://github.com/MaterializeInc/terraform-provider-materialize/issues/646