Suppose there are multiple tables from a upstream database joined together via CREATE SINK INTO into a target table. When the upstream table adds a column, the target table needs to be altered accordingly.
This is is a crucial ability in data enrichment scenarios. We used to consider ALTER MATERIALIZED VIEW or something similar to tackle this issue, but it'd likely introduce substantial complexity. The semantic of MATERIALIZED VIEW inherently couples both the streaming logic and result table, and therefore being more difficult to alter.
"CREATE SINK INTO + adding columns in both target and source tables" decouples the result table and streaming logic, allowing the logic changes without affecting result serving. "CREATE SINK + altering columns in external systems like Snowflake or StarRocks" has been a widely adopted pattern for ETL pipelines. We can simply view the RisingWave target table as an external table.
Specific steps for this process:
ALTER TABLE ADD COLUMN in the source table.
DROP SINK and the sinking process will be temporarily paused.
Edited by @neverchanje
Here is the use case:
Suppose there are multiple tables from a upstream database joined together via CREATE SINK INTO into a target table. When the upstream table adds a column, the target table needs to be altered accordingly.
This is is a crucial ability in data enrichment scenarios. We used to consider ALTER MATERIALIZED VIEW or something similar to tackle this issue, but it'd likely introduce substantial complexity. The semantic of MATERIALIZED VIEW inherently couples both the streaming logic and result table, and therefore being more difficult to alter.
"CREATE SINK INTO + adding columns in both target and source tables" decouples the result table and streaming logic, allowing the logic changes without affecting result serving. "CREATE SINK + altering columns in external systems like Snowflake or StarRocks" has been a widely adopted pattern for ETL pipelines. We can simply view the RisingWave target table as an external table.
Specific steps for this process: