redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.14k stars 840 forks source link

snowpipe: batch schema migration #3008

Closed rockwotj closed 2 days ago

rockwotj commented 1 week ago

When first running a pipeline with heterogeneous data, you might need to do a lot of column additions. Support a batch path for updating columns so that we don't have to fallback to retrying the whole batch - also it makes the experience much faster.