search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.14k stars 840 forks source link

pgcdc: remove batch_transactions #3023

Closed rockwotj closed 1 day ago

rockwotj commented 1 day ago

Just support emitting begin and commit txn markers, which then a custom batch policy for batching transactions can be created. This means that each message is a single row, which is a very simple mental model.

This still has the slow performance when not emitting commit txn markers so that we don't reprocess the last txn, and I can fix that in another PR by using a layer of indirection with the LSN that the plugin is acking and what we actually send to postgres.

rockwotj commented 1 day ago

Thanks @Jeffail! I'm sneaking in the docs updates from the benthos upgrade into this PR