Postgres connector cannot resume if no events in CDC stream received

getdozer / dozer

Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.

https://getdozer.io

GNU Affero General Public License v3.0

1.47k stars 116 forks source link

Postgres connector cannot resume if no events in CDC stream received #2287

Open karolisg opened 6 months ago

karolisg commented 6 months ago

This bug happens when we don't receive any message in CDC stream after reading the initial snapshot of data in postgres database. In this case, we have LSN of publication slot, which can be used for snapshotting done event.

https://link.excalidraw.com/readonly/cBRn1ElaavxaTJtJ6CvP

chubei commented 6 months ago

Yes, we should send a Some(OpIdentifier::new(lsn, MAGIC_NUMBER)) to make it resumable once snapshotting is done.