search

replicase / pgcapture

A scalable Netflix DBLog implementation for PostgreSQL
Apache License 2.0
242 stars 31 forks source link

NATS jetstream support #75

Open tamalsaha opened 1 month ago

tamalsaha commented 1 month ago

Is it possible to publish these changes directly to a NATS jetsream?

rueian commented 1 month ago

Hi @tamalsaha, Apache Pulsar is currently the only implemented external sink and source.

A new sink and source can be implemented by embedding BaseSink and BaseSource. https://github.com/replicase/pgcapture/blob/1a64198cd12b987801cebbb0bd568f093272ced5/pkg/sink/main.go#L24-L31 https://github.com/replicase/pgcapture/blob/1a64198cd12b987801cebbb0bd568f093272ced5/pkg/source/main.go#L33-L40

rueian commented 1 month ago

Hi @tamalsaha, we are considering expanding the support of different messaging systems. Just curious, what makes you want to use NATS?

tamalsaha commented 1 month ago

We are coming from the cloud native / Kubernetes world. NATs is a very lightweight (compared to Kafka) message queue with its Jetstream feature can be used to implement Outbox pattern. NATs is a top level project for CNCF https://www.cncf.io/projects/nats/ . So, NATs integration will be a very welcome addition for us.

rueian commented 2 weeks ago

Just adding an important note here: NATs has a default 8MB message limit. We probably need to enlarge the limit if we store WALs into NATs.

Imtiaz246 commented 1 week ago

Hello @rueian, I’m interested in the progress of adding NATS jetstream as an external sink/source in pgcapture. Could you provide an update on recent developments or anticipated timelines for this feature?

rueian commented 1 week ago

Hi @Imtiaz24,

Thank you for your interest in the NATS integration but we don't have a timeline right now. We probably would like to have a integration for Kafka first.