search

akka / alpakka

Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
https://doc.akka.io/libraries/alpakka/current/
Other
1.26k stars 645 forks source link

PostgreSQL: Lightweight Change Data Capture / Event Log #889

Open sebastianharko opened 6 years ago

sebastianharko commented 6 years ago

On a branch in my fork, I have implemented a working, lightweight CDC (change data capture) for PostgreSQL. It is basically a source that streams changes from the database (and emits elements such as RowDeleted, RowUpdated, RowInserted (...) ) . It's basically conceptually similar to things such as Debezium(http://debezium.io), but way easier to get started with -> since it's just an Akka Stream Source :-) :-).

Again, it's very lightweight (zero dependencies) and uses the "logical_decoding" feature of PostgreSQL (PostgreSQL already comes with this feature, the user just has to enable it and it's very simple and has almost no performance overhead). PG v > 9.4 required though.

I'm curious if this is something that has a place in the Alpakka project (I'd think it does!!). If so, please let me know so I can PR my branch (already have proper docs btw).

Cheers!

ennru commented 6 years ago

Yes, that fits well in Alpakka. Would be interesting to see if this can be implemented for other "table-oriented" datastores, as well.

sebastianharko commented 6 years ago

@ennru Thanks, I'll make a PR. And to answer you questions yes it can be implemented for other "table-oriented" data stores as well (as a matter of fact , I also have a semi-functional MS SQL implementation of this same thing lying around somewhere ;-) ).