connecting bottledwater with remote amazon rds postgres

confluentinc / bottledwater-pg

Change data capture from PostgreSQL into Kafka

http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/

Apache License 2.0

2 stars 0 forks source link

connecting bottledwater with remote amazon rds postgres #138

Open gyan opened 7 years ago

gyan commented 7 years ago

Is there a way to connect bottlewater with a remote db as amazon rds?

jong-eatsa commented 7 years ago

How can we capture CDC stream in a Postgres RDS instance in general? I need to capture this and send it to an AWS Kinesis Stream.

The only way I can see is to use AWS DMS continuous replication into an S3 target, then use a Lambda function to route this data into Kinesis. This seems rather convoluted (unlesss DMS can directly inject into Kinesis or trigger a Lambda)

ewencp commented 7 years ago

I believe anything that gets raw data from logical decoding requires an extra plugin since the only output plugin that ships with Postgres is simple test plugin. This is why both bottledwater and debezium have their own plugin.

The alternative would be to try to get changes indirectly, e.g. via the JDBC connector. This isn't as good as what a CDC connector can provide, but works generically across all databases with JDBC support, without requiring any modification of the database server itself.

mwarkentin commented 6 years ago

RDS postgres now supports wal2json output on postgres 9.5.7 and 9.6.3: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.Extensions

gunnarmorling commented 6 years ago

You might take a look at Debezium's Postgres connector. We also support wal2json as a logical decoding plug-in now.