search

PeerDB-io / peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
https://peerdb.io
Other
2.26k stars 92 forks source link

Redshift Support? #641

Open woodhull opened 1 year ago

woodhull commented 1 year ago

I can't tell if this product will work with Redshift. Is it officially supported?

Does peerDB batch many small operations into larger chunks? In my experience trying to do lots of small inserts or updates into Redshift can cause serious performance issues at scale vs. a more batch style of data loading -- even if its once-per-minute.

serprex commented 1 year ago

No. Unfortunately our postgres connector code isn't compatible with postgres 8.0, so we'll have to make an explicit effort to support Redshift

saary commented 10 months ago

Is Redshift support on your roadmap? I would love to experiment with it.

serprex commented 10 months ago

Not in the immediate term (1-2 months). But we have that in the backlog for medium term (3-6 months).

In case someone else wants to take a stab it, the issue is that redshift lacks typarray which causes our queries on pg_type to fail

We use JSON types for syncing records before normalization, redshift lacks json types, so would have to make type on redshift text & use json functions https://docs.aws.amazon.com/redshift/latest/dg/json-functions.html