search

PeerDB-io / peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
https://peerdb.io
Other
2.26k stars 92 forks source link

Support for Apache Iceberg #1830

Open mvgijssel opened 5 months ago

mvgijssel commented 5 months ago

Hi folks!

Not sure if this is the right place to ask, but are there plans to support Apache Iceberg natively? I understand it's possible to create a S3 peer to store files in the AVRO format, but this would still require an additional pipeline to convert those files for Apache Iceberg.

serprex commented 5 months ago

1792

mvgijssel commented 5 months ago

I need to learn how to search 🙈

iamKunalGupta commented 5 months ago

Hi there @mvgijssel! Just a quick question, what are you planning to use as the underlying iceberg catalog?

mvgijssel commented 5 months ago

I was contemplating loading it into Snowflake Iceberg tables!

iamKunalGupta commented 5 months ago

I was contemplating loading it into Snowflake Iceberg tables!

Are you planning to use the Snowflake Catalog or or use some other Catalog?

mvgijssel commented 5 months ago

Snowflake catalog sounds the easiest! If that doesn’t work we can run our own / check if AWS has something.

iamKunalGupta commented 5 months ago

1697

1869

1870