memiiso / debezium-server-iceberg

Replicates any database (CDC events) to Apache Iceberg (To Cloud Storage)
Apache License 2.0
171 stars 35 forks source link

Iceberg Metadata Catalog Config #166

Open fritzwijaya opened 1 year ago

fritzwijaya commented 1 year ago

How to set the catalog implementation? Is there any docs for reference to set the config for supported metadata catalog?

ismailsimsek commented 1 year ago

@fritzwijaya its explained in the last config line

and example config is here

fritzwijaya commented 1 year ago

Thanks. Would take a look at it.

fritzwijaya commented 1 year ago

@ismailsimsek does it support to use Glue metadata catalog?

ismailsimsek commented 1 year ago

@fritzwijaya yes it supports it you just need to configure the catalog with your deployment. please feel free to post it here if you get any issue.

fritzwijaya commented 1 year ago

@ismailsimsek could you please share the sample config for set catalog impl?

ismailsimsek commented 1 year ago

@fritzwijaya glue catalog config should look like something below. additionally please make sure you have all the libraries and necessary access to glue

debezium.sink.iceberg.catalog-name=my_catalog
debezium.sink.iceberg.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
debezium.sink.iceberg.warehouse=s3a://my-bucket/iceberg_warehouse
debezium.sink.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO