search

tabular-io / iceberg-kafka-connect

Apache License 2.0
177 stars 32 forks source link

Iceberg Kafka Connect Support For GCP [GCS+Bigquery] #143

Closed devorbit closed 7 months ago

devorbit commented 8 months ago

Hi Team,

I wanted to quickly check if the connector could be used with GCP services like GCS and BigQuery.

I did not see any reference to that, and that got me worried. Also, I consider that BigQuery still doesn't have good support for open table formats and that could be one of the reasons.

Please share your thoughts.

Thanks Jay

bryanck commented 8 months ago

The sink can write to GCS and you can use the Iceberg Hive catalog to write to a HMS-compatible metastore. I think BigLake Metastore should work but I haven't personally tried that so feel free to open a ticket for any issues you find.

devorbit commented 8 months ago

Thank you @bryanck for quick response. I am facing some issues with the configurations. Can you share some thoughts on these configs?

"iceberg.catalog.uri": "http://localhost:19120/api/v1",
"iceberg.catalog.type": "hive"

Do I need to supply credentials?

bryanck commented 8 months ago

It looks like this is being worked on in Iceberg, see this PR: https://github.com/apache/iceberg/pull/7412. In the PR comments there is reference to a separate catalog library you can use until this is merged: https://cloud.google.com/bigquery/docs/iceberg-tables

bryanck commented 7 months ago

I'm going to close this, you can open a ticket in the Iceberg repo, as this is more of an Iceberg concern.