search

databricks / iceberg-kafka-connect

Apache License 2.0
220 stars 49 forks source link

Data not visible from Dremio #310

Open thnilamani opened 3 days ago

thnilamani commented 3 days ago

Hi,

With configuration below, I ran sync in local and saw data updated in Minio under bucket path warehouse/data. But when I tried to access from Dremio Nessie datasource, no data shown. If data are inserted through Dremio, it is displayed, and data are created under different folder in Minio. Not sure, this is something to do with Nessie version.

iceberg-connector-config.properties

....
iceberg.tables.dynamic-enabled=true
iceberg.tables.route-field=db_table
iceberg.tables.auto-create-enabled=true
iceberg.catalog.catalog-impl=org.apache.iceberg.nessie.NessieCatalog
iceberg.catalog.uri=http://nessie:19120/api/v2
iceberg.catalog.ref=main
iceberg.catalog.authentication.type=NONE
iceberg.catalog.warehouse=s3a://warehouse
iceberg.catalog.s3.endpoint=http://minio:9000
iceberg.catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
iceberg.catalog.client.region=us-east-1
iceberg.catalog.s3.path-style-access=true

iceberg.tables.cdc-field=_cdc_op
iceberg.table.systemtable.id-columns=node_uuid
iceberg.tables.upsert-mode-enabled=true
iceberg.table.systemtable.partition-by=cluster_uuid