databricks / iceberg-kafka-connect

Apache License 2.0
219 stars 49 forks source link

Glue catalog Invalid table identifier #195

Closed big8extreme closed 9 months ago

big8extreme commented 9 months ago

Hello all, i have this issue [Worker-0b160bbbb97a222f7] java.lang.IllegalArgumentException: Invalid table identifier. this is my configuration:

tasks.max=1
topics=project-aggregated-dev
connector.class=io.tabular.iceberg.connect.IcebergSinkConnector
iceberg.catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
iceberg.catalog.client.region=eu-west-1
iceberg.catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
iceberg.catalog.warehouse=s3://project-dev-s3/project-catalog
iceberg.tables.schema-force-optional=true
iceberg.tables.evolve-schema-enabled=true
iceberg.tables.upsert-mode-enabled=true
iceberg.tables.default-id-columns=imo,timestamp
iceberg.tables.auto-create-enabled=true
iceberg.catalog=project-dev
iceberg.tables=one-minute-agg
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.storage.StringConverter

First of all, i need your help regarding some understanding is AWS Glue. do i have to create data catalog table or Stream schema registries

Capture d’écran 2024-02-21 à 18 00 29

I already created the two and its not working.

this is the error i get

[2024-02-21 17:02:38,509] ERROR [project-dev-1min-agg-sink-auto|task-0] WorkerSinkTask{id=project-dev-1min-agg-sink-auto-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted. Error: Invalid table identifier: project-dev.one-minute-agg (org.apache.kafka.connect.runtime.WorkerSinkTask:612)
 java.lang.IllegalArgumentException: Invalid table identifier: project-dev.one-minute-agg
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:218)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.<init>(BaseMetastoreCatalog.java:146)
at org.apache.iceberg.BaseMetastoreCatalog.buildTable(BaseMetastoreCatalog.java:94)
at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:71)
at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:93)
 .....
big8extreme commented 9 months ago

Solution find, for Glue schema registry, you have to create a table with only lowercase and nothing else.