getindata / kafka-connect-iceberg-sink

Apache License 2.0
77 stars 30 forks source link

iceberg.table-default.* properties seems not change anything #21

Closed nryanov closed 1 year ago

nryanov commented 1 year ago

Example connector configuration (configuration overrides doesn't matter, just an example):

{
        "connector.class": "com.getindata.kafka.connect.iceberg.sink.IcebergSink",
        "tasks.max": 1,
        "topics": "source-topic",
        "upsert": true,
        "upsert.keep-deletes": true,
        "table.auto-create": true,
        "table.namespace": "iceberg",
        "table.prefix": "prefix_",
        "iceberg.catalog-impl": "org.apache.iceberg.nessie.NessieCatalog",
        "iceberg.uri": "http://nessie:19120/api/v1",
        "iceberg.ref": "main",
        "iceberg.authentication.type": "NONE",
        "iceberg.warehouse": "s3://warehouse",
        "iceberg.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
        "iceberg.s3.endpoint": "http://minio:9000",
        "iceberg.s3.path-style-access": true,
        "iceberg.s3.access-key-id": "minioadmin",
        "iceberg.s3.secret-access-key": "minioadmin",
        "consumer.override.max.poll.records": 5000,
        "iceberg.table-default.write.format.default": "parquet",
        "iceberg.table-default.write.delete.mode": "merge-on-read",
        "iceberg.table-default.write.update.mode": "merge-on-read",
        "iceberg.table-default.write.merge.mode": "merge-on-read",
        "iceberg.table-default.write.metadata.previous-versions-max": 10,
        "iceberg.table-default.write.metadata.delete-after-commit.enabled": "true",
        "iceberg.table-default.commit.manifest.min-count-to-merge": 10
        }

Expected behaviour: configurations for write.*.mode were applied Result: still get copy-on-write

Connector version: https://github.com/getindata/kafka-connect-iceberg-sink/releases/tag/0.2.1