search

getindata / kafka-connect-iceberg-sink

Apache License 2.0
76 stars 27 forks source link

AWS authentication fails with keys as part of sink configuration #13

Open gma2th-jt opened 1 year ago

gma2th-jt commented 1 year ago

With AWS access and secret key as part of the sink configuration, the connector does not read the keys and raises the following exception :

software.amazon.awssdk.services.glue.model.AccessDeniedException:
User: <server default user> is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-west-1:<id>:catalog because no identity-based policy allows the glue:GetTable action

Using the same keys as environment variables, the sink connector is able to connect to AWS.

Here is the config used :

    connector.class: "com.getindata.kafka.connect.iceberg.sink.IcebergSink"
    iceberg.catalog-impl: "org.apache.iceberg.aws.glue.GlueCatalog"
    iceberg.warehouse: "s3a://my-bucket"
    iceberg.fs.s3a.access.key: "<access key>"
    iceberg.fs.s3a.secret.key: "<secret key>"