search

tabular-io / iceberg-kafka-connect

Apache License 2.0
203 stars 46 forks source link

AWS Lake Formation - with STS Assume role #156

Open pixie79 opened 10 months ago

pixie79 commented 10 months ago

Hi,

Do you have an example Lake Formation role and policy set that works with the connector when the connector is assuming an AWS role in the account? We have been trying but seem to be getting a cryptic 400 error with access denied from Lake Formation. The role has DATA_LOCATION_ACCESS granted, along with ALL access to the Database and Tables. For Glue, S3 and KMS we have given full access for now. The KMS and S3 policies also specifically allow the role full access to (while we are testing).

Or could we be missing a config in the connector to make this work?

The connector is running in account A whilst the other AWS services are in account B (hence it assuming a role in account B)

Thanks

pixie79 commented 10 months ago

I have done some further investigation and it appears that the flag inside AWS Lakeformation for: AllowFullTableExternalDataAccess - (boolean) Whether to allow a third-party query engine to get data access credentials without session tags when a caller has full data access permissions. Needs to be set to enabled currently if LakeFormation is set to true in the Iceberg config (this is a setting we have to have set to true due to needing to use Iceberg)

Currently it looks like STS SessionTags are not being set by the connector correctly to be compatible with LakeFormation hence this override needing to be set.