search

aws / aws-sdk-pandas

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
https://aws-sdk-pandas.readthedocs.io
Apache License 2.0
3.94k stars 702 forks source link

iam policies required for to_iceberg api #2644

Closed subtype1 closed 9 months ago

subtype1 commented 10 months ago

My lambda function execution role has all the necessary permissions to s3, glue and athena.

However, to_iceberg call is failing with access denied issues for s3 bucket.

what are the necessary permissions to get to_iceberg call working

jaidisido commented 10 months ago

As a general rule we do not provide IAM policies as we don't know the level of security required for your use case nor the specific resources involved. What is the specific Access Denied exception that is raised?

subtype1 commented 10 months ago

The exact exception I get is

"Access to the specified s3 resource is denied, please check your iam/s3 bucket policies and any relevant kms permission".

Problem is - the lambda execution role has all the necessary permissions and the error message is so very general and hence difficult to debug on where exactly the issue is

subtype1 commented 10 months ago

Also - just a general layout of the permissions will do or any pointers on how to debug this further will help as well

kukushking commented 10 months ago

@subtype1 Check LakeFormation permissions if it is enabled. It's common to miss lakefromation:GetDataAccess.