search

data-dot-all / dataall

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.
https://data-dot-all.github.io/dataall/
Apache License 2.0
228 stars 82 forks source link

Insufficient Glue permissions to access table #588

Closed mvidhu closed 1 year ago

mvidhu commented 1 year ago

Describe the bug

Share request are failing with Insufficient glue permissions to access table error. Error: Batch Revoking [{'Id': 'da06539a-1429-4ef5-9213-7d03419f89ac', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'DatabaseName': 'db', 'Name': 'aggregates-xxxxxxxxxx', 'CatalogId': 'xxxxxxxxxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}] Batch Revoke response: {'ResponseMetadata': {'RequestId': '637122ea-faf4-4cdc-aa4f-607d7b313857', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:44:58 GMT', 'content-type': 'application/json', 'content-length': '625', 'connection': 'keep-alive', 'x-amzn-requestid': '637122ea-faf4-4cdc-aa4f-607d7b313857', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}, 'Failures': [{'RequestEntry': {'Id': 'da06539a-1429-4ef5-9213-7d03419f89ac', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-xxxxxxxxxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}, 'Error': {'ErrorCode': 'InvalidInputException', 'ErrorMessage': 'No permissions revoked. Grantee does not does not have:[ALL]'}}]} Successfully granted principal xxxxxxxxxx permissions ['DESCRIBE', 'SELECT'] to db.aggregates-xxxxxxxxxx: {'ResponseMetadata': {'RequestId': '6ae07e0b-2516-4661-af14-c5920bf2f617', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:00 GMT', 'content-type': 'application/json', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': '6ae07e0b-2516-4661-af14-c5920bf2f617', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}} Granted access to table aggregates-xxxxxxxxxx to external account xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Found resource_share_associations : [{'resourceShareArn': 'arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093', 'resourceShareName': 'LakeFormation-V3-TR5NX6NN4N', 'associatedEntity': 'arn:aws:glue:eu-west-1:xxxxxxxxxx:table/xxxxxxxxxx', 'associationType': 'RESOURCE', 'status': 'FAILED', 'creationTime': datetime.datetime(2023, 7, 19, 11, 45, 0, 179000, tzinfo=tzlocal()), 'lastUpdatedTime': datetime.datetime(2023, 7, 19, 11, 45, 1, 325000, tzinfo=tzlocal()), 'external': False}] Listing invitations for resourceShareArns: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Found 0 RAM invitations for resourceShareArn: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Remote boto3 session using pivot role for account= xxxxxxxxxx Creating ResourceLink aggregates-xxxxxxxxxx in database xxxxxxxxxx://xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table not found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'xxxxxxxxxx', 'tablename': 'aggregates-xxxxxxxxxx'} Successfully created ResourceLink aggregates-xxxxxxxxxx in database xxxxxxxxxx://xxxxxxxxxx Granted resource link DESCRIBE access to principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 on xxxxxxxxxx://xxxxxxxxxx/aggregates-xxxxxxxxxx Granted resource link DESCRIBE access to principal arn:aws:quicksight:eu-west-1:xxxxxxxxxx:group/default/dataall on xxxxxxxxxx://xxxxxxxxxx/aggregates-xxxxxxxxxx Failed granting principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 read access to resource link on target xxxxxxxxxx://xxxxxxxxxx due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-xxxxxxxxxx Resource Link {'Name': 'aggregates-xxxxxxxxxx', 'TargetTable': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-xxxxxxxxxx'}} was not created due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-xxxxxxxxxx Failed to share table aggregates-xxxxxxxxxx from source account xxxxxxxxxx//eu-west-1 with target account xxxxxxxxxx/eu-west-1due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-xxxxxxxxxx Triggering share failure alarm... Sending deployment failure notification Updating share item in DB hoc9wl41 status to Share_Failed Sharing table aggregates-production_unit_ammoniastorages... Updating share item in DB i7if7icb status to Share_In_Progress Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'db', 'tablename': 'aggregates-production_unit_ammoniastorages'} Remote boto3 session using pivot role for account= xxxxxxxxxx Revoking IAMAllowedGroups Super permission for table db|aggregates-production_unit_ammoniastorages Batch Revoking [{'Id': '727ea020-3fbc-4ae0-ab8b-42d9a11d3c6c', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'DatabaseName': 'db', 'Name': 'aggregates-production_unit_ammoniastorages', 'CatalogId': 'xxxxxxxxxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}] Batch Revoke response: {'ResponseMetadata': {'RequestId': '9feb2dc0-d631-4388-b5f3-529194b927a9', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:08 GMT', 'content-type': 'application/json', 'content-length': '635', 'connection': 'keep-alive', 'x-amzn-requestid': '9feb2dc0-d631-4388-b5f3-529194b927a9', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}, 'Failures': [{'RequestEntry': {'Id': '727ea020-3fbc-4ae0-ab8b-42d9a11d3c6c', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-production_unit_ammoniastorages'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}, 'Error': {'ErrorCode': 'InvalidInputException', 'ErrorMessage': 'No permissions revoked. Grantee does not does not have:[ALL]'}}]} Successfully granted principal xxxxxxxxxx permissions ['DESCRIBE', 'SELECT'] to db.aggregates-production_unit_ammoniastorages: {'ResponseMetadata': {'RequestId': 'd65eda16-f892-426a-adc9-c72d31488872', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:11 GMT', 'content-type': 'application/json', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': 'd65eda16-f892-426a-adc9-c72d31488872', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}} Granted access to table aggregates-production_unit_ammoniastorages to external account xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Found resource_share_associations : [{'resourceShareArn': 'arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093', 'resourceShareName': 'LakeFormation-V3-TR5NX6NN4N', 'associatedEntity': 'arn:aws:glue:eu-west-1:xxxxxxxxxx:table/db/aggregates-production_unit_ammoniastorages', 'associationType': 'RESOURCE', 'status': 'FAILED', 'creationTime': datetime.datetime(2023, 7, 19, 11, 45, 10, 952000, tzinfo=tzlocal()), 'lastUpdatedTime': datetime.datetime(2023, 7, 19, 11, 45, 11, 166000, tzinfo=tzlocal()), 'external': False}] Listing invitations for resourceShareArns: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Found 0 RAM invitations for resourceShareArn: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Remote boto3 session using pivot role for account= xxxxxxxxxx Creating ResourceLink aggregates-production_unit_ammoniastorages in database xxxxxxxxxx://xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table not found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'xxxxxxxxxx', 'tablename': 'aggregates-production_unit_ammoniastorages'} Successfully created ResourceLink aggregates-production_unit_ammoniastorages in database xxxxxxxxxx://xxxxxxxxxx Granted resource link DESCRIBE access to principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 on xxxxxxxxxx://xxxxxxxxxx/aggregates-production_unit_ammoniastorages Granted resource link DESCRIBE access to principal arn:aws:quicksight:eu-west-1:xxxxxxxxxx:group/default/dataall on xxxxxxxxxx://xxxxxxxxxx/aggregates-production_unit_ammoniastorages Failed granting principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 read access to resource link on target xxxxxxxxxx://db/aggregates-production_unit_ammoniastorages due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_ammoniastorages Resource Link {'Name': 'aggregates-production_unit_ammoniastorages', 'TargetTable': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-production_unit_ammoniastorages'}} was not created due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_ammoniastorages Failed to share table aggregates-production_unit_ammoniastorages from source account xxxxxxxxxx//eu-west-1 with target account xxxxxxxxxx/eu-west-1due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_ammoniastorages Triggering share failure alarm... Sending deployment failure notification Updating share item in DB i7if7icb status to Share_Failed Sharing table aggregates-production_unit_genericmaterialhandling... Updating share item in DB 0y3drs6b status to Share_In_Progress Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'db', 'tablename': 'aggregates-production_unit_genericmaterialhandling'} Remote boto3 session using pivot role for account= xxxxxxxxxx Revoking IAMAllowedGroups Super permission for table db|aggregates-production_unit_genericmaterialhandling Batch Revoking [{'Id': '920164a2-ebb0-4895-9967-96e10db39435', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'DatabaseName': 'db', 'Name': 'aggregates-production_unit_genericmaterialhandling', 'CatalogId': 'xxxxxxxxxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}] Batch Revoke response: {'ResponseMetadata': {'RequestId': '5905f442-5f72-4b2c-86ae-8afc5241a811', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:19 GMT', 'content-type': 'application/json', 'content-length': '643', 'connection': 'keep-alive', 'x-amzn-requestid': '5905f442-5f72-4b2c-86ae-8afc5241a811', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}, 'Failures': [{'RequestEntry': {'Id': '920164a2-ebb0-4895-9967-96e10db39435', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-production_unit_genericmaterialhandling'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}, 'Error': {'ErrorCode': 'InvalidInputException', 'ErrorMessage': 'No permissions revoked. Grantee does not does not have:[ALL]'}}]} Successfully granted principal xxxxxxxxxx permissions ['DESCRIBE', 'SELECT'] to db.aggregates-production_unit_genericmaterialhandling: {'ResponseMetadata': {'RequestId': 'b0ceec41-e2b9-4fd9-835d-d17cd747304d', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:21 GMT', 'content-type': 'application/json', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': 'b0ceec41-e2b9-4fd9-835d-d17cd747304d', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}} Granted access to table aggregates-production_unit_genericmaterialhandling to external account xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Found resource_share_associations : [{'resourceShareArn': 'arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093', 'resourceShareName': 'LakeFormation-V3-TR5NX6NN4N', 'associatedEntity': 'arn:aws:glue:eu-west-1:xxxxxxxxxx:table/db/aggregates-production_unit_genericmaterialhandling', 'associationType': 'RESOURCE', 'status': 'FAILED', 'creationTime': datetime.datetime(2023, 7, 19, 11, 45, 20, 980000, tzinfo=tzlocal()), 'lastUpdatedTime': datetime.datetime(2023, 7, 19, 11, 45, 21, 229000, tzinfo=tzlocal()), 'external': False}] Listing invitations for resourceShareArns: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Found 0 RAM invitations for resourceShareArn: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Remote boto3 session using pivot role for account= xxxxxxxxxx Creating ResourceLink aggregates-production_unit_genericmaterialhandling in database xxxxxxxxxx://xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table not found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'xxxxxxxxxx', 'tablename': 'aggregates-production_unit_genericmaterialhandling'} Successfully created ResourceLink aggregates-production_unit_genericmaterialhandling in database xxxxxxxxxx://xxxxxxxxxx Granted resource link DESCRIBE access to principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 on xxxxxxxxxx://xxxxxxxxxx/aggregates-production_unit_genericmaterialhandling Granted resource link DESCRIBE access to principal arn:aws:quicksight:eu-west-1:xxxxxxxxxx:group/default/dataall on xxxxxxxxxx://xxxxxxxxxx/aggregates-production_unit_genericmaterialhandling Failed granting principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 read access to resource link on target xxxxxxxxxx://db/aggregates-production_unit_genericmaterialhandling due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_genericmaterialhandling Resource Link {'Name': 'aggregates-production_unit_genericmaterialhandling', 'TargetTable': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-production_unit_genericmaterialhandling'}} was not created due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_genericmaterialhandling Failed to share table aggregates-production_unit_genericmaterialhandling from source account xxxxxxxxxx//eu-west-1 with target account xxxxxxxxxx/eu-west-1due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_genericmaterialhandling Triggering share failure alarm... Sending deployment failure notification Updating share item in DB 0y3drs6b status to Share_Failed Sharing table aggregates-xxx... Updating share item in DB 3wywrutq status to Share_In_Progress Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'db', 'tablename': 'aggregates-xxx'} Remote boto3 session using pivot role for account= xxxxxxxxxx Revoking IAMAllowedGroups Super permission for table db|aggregates-xxx Batch Revoking [{'Id': '306566e2-c00c-4850-8e00-7db838b57970', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'DatabaseName': 'db', 'Name': 'aggregates-xxx', 'CatalogId': 'xxxxxxxxxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}] Batch Revoke response: {'ResponseMetadata': {'RequestId': '2a894280-e0d9-420a-b2c0-a4c26b1ab86c', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:29 GMT', 'content-type': 'application/json', 'content-length': '623', 'connection': 'keep-alive', 'x-amzn-requestid': '2a894280-e0d9-420a-b2c0-a4c26b1ab86c', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}, 'Failures': [{'RequestEntry': {'Id': '306566e2-c00c-4850-8e00-7db838b57970', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-xxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}, 'Error': {'ErrorCode': 'InvalidInputException', 'ErrorMessage': 'No permissions revoked. Grantee does not does not have:[ALL]'}}]} Successfully granted principal xxxxxxxxxx permissions ['DESCRIBE', 'SELECT'] to db.aggregates-xxx: {'ResponseMetadata': {'RequestId': '03133308-20fa-4ad1-a65f-0ad98b5a9454', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:31 GMT', 'content-type': 'application/json', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': '03133308-20fa-4ad1-a65f-0ad98b5a9454', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}} Granted access to table aggregates-xxx to external account xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Found resource_share_associations : [{'resourceShareArn': 'arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093', 'resourceShareName': 'LakeFormation-V3-TR5NX6NN4N', 'associatedEntity': 'arn:aws:glue:eu-west-1:xxxxxxxxxx:table/db/aggregates-xxx', 'associationType': 'RESOURCE', 'status': 'FAILED', 'creationTime': datetime.datetime(2023, 7, 19, 11, 45, 31, 373000, tzinfo=tzlocal()), 'lastUpdatedTime': datetime.datetime(2023, 7, 19, 11, 45, 31, 832000, tzinfo=tzlocal()), 'external': False}] Listing invitations for resourceShareArns: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Found 0 RAM invitations for resourceShareArn: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Remote boto3 session using pivot role for account= xxxxxxxxxx Creating ResourceLink aggregates-xxx in database xxxxxxxxxx://xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table not found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'xxxxxxxxxx', 'tablename': 'aggregates-xxx'} Successfully created ResourceLink aggregates-xxx in database xxxxxxxxxx://xxxxxxxxxx Granted resource link DESCRIBE access to principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 on xxxxxxxxxx://xxxxxxxxxx/aggregates-xxx Granted resource link DESCRIBE access to principal arn:aws:quicksight:eu-west-1:xxxxxxxxxx:group/default/dataall on xxxxxxxxxx://xxxxxxxxxx/aggregates-xxx Failed granting principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 read access to resource link on target xxxxxxxxxx://db/aggregates-xxx due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-xxx Resource Link {'Name': 'aggregates-xxx', 'TargetTable': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-xxx'}} was not created due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-xxx Failed to share table aggregates-xxx from source account xxxxxxxxxx//eu-west-1 with target account xxxxxxxxxx/eu-west-1due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-xxx Triggering share failure alarm... Sending deployment failure notification Updating share item in DB 3wywrutq status to Share_Failed Sharing table aggregates-xxx... Updating share item in DB ge7c11el status to Share_In_Progress Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'db', 'tablename': 'aggregates-xxx'} Remote boto3 session using pivot role for account= xxxxxxxxxx Revoking IAMAllowedGroups Super permission for table db|aggregates-xxx Batch Revoking [{'Id': '09642459-8237-4025-b982-eb084f10bc95', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'DatabaseName': 'db', 'Name': 'aggregates-xxx', 'CatalogId': 'xxxxxxxxxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}] Batch Revoke response: {'ResponseMetadata': {'RequestId': '623b4489-e329-4ce7-8ac1-cc05379b4eb8', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:39 GMT', 'content-type': 'application/json', 'content-length': '631', 'connection': 'keep-alive', 'x-amzn-requestid': '623b4489-e329-4ce7-8ac1-cc05379b4eb8', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}, 'Failures': [{'RequestEntry': {'Id': '09642459-8237-4025-b982-eb084f10bc95', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-xxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}, 'Error': {'ErrorCode': 'InvalidInputException', 'ErrorMessage': 'No permissions revoked. Grantee does not does not have:[ALL]'}}]} Successfully granted principal xxxxxxxxxx permissions ['DESCRIBE', 'SELECT'] to db.aggregates-xxx: {'ResponseMetadata': {'RequestId': '45c7ad23-7ccc-47f8-b2c2-a92debbad3c3', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:41 GMT', 'content-type': 'application/json', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': '45c7ad23-7ccc-47f8-b2c2-a92debbad3c3', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}} Granted access to table aggregates-xxx to external account xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Found resource_share_associations : [{'resourceShareArn': 'arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093', 'resourceShareName': 'LakeFormation-V3-TR5NX6NN4N', 'associatedEntity': 'arn:aws:glue:eu-west-1:xxxxxxxxxx:table/db/aggregates-xxx', 'associationType': 'RESOURCE', 'status': 'FAILED', 'creationTime': datetime.datetime(2023, 7, 19, 11, 45, 41, 181000, tzinfo=tzlocal()), 'lastUpdatedTime': datetime.datetime(2023, 7, 19, 11, 45, 41, 565000, tzinfo=tzlocal()), 'external': False}] Listing invitations for resourceShareArns: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Found 0 RAM invitations for resourceShareArn: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Remote boto3 session using pivot role for account= xxxxxxxxxx Creating ResourceLink aggregates-xxx in database xxxxxxxxxx://xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table not found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'xxxxxxxxxx', 'tablename': 'aggregates-xxx'} Successfully created ResourceLink aggregates-xxx in database xxxxxxxxxx://xxxxxxxxxx Granted resource link DESCRIBE access to principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 on xxxxxxxxxx://xxxxxxxxxx/aggregates-xxx Granted resource link DESCRIBE access to principal arn:aws:quicksight:eu-west-1:xxxxxxxxxx:group/default/dataall on xxxxxxxxxx://xxxxxxxxxx/aggregates-xxx Failed granting principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 read access to resource link on target xxxxxxxxxx://db/aggregates-xxx due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-xxx Resource Link {'Name': 'aggregates-xxx', 'TargetTable': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-xxx'}} was not created due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-xxx Failed to share table aggregates-xxx from source account xxxxxxxxxx//eu-west-1 with target account xxxxxxxxxx/eu-west-1due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-xxx Triggering share failure alarm... Sending deployment failure notification Updating share item in DB ge7c11el status to Share_Failed Sharing table aggregates-production_unit_na4... Updating share item in DB y9wckb33 status to Share_In_Progress Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'db', 'tablename': 'aggregates-production_unit_na4'} Remote boto3 session using pivot role for account= xxxxxxxxxx Revoking IAMAllowedGroups Super permission for table db|aggregates-production_unit_na4 Batch Revoking [{'Id': 'e381fa96-730b-455b-b552-b1a80f3e9ee5', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'DatabaseName': 'db', 'Name': 'aggregates-production_unit_na4', 'CatalogId': 'xxxxxxxxxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}] Batch Revoke response: {'ResponseMetadata': {'RequestId': 'cd30d20e-7ac7-438a-b216-7220d254c86a', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:49 GMT', 'content-type': 'application/json', 'content-length': '623', 'connection': 'keep-alive', 'x-amzn-requestid': 'cd30d20e-7ac7-438a-b216-7220d254c86a', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}, 'Failures': [{'RequestEntry': {'Id': 'e381fa96-730b-455b-b552-b1a80f3e9ee5', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-production_unit_na4'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}, 'Error': {'ErrorCode': 'InvalidInputException', 'ErrorMessage': 'No permissions revoked. Grantee does not does not have:[ALL]'}}]} Successfully granted principal xxxxxxxxxx permissions ['DESCRIBE', 'SELECT'] to db.aggregates-production_unit_na4: {'ResponseMetadata': {'RequestId': 'e8e62e2a-aa89-4dd9-be4e-a7ed16474c29', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:51 GMT', 'content-type': 'application/json', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': 'e8e62e2a-aa89-4dd9-be4e-a7ed16474c29', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}} Granted access to table aggregates-production_unit_na4 to external account xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Found resource_share_associations : [{'resourceShareArn': 'arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093', 'resourceShareName': 'LakeFormation-V3-TR5NX6NN4N', 'associatedEntity': 'arn:aws:glue:eu-west-1:xxxxxxxxxx:table/db/aggregates-production_unit_na4', 'associationType': 'RESOURCE', 'status': 'FAILED', 'creationTime': datetime.datetime(2023, 7, 19, 11, 45, 50, 917000, tzinfo=tzlocal()), 'lastUpdatedTime': datetime.datetime(2023, 7, 19, 11, 45, 51, 400000, tzinfo=tzlocal()), 'external': False}] Listing invitations for resourceShareArns: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Found 0 RAM invitations for resourceShareArn: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Remote boto3 session using pivot role for account= xxxxxxxxxx Creating ResourceLink aggregates-production_unit_na4 in database xxxxxxxxxx://xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table not found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'xxxxxxxxxx', 'tablename': 'aggregates-production_unit_na4'} Successfully created ResourceLink aggregates-production_unit_na4 in database xxxxxxxxxx://xxxxxxxxxx Granted resource link DESCRIBE access to principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 on xxxxxxxxxx://xxxxxxxxxx/aggregates-production_unit_na4 Granted resource link DESCRIBE access to principal arn:aws:quicksight:eu-west-1:xxxxxxxxxx:group/default/dataall on xxxxxxxxxx://xxxxxxxxxx/aggregates-production_unit_na4 Failed granting principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 read access to resource link on target xxxxxxxxxx://db/aggregates-production_unit_na4 due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_na4 Resource Link {'Name': 'aggregates-production_unit_na4', 'TargetTable': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-production_unit_na4'}} was not created due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_na4 Failed to share table aggregates-production_unit_na4 from source account xxxxxxxxxx//eu-west-1 with target account xxxxxxxxxx/eu-west-1due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_na4 Triggering share failure alarm... Sending deployment failure notification Updating share item in DB y9wckb33 status to Share_Failed Sharing table aggregates-production_unit_npk1... Updating share item in DB rx401wu9 status to Share_In_Progress Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'db', 'tablename': 'aggregates-production_unit_npk1'} Remote boto3 session using pivot role for account= xxxxxxxxxx Revoking IAMAllowedGroups Super permission for table db|aggregates-production_unit_npk1 Batch Revoking [{'Id': '05ea1f3a-3bab-4da3-bfe3-e3c6a8e312d4', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'DatabaseName': 'db', 'Name': 'aggregates-production_unit_npk1', 'CatalogId': 'xxxxxxxxxx'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}] Batch Revoke response: {'ResponseMetadata': {'RequestId': '3677b320-2ef3-46b5-9e12-90a91b1ebadd', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:45:59 GMT', 'content-type': 'application/json', 'content-length': '624', 'connection': 'keep-alive', 'x-amzn-requestid': '3677b320-2ef3-46b5-9e12-90a91b1ebadd', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}, 'Failures': [{'RequestEntry': {'Id': '05ea1f3a-3bab-4da3-bfe3-e3c6a8e312d4', 'Principal': {'DataLakePrincipalIdentifier': 'EVERYONE'}, 'Resource': {'Table': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-production_unit_npk1'}}, 'Permissions': ['ALL'], 'PermissionsWithGrantOption': []}, 'Error': {'ErrorCode': 'InvalidInputException', 'ErrorMessage': 'No permissions revoked. Grantee does not does not have:[ALL]'}}]} Successfully granted principal xxxxxxxxxx permissions ['DESCRIBE', 'SELECT'] to db.aggregates-production_unit_npk1: {'ResponseMetadata': {'RequestId': 'fab29e1e-f4cf-4ae4-bfeb-0a8c44376ba1', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 19 Jul 2023 11:46:01 GMT', 'content-type': 'application/json', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': 'fab29e1e-f4cf-4ae4-bfeb-0a8c44376ba1', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}} Granted access to table aggregates-production_unit_npk1 to external account xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Found resource_share_associations : [{'resourceShareArn': 'arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093', 'resourceShareName': 'LakeFormation-V3-TR5NX6NN4N', 'associatedEntity': 'arn:aws:glue:eu-west-1:xxxxxxxxxx:table/db/aggregates-production_unit_npk1', 'associationType': 'RESOURCE', 'status': 'FAILED', 'creationTime': datetime.datetime(2023, 7, 19, 11, 46, 0, 782000, tzinfo=tzlocal()), 'lastUpdatedTime': datetime.datetime(2023, 7, 19, 11, 46, 1, 184000, tzinfo=tzlocal()), 'external': False}] Listing invitations for resourceShareArns: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Found 0 RAM invitations for resourceShareArn: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093'] Remote boto3 session using pivot role for account= xxxxxxxxxx Creating ResourceLink aggregates-production_unit_npk1 in database xxxxxxxxxx://xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Remote boto3 session using pivot role for account= xxxxxxxxxx Glue table not found: {'accountid': 'xxxxxxxxxx', 'region': 'eu-west-1', 'database': 'xxxxxxxxxx', 'tablename': 'aggregates-production_unit_npk1'} Successfully created ResourceLink aggregates-production_unit_npk1 in database xxxxxxxxxx://xxxxxxxxxx Granted resource link DESCRIBE access to principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 on xxxxxxxxxx://xxxxxxxxxx/aggregates-production_unit_npk1 Granted resource link DESCRIBE access to principal arn:aws:quicksight:eu-west-1:xxxxxxxxxx:group/default/dataall on xxxxxxxxxx://xxxxxxxxxx/aggregates-production_unit_npk1 Failed granting principal arn:aws:iam::xxxxxxxxxx:role/dataall-commons-lrcy2988 read access to resource link on target xxxxxxxxxx://db/aggregates-production_unit_npk1 due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_npk1 Resource Link {'Name': 'aggregates-production_unit_npk1', 'TargetTable': {'CatalogId': 'xxxxxxxxxx', 'DatabaseName': 'db', 'Name': 'aggregates-production_unit_npk1'}} was not created due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_npk1 Failed to share table aggregates-production_unit_npk1 from source account xxxxxxxxxx//eu-west-1 with target account xxxxxxxxxx/eu-west-1due to: An error occurred (AccessDeniedException) when calling the GrantPermissions operation: Insufficient Glue permissions to access table aggregates-production_unit_npk1 Triggering share failure alarm... Sending deployment failure notification Updating share item in DB rx401wu9 status to Share_Failed sharing tables succeeded = False Updating share object ffrols6v in DB from Share_In_Progress to state Processed Sharing task finished successfully

How to Reproduce

*P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.*

User has imported a data set from S3 to data.all with pre-defined glue database and tables. Team B requested access to dataset and all the tables. Once share request is approved, sharing tables failed with Insufficient Glue permission error.

Expected behavior

Share request should be completed.

Your project

No response

Screenshots

No response

OS

Mac

Python version

3.11

AWS data.all version

1.5.4

Additional context

I have verified the below before raising the ticket

dlpzx commented 1 year ago

Hi @mvidhu thanks for opening the issue. I see that the root cause of the missing Glue permissions is that the RAM invitation failed:

Found resource_share_associations : [{'resourceShareArn': 'arn:aws🐏eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093', 'resourceShareName': 'LakeFormation-V3-TR5NX6NN4N', 'associatedEntity': 'arn:aws:glue:eu-west-1:xxxxxxxxxx:table/xxxxxxxxxx', 'associationType': 'RESOURCE', 'status': 'FAILED', 'creationTime': datetime.datetime(2023, 7, 19, 11, 45, 0, 179000, tzinfo=tzlocal()), 'lastUpdatedTime': datetime.datetime(2023, 7, 19, 11, 45, 1, 325000, tzinfo=tzlocal()), 'external': False}]

As a result, when we try to accept the RAM invitation from the target we do not see any invitation, we do not accept it and there won't be the required permissions in Glue.

Found 0 RAM invitations for resourceShareArn...

RAM is a little tricky to debug but there are a couple of things we can do:

If the above do not work, let's go ahead and schedule a call to debug and use CloudTrail and CloudWatch together. Bests,

mvidhu commented 1 year ago

@dlpzx Thanks for the quick reply. I verified the pivot Role has 'ram:UpdateResourceShare' and 'ram:DeleteResourceShare' permissions and I deleted RAM resource and retried share, but still share failed.

dlpzx commented 1 year ago

After debugging we could conclude that the root issue was on the creation of the RAM invitations. In the CloudWatch logs they appeared as 'Failed' and later when data.all tries to accept the invitation from the target account it cannot find any missing invitation. The calls to RAM are effectuated by Lake Formation when we share a table with an external account, so it looked more like a configuration issue than a code issue. 

Found resource_share_associations : 
[{'resourceShareArn': ......, **'status': 'FAILED',** 
[...]
Found 0 RAM invitations for resourceShareArn: ['arn:aws:ram:eu-west-1:xxxxxxxxxx:resource-share/780c5fc2-50a0-4207-9ff9-e0304cc64093']

Typical scenarios when this happens:

In this case the issue was on the Glue Catalog settings. A new policy was added to the Glue Catalog to allow some Quicksight actions outside of data.all. It resulted in blocking data catalog asset sharing to other accounts. We just needed to add some statements to the Glue Catalog Settings resource policy. For this I recommend to use the AWS CLI to effectuate the changes, as there are some options that are not available in the AWS Console.

@mvidhu let me know if you agree with this summary of the issue and let's close it :)