Lake Formation: AccessDeniedException on GetTemporaryGlueTableCredentials

[ghosalya]

Describe the bug

I'm trying to get a temporary credential that have access to a table's underlying S3 location. The endpoint get_temporary_glue_table_credentials looks like this will do just that, but when I try to use boto3 it raises AccessDeniedException without further explanation.

Steps to reproduce

The following is a simple call to get_temporary_glue_table_credentials, where the TableArn is pointing towards a valid Glue table. I've tried this with both Lake Formation at us-east-1 and ap-northeast-1.

import boto3

lakeformation = boto3.client('lakeformation')
response = lakeformation.get_temporary_glue_table_credentials(
    TableArn="arn:aws:glue:<region>:<aws_account_id>:table/<database>/<table>",
    SupportedPermissionTypes=["COLUMN_PERMISSION"],
)

Expected behavior An AWS temporary credential is returned as part of response (e.g. Access Key, Secret Key)

Debug logs

The method returns AccessDeniedException without further explanation.

Traceback (most recent call last):
  File "create_nr_table.py", line 40, in <module>
    "COLUMN_PERMISSION",
  File "/home/gede/work/tra/acme-c360-api/.direnv/python-3.7.3/lib/python3.7/site-packages/botocore/client.py", line 395, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/gede/work/tra/acme-c360-api/.direnv/python-3.7.3/lib/python3.7/site-packages/botocore/client.py", line 728, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the GetTemporaryGlueTableCredentials operation: None

It seems like the endpoint is returning http error 400 (see Full Stack Trace).

---

Full Stack Trace

``` 2022-02-28 12:46:45,383 - botocore.hooks - DEBUG - Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2022-02-28 12:46:45,385 - botocore.hooks - DEBUG - Changing event name from before-call.apigateway to before-call.api-gateway 2022-02-28 12:46:45,386 - botocore.hooks - DEBUG - Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2022-02-28 12:46:45,387 - botocore.hooks - DEBUG - Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2022-02-28 12:46:45,388 - botocore.hooks - DEBUG - Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2022-02-28 12:46:45,388 - botocore.hooks - DEBUG - Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2022-02-28 12:46:45,389 - botocore.hooks - DEBUG - Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section 2022-02-28 12:46:45,391 - botocore.hooks - DEBUG - Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2022-02-28 12:46:45,391 - botocore.hooks - DEBUG - Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section 2022-02-28 12:46:45,391 - botocore.hooks - DEBUG - Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2022-02-28 12:46:45,391 - botocore.hooks - DEBUG - Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section 2022-02-28 12:46:45,399 - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/ 2022-02-28 12:46:45,400 - botocore.credentials - DEBUG - Looking for credentials via: env 2022-02-28 12:46:45,401 - botocore.credentials - DEBUG - Looking for credentials via: assume-role 2022-02-28 12:46:45,401 - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity 2022-02-28 12:46:45,401 - botocore.credentials - DEBUG - Looking for credentials via: sso 2022-02-28 12:46:45,401 - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file 2022-02-28 12:46:45,401 - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials 2022-02-28 12:46:45,402 - botocore.loaders - DEBUG - Loading JSON file: /home/gede/work/tra/acme-c360-api/.direnv/python-3.7.3/lib/python3.7/site-packages/botocore/data/endpoints.json 2022-02-28 12:46:45,411 - botocore.loaders - DEBUG - Loading JSON file: /home/gede/work/tra/acme-c360-api/.direnv/python-3.7.3/lib/python3.7/site-packages/botocore/data/sdk-default-configuration.json 2022-02-28 12:46:45,411 - botocore.hooks - DEBUG - Event choose-service-name: calling handler 2022-02-28 12:46:45,417 - botocore.loaders - DEBUG - Loading JSON file: /home/gede/work/tra/acme-c360-api/.direnv/python-3.7.3/lib/python3.7/site-packages/botocore/data/lakeformation/2017-03-31/service-2.json 2022-02-28 12:46:45,421 - botocore.hooks - DEBUG - Event creating-client-class.lakeformation: calling handler 2022-02-28 12:46:45,422 - botocore.endpoint - DEBUG - Setting lakeformation timeout as (60, 60) 2022-02-28 12:46:45,423 - botocore.loaders - DEBUG - Loading JSON file: /home/gede/work/tra/acme-c360-api/.direnv/python-3.7.3/lib/python3.7/site-packages/botocore/data/_retry.json 2022-02-28 12:46:45,423 - botocore.client - DEBUG - Registering retry handlers for service: lakeformation 2022-02-28 12:46:45,424 - botocore.hooks - DEBUG - Event before-parameter-build.lakeformation.GetTemporaryGlueTableCredentials: calling handler 2022-02-28 12:46:45,424 - botocore.hooks - DEBUG - Event before-call.lakeformation.GetTemporaryGlueTableCredentials: calling handler 2022-02-28 12:46:45,424 - botocore.hooks - DEBUG - Event before-call.lakeformation.GetTemporaryGlueTableCredentials: calling handler 2022-02-28 12:46:45,424 - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetTemporaryGlueTableCredentials) with params: {'url_path': '/GetTemporaryGlueTableCredentials', 'query_string': {}, 'method': 'POST', 'headers': {'Content-Type': 'application/json', 'User-Agent': 'Boto3/1.21.8 Python/3.7.3 Linux/5.13.0-30-generic Botocore/1.24.8'}, 'body': b'{"TableArn": "arn:aws:glue:ap-northeast-1:194389615025:table/nresource_test/experiment_nontabular_png", "SupportedPermissionTypes": ["COLUMN_PERMISSION"]}', 'url': 'https://lakeformation.ap-northeast-1.amazonaws.com/GetTemporaryGlueTableCredentials', 'context': {'client_region': 'ap-northeast-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2022-02-28 12:46:45,425 - botocore.hooks - DEBUG - Event request-created.lakeformation.GetTemporaryGlueTableCredentials: calling handler > 2022-02-28 12:46:45,425 - botocore.hooks - DEBUG - Event choose-signer.lakeformation.GetTemporaryGlueTableCredentials: calling handler 2022-02-28 12:46:45,425 - botocore.auth - DEBUG - Calculating signature using v4 auth. 2022-02-28 12:46:45,425 - botocore.auth - DEBUG - CanonicalRequest: POST /GetTemporaryGlueTableCredentials content-type:application/json host:lakeformation.ap-northeast-1.amazonaws.com x-amz-date:20220228T044645Z content-type;host;x-amz-date e02e02596409638bdaa3f98038585833ffb015d51a93509dc6fe4568afdeed3d 2022-02-28 12:46:45,425 - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20220228T044645Z 20220228/ap-northeast-1/lakeformation/aws4_request 490ba1019b1122168fedc3019386fd9387ff700776a83532ed60013a584567c6 2022-02-28 12:46:45,425 - botocore.auth - DEBUG - Signature: c9e38bff9841baba62e40e089c1974f10bd727ec5bdd04f1116e0c8a70361904 2022-02-28 12:46:45,425 - botocore.hooks - DEBUG - Event request-created.lakeformation.GetTemporaryGlueTableCredentials: calling handler 2022-02-28 12:46:45,426 - botocore.endpoint - DEBUG - Sending http request: 2022-02-28 12:46:45,426 - botocore.httpsession - DEBUG - Certificate path: /home/gede/work/tra/acme-c360-api/.direnv/python-3.7.3/lib/python3.7/site-packages/certifi/cacert.pem 2022-02-28 12:46:45,426 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): lakeformation.ap-northeast-1.amazonaws.com:443 2022-02-28 12:46:46,245 - urllib3.connectionpool - DEBUG - https://lakeformation.ap-northeast-1.amazonaws.com:443 "POST /GetTemporaryGlueTableCredentials HTTP/1.1" 400 16 2022-02-28 12:46:46,246 - botocore.parsers - DEBUG - Response headers: {'Date': 'Mon, 28 Feb 2022 04:46:46 GMT', 'Content-Type': 'application/json', 'Content-Length': '16', 'Connection': 'keep-alive', 'x-amzn-RequestId': '78abe3a2-3f9d-43fc-a2fd-2fc31d2032b9', 'x-amzn-ErrorType': 'AccessDeniedException:http://internal.amazon.com/coral/com.amazonaws.michigan.common.exceptions/', 'Cache-Control': 'no-cache'} 2022-02-28 12:46:46,246 - botocore.parsers - DEBUG - Response body: b'{"Message":null}' 2022-02-28 12:46:46,249 - botocore.parsers - DEBUG - Response headers: {'Date': 'Mon, 28 Feb 2022 04:46:46 GMT', 'Content-Type': 'application/json', 'Content-Length': '16', 'Connection': 'keep-alive', 'x-amzn-RequestId': '78abe3a2-3f9d-43fc-a2fd-2fc31d2032b9', 'x-amzn-ErrorType': 'AccessDeniedException:http://internal.amazon.com/coral/com.amazonaws.michigan.common.exceptions/', 'Cache-Control': 'no-cache'} 2022-02-28 12:46:46,249 - botocore.parsers - DEBUG - Response body: b'{"Message":null}' 2022-02-28 12:46:46,249 - botocore.hooks - DEBUG - Event needs-retry.lakeformation.GetTemporaryGlueTableCredentials: calling handler 2022-02-28 12:46:46,249 - botocore.retryhandler - DEBUG - No retry needed. Traceback (most recent call last): File "create_nr_table.py", line 40, in "COLUMN_PERMISSION", File "/home/gede/work/tra/acme-c360-api/.direnv/python-3.7.3/lib/python3.7/site-packages/botocore/client.py", line 395, in _api_call return self._make_api_call(operation_name, kwargs) File "/home/gede/work/tra/acme-c360-api/.direnv/python-3.7.3/lib/python3.7/site-packages/botocore/client.py", line 728, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the GetTemporaryGlueTableCredentials operation: None ```

[tim-finnigan]

Hi @ghosalya thanks for reaching out. Can you confirm that your credentials are set correctly in ~/.aws/credentials and that you have the necessary IAM permissions assigned?

[ghosalya]

@tim-finnigan Hello, thank you for the quick response!

Hi @ghosalya thanks for reaching out. Can you confirm that your credentials are set correctly in ~/.aws/credentials

Yes, confirmed by running sts.get_caller_identity()

and that you have the necessary IAM permissions assigned?

Ah, I should've mentioned this earlier. I'm currently using lakeformation:* in my policy because from the IAM UI, I wasn't able to find GetTemporaryGlueTableCredentials under Lake Formation or Glue.

Screenshot of IAM UI


[tim-finnigan]

Hi @ghosalya thanks for following up. I was trying to find the API documentation for GetTemporaryGlueTableCredentials but was directed to a broken link. I created a ticket here in our shared SDK repository for that: https://github.com/aws/aws-sdk/issues/196

I’m not sure which IAM permissions you need but the examples in this documentation might help: https://docs.aws.amazon.com/glue/latest/dg/using-identity-based-policies.html. Have you tried adding "glue:GetTable" and "glue:GetTables" permissions?

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[nickdelnano]

@ghosalya this API is reserved for registered third party query engines only. It must be called by an IAM role with a session tag registered to Lake Formation. See the following documentation pages:

• https://docs.aws.amazon.com/lake-formation/latest/dg/api-overview.html
• https://docs.aws.amazon.com/lake-formation/latest/dg/register-query-engine.html
• https://docs.aws.amazon.com/lake-formation/latest/dg/permitting-third-party-call.html

One piece of information I find missing from the documentation, is how Lake Formation authorizes the IAM role against the TableArn parameter. LF table permissions are used.