search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.92k stars 1.86k forks source link

CloudFront KeyValueStore client API fails with assumed role/STS credentials #3980

Closed pitkley closed 4 days ago

pitkley commented 8 months ago

Describe the bug

Using the cloudfront-keyvaluestore client, trying to invoke the describe_key_value_store function when using an assumed role fails with an Authentication failed error. Using an access-key-ID and secret-access-key pair of the target-account works without issues. Both the role tested and the direct access key have the same full AdministratorAccess permissions.

Expected Behavior

Just like for all other AWS services used with boto3 (at least the ones I have ever used), assuming a role should work with the cloudfront-keyvaluestore client and operations like DescribeKeyValueStore should not fail.

Current Behavior

The describe_key_value_store operation fails with this error when using an assumed role:

Traceback (most recent call last):
  File "...", line 34, in <module>
    boto3.client("cloudfront-keyvaluestore").describe_key_value_store(KvsARN=kvs_arn)
  File ".../python3.12/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the DescribeKeyValueStore operation: Authentication failed

Reproduction Steps

The following is a minimal script that reproduces the error. It expects to be invoked with an environment variable ASSUMED_IAM_ROLE set to the ARN of the IAM role that should be assumed.

import os
import boto3

ASSUMED_IAM_ROLE = os.environ["ASSUMED_IAM_ROLE"]

session_base = boto3.Session()

assumed_role = session_base.client("sts").assume_role(
    RoleArn=ASSUMED_IAM_ROLE,
    RoleSessionName="cfkvs_403_reproducer",
)
print(f"++ Assumed role: {assumed_role['AssumedRoleUser']['Arn']}")
assumed_credentials = assumed_role["Credentials"]
session_assumed = boto3.Session(
    aws_access_key_id=assumed_credentials["AccessKeyId"],
    aws_secret_access_key=assumed_credentials["SecretAccessKey"],
    aws_session_token=assumed_credentials["SessionToken"],
)

# Get a KVS ARN via the automatically assumed client, showing that role-assumption works
kvs_arn = session_assumed.client("cloudfront").list_key_value_stores()["KeyValueStoreList"]["Items"][0]["ARN"]
print(f"++ Retrieved KVS ARN with assumed session: {kvs_arn}")

# Try to describe the KVS using the automatically assumed client, which fails
try:
    response = session_assumed.client("cloudfront-keyvaluestore").describe_key_value_store(KvsARN=kvs_arn)
    print(f"++ Describe KVS with assumed session: KvsARN={response['KvsARN']}, ETag={response['ETag']}")
except Exception as e:
    print("!! Assumed session: CloudFrontKeyValueStore.describe_key_value_store failed")
    raise e

The script will print output like this:

++ Assumed role: arn:aws:sts::123412341234:assumed-role/RoleName/cfkvs_403_reproducer
++ Retrieved KVS ARN with assumed session: arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234
!! Assumed session: CloudFrontKeyValueStore.describe_key_value_store failed
Traceback (most recent call last):
  File ".../reproducer.py", line 30, in <module>
    raise e
  File ".../reproducer.py", line 26, in <module>
    response = session_assumed.client("cloudfront-keyvaluestore").describe_key_value_store(KvsARN=kvs_arn)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the DescribeKeyValueStore operation: Authentication failed

If it were to succeed, the output should look like the following. It has never succeeded in any of my tests.

++ Assumed role: arn:aws:sts::123412341234:assumed-role/RoleName/cfkvs_403_reproducer
++ Retrieved KVS ARN with assumed session: arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234
++ Describe KVS with assumed session: KvsARN=arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234, ETag=KV1A23B4C5DEF6G

Possible Solution

No response

Additional Information/Context

Interestingly enough, using the aws-cli (aws-cli/2.15.2 Python/3.11.6 Darwin/22.6.0 exe/x86_64 prompt/off) works without issues:

$ aws --profile <temporary-profile> cloudfront-keyvaluestore describe-key-value-store --kvs-arn arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234
{
    "ETag": "KV1A23B4C5DEF6G",
    "ItemCount": 30,
    "TotalSizeInBytes": 792,
    "KvsARN": "arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234",
    "Created": "...",
    "LastModified": "..."
}

Here, the temporary-profile profile is configured through the ~/.aws/credentials file after invoking aws sts assume-role --role-arn "$ASSUMED_IAM_ROLE" --role-session-name aws-cli, looking roughly like this:

[temporary-profile]
aws_access_key_id = AKIA...
aws_secret_access_key = ...
aws_session_token = ...

I have looked at debug-output of both boto3 (through boto3.set_stream_logger("botocore")) and the aws-cli (using the --debug CLI flag). One difference I did notice in the output (which might not be the only one, just one I did notice), is that the aws-cli will have output like this:

...
2023-12-21 17:19:09,635 - MainThread - botocore.hooks - DEBUG - Event request-created.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1104eb4d0>>
2023-12-21 17:19:09,635 - MainThread - botocore.hooks - DEBUG - Event choose-signer.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <function set_operation_specific_signer at 0x10e6087c0>
[INFO] [2023-12-21T16:19:09Z] [0000000202e7e280] [AuthCredentialsProvider] - (id=0x600000423b70) Static credentials provider successfully sourced credentials
[INFO] [2023-12-21T16:19:09Z] [0000000202e7e280] [AuthSigning] - (id=0x600002930230) Signing successfully built canonical request for algorithm SigV4Asymmetric, with contents
...
[INFO] [2023-12-21T16:19:09Z] [0000000202e7e280] [AuthSigning] - (id=0x600002930230) Signing successfully built string-to-sign via algorithm SigV4Asymmetric, with contents
...
[INFO] [2023-12-21T16:19:09Z] [0000000202e7e280] [AuthSigning] - (id=0x600002930230) Http request successfully built final authorization value via algorithm SigV4Asymmetric, with contents
...
2023-12-21 17:19:09,638 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest ...>

When running the reproducer-script above with debug output enabled, will have similar botocore.hooks log-statements, but showing none of the AuthCredentialsProvider or AuthSigning output. (Maybe this is expected, since it does not show for the CloudFront ListKeyValueStores API-call either.)

2023-12-21 17:21:41,894 botocore.hooks [DEBUG] Event request-created.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x107c935c0>>
2023-12-21 17:21:41,894 botocore.hooks [DEBUG] Event choose-signer.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <function set_operation_specific_signer at 0x105c0e7a0>
2023-12-21 17:21:41,895 botocore.hooks [DEBUG] Event request-created.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <function add_retry_headers at 0x105c30b80>
2023-12-21 17:21:41,895 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest ...>

SDK version used

boto3 1.34.4 (botocore 1.34.4)

Environment details (OS name and version, etc.)

macOS 13.6.3, Python 3.12.1

nzspambot commented 1 month ago

@pitkley since there is no update I will give you the fix from AWS Enterprise support:

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-function-restrictions.html#regional-endpoint-for-key-value-store

You must use a regional STS endpoint when calling the keyvalue api endpoint

Enjoy :)

RyanFitzSimmonsAK commented 2 weeks ago

Hi @pitkley, thanks for reaching out and for your patience. Are you still having trouble with this? If so, could you talk about how you're figuring your region in both Boto3 and AWS CLI? It might also be worthwhile looking at the sts_regional_endpoints configuration option, which will let you hit a regional endpoint as mentioned above. Thanks!

github-actions[bot] commented 1 week ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.