search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.92k stars 1.86k forks source link

Generate V2Token in STS #4193

Open AvanishCodes opened 1 month ago

AvanishCodes commented 1 month ago

Describe the bug

I want to generate the presigned URL for a multi-region access point using the SDK. I believe I can use CrtS3SigV4AsymQueryAuth for this, popularly known as SigV4a-QueryParameter.

While running the code in AWS Lambda, I want to assume the role using STS, and execute the code. I shall be using the credentials provided by the below code (or simliar):

def assume_role(role_arn, session_name):
    sts_client = boto3.client('sts', region_name='us-east-1')
    try:
        response = sts_client.assume_role(
            RoleArn=role_arn,
            RoleSessionName=session_name
        )
        return response['Credentials']
    except (NoCredentialsError, PartialCredentialsError) as e:
        print(f"Error in assuming role: {e}")
        return None

The token that this piece of code is returning, is not helping me generate a presigned URL that works. I am getting the below error:

status_code: 400 
object text: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>UnsupportedSignature</Code><Message>The provided request is signed with an unsupported STS Token version or the signature version is not supported.</Message><RequestId>SOMETHING</RequestId><HostId>SomeCode</HostId></Error>

Expected Behavior

If I run the code to generate the presigned URL by providing the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN generated using AWS CLI (v2.15.58) command:

aws sts assume-role --role-arn arn:aws:iam::AWS_ACCOUNT_ID:role/admin --role-session-name SessionName

It works completely fine

Current Behavior

In the current setup, the SDK returns token which can't be used to get the

Reproduction Steps

As described above:

Fails when:

Succeeds when:

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.34.141

Environment details (OS name and version, etc.)

macOS 14

tim-finnigan commented 1 month ago

Thanks for reaching out. Both the Boto3 assume_role and CLI assume-role commands make calls to the STS AssumeRole API. So the responses should be consistent here. Can you confirm that you're using the correct profile? Also for reference here is the documentation for generating presigned URLs: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html.

I haven't seen an UnsupportedSignature error before, usually if there are signature issues then a SignatureDoesNotMatch error is returned. (See documentation here on troubleshooting signature issues: https://docs.aws.amazon.com/IAM/latest/UserGuide/signature-v4-troubleshooting.html). If you could share your full debug logs (with sensitive info redacted) by adding boto3.set_stream_logger('') to your script, then that may give us more insight into the underlying issue.

AvanishCodes commented 1 month ago

While setting the AWS CLI, if I don't set the region, the responses are same. But for boto3, if I provide the region while creating the STS client, or use AWS_REGION environment variable, the token remains the same.

However, boto3 and AWS CLI give the same kind of token when using AWS_STS_REGIONAL_ENDPOINTS=regional.

tim-finnigan commented 1 month ago

Thanks for following up, can you please share the information requested in my earlier comment?

Can you confirm that you're using the correct profile? Also for reference here is the documentation for generating presigned URLs: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html.

I haven't seen an UnsupportedSignature error before, usually if there are signature issues then a SignatureDoesNotMatch error is returned. (See documentation here on troubleshooting signature issues: https://docs.aws.amazon.com/IAM/latest/UserGuide/signature-v4-troubleshooting.html). If you could share your full debug logs (with sensitive info redacted) by adding boto3.set_stream_logger('') to your script, then that may give us more insight into the underlying issue.

github-actions[bot] commented 3 weeks ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

AvanishCodes commented 3 weeks ago

Hi @tim-finnigan I am sure that I am using the correct profile. It also has admin access on the AWS account.

tim-finnigan commented 2 weeks ago

If you could share your full debug logs (with sensitive info redacted) by adding boto3.set_stream_logger('') to your script, then that may give us more insight into the underlying issue.

AvanishCodes commented 2 weeks ago

Debug Logs for CLI

2024-08-08 05:29:08,533 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.15.58 Python/3.11.9 Darwin/23.5.0 source/arm64
2024-08-08 05:29:08,533 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sts', 'assume-role', '--role-arn', 'arn:aws:iam::999999999999:role/RootRoleName', '--role-session-name', 'Session1', '--debug']
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x105280540>
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x10508f9c0>
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x104ccc2c0>
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x104ccd940>
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x1052985e0>
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x1050f2b60>
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x105283560>
2024-08-08 05:29:08,539 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x105341650>>
2024-08-08 05:29:08,539 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.15.58/libexec/lib/python3.11/site-packages/awscli/data/cli.json
2024-08-08 05:29:08,540 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x1051b2480>
2024-08-08 05:29:08,540 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x1051b27a0>
2024-08-08 05:29:08,540 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x1051b2700>
2024-08-08 05:29:08,540 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x1051b28e0>
2024-08-08 05:29:08,540 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x1051b2840>
2024-08-08 05:29:08,540 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x10533a9c0>
2024-08-08 05:29:08,540 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.15.58 Python/3.11.9 Darwin/23.5.0 source/arm64
2024-08-08 05:29:08,540 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sts', 'assume-role', '--role-arn', 'arn:aws:iam::999999999999:role/RootRoleName', '--role-session-name', 'Session1', '--debug']
2024-08-08 05:29:08,540 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x105280f40>
2024-08-08 05:29:08,541 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x104ad0b80>
2024-08-08 05:29:08,541 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x105305a80>
2024-08-08 05:29:08,541 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x104998360>
2024-08-08 05:29:08,541 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x104aeca40>
2024-08-08 05:29:08,542 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-08-08 05:29:08,545 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x1050f2340>
2024-08-08 05:29:08,545 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x104d2ba60>
2024-08-08 05:29:08,550 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.15.58/libexec/lib/python3.11/site-packages/awscli/botocore/data/sts/2011-06-15/service-2.json
2024-08-08 05:29:08,551 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts: calling handler <function add_waiters at 0x105283560>
2024-08-08 05:29:08,556 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x105341650>>
2024-08-08 05:29:08,556 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('role-arn', <awscli.arguments.CLIArgument object at 0x1053bf0d0>), ('role-session-name', <awscli.arguments.CLIArgument object at 0x1053bf2d0>), ('policy-arns', <awscli.arguments.ListArgument object at 0x1053bf550>), ('policy', <awscli.arguments.CLIArgument object at 0x1053bf790>), ('duration-seconds', <awscli.arguments.CLIArgument object at 0x1053bf8d0>), ('tags', <awscli.arguments.ListArgument object at 0x1053bfb10>), ('transitive-tag-keys', <awscli.arguments.ListArgument object at 0x1053bfcd0>), ('external-id', <awscli.arguments.CLIArgument object at 0x1053bfe90>), ('serial-number', <awscli.arguments.CLIArgument object at 0x1053c0e10>), ('token-code', <awscli.arguments.CLIArgument object at 0x1053c0290>), ('source-identity', <awscli.arguments.CLIArgument object at 0x1053c0410>), ('provided-contexts', <awscli.arguments.ListArgument object at 0x1053c0590>)])
2024-08-08 05:29:08,557 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role: calling handler <function add_streaming_output_arg at 0x105281580>
2024-08-08 05:29:08,557 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role: calling handler <function add_cli_input_json at 0x104aed760>
2024-08-08 05:29:08,557 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role: calling handler <function add_cli_input_yaml at 0x104aed800>
2024-08-08 05:29:08,557 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role: calling handler <function unify_paging_params at 0x1050bc180>
2024-08-08 05:29:08,562 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.15.58/libexec/lib/python3.11/site-packages/awscli/botocore/data/sts/2011-06-15/paginators-1.json
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role: calling handler <function add_generate_skeleton at 0x1051b0b80>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.assume-role: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x1053c09d0>>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.assume-role: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x1053bdcd0>>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.assume-role: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x1053c9fd0>>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts_assume-role: calling handler <function add_waiters at 0x105283560>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts_assume-role: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x105341650>>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.role-arn: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.sts.assume-role: calling handler <awscli.argprocess.ParamShorthandParser object at 0x1049c4090>
2024-08-08 05:29:08,562 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'arn:aws:iam::999999999999:role/RootRoleName' for parameter "role_arn": 'arn:aws:iam::999999999999:role/RootRoleName'
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.role-session-name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.sts.assume-role: calling handler <awscli.argprocess.ParamShorthandParser object at 0x1049c4090>
2024-08-08 05:29:08,562 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'Session1' for parameter "role_session_name": 'Session1'
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.policy-arns: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.policy: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.duration-seconds: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.tags: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.transitive-tag-keys: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.external-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.serial-number: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.token-code: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.source-identity: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.provided-contexts: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x104a44050>
2024-08-08 05:29:08,562 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.assume-role: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x1053c09d0>>
2024-08-08 05:29:08,563 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.assume-role: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x1053bdcd0>>
2024-08-08 05:29:08,563 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.assume-role: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x1053c9fd0>>
2024-08-08 05:29:08,563 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-08-08 05:29:08,563 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2024-08-08 05:29:08,563 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2024-08-08 05:29:08,563 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2024-08-08 05:29:08,563 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2024-08-08 05:29:08,563 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2024-08-08 05:29:08,563 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.15.58/libexec/lib/python3.11/site-packages/awscli/botocore/data/endpoints.json
2024-08-08 05:29:08,568 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x103b68c20>
2024-08-08 05:29:08,574 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.15.58/libexec/lib/python3.11/site-packages/awscli/botocore/data/sts/2011-06-15/endpoint-rule-set-1.json
2024-08-08 05:29:08,574 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.15.58/libexec/lib/python3.11/site-packages/awscli/botocore/data/partitions.json
2024-08-08 05:29:08,574 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.sts: calling handler <function add_generate_presigned_url at 0x103aa7f60>
2024-08-08 05:29:08,574 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for sts via: environment_service
2024-08-08 05:29:08,574 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for sts via: environment_global
2024-08-08 05:29:08,574 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for sts via: config_service
2024-08-08 05:29:08,575 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for sts via: config_global
2024-08-08 05:29:08,575 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-08-08 05:29:08,575 - MainThread - botocore.endpoint - DEBUG - Setting sts timeout as (60, 60)
2024-08-08 05:29:08,576 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.sts.AssumeRole: calling handler <function base64_decode_input_blobs at 0x105305b20>
2024-08-08 05:29:08,576 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.sts.AssumeRole: calling handler <function generate_idempotent_uuid at 0x103b6b060>
2024-08-08 05:29:08,576 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False, 'UseGlobalEndpoint': False}
2024-08-08 05:29:08,576 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://sts.us-east-1.amazonaws.com
2024-08-08 05:29:08,576 - MainThread - botocore.hooks - DEBUG - Event before-call.sts.AssumeRole: calling handler <function inject_api_version_header_if_needed at 0x103b9cb80>
2024-08-08 05:29:08,576 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=AssumeRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.15.58 md/awscrt#0.19.19 ua/2.0 os/macos#23.5.0 md/arch#arm64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#sts.assume-role'}, 'body': {'Action': 'AssumeRole', 'Version': '2011-06-15', 'RoleArn': 'arn:aws:iam::999999999999:role/RootRoleName', 'RoleSessionName': 'Session1'}, 'url': 'https://sts.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x109962350>, 'has_streaming_input': False, 'auth_type': None}}
2024-08-08 05:29:08,576 - MainThread - botocore.hooks - DEBUG - Event request-created.sts.AssumeRole: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1053c0050>>
2024-08-08 05:29:08,576 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sts.AssumeRole: calling handler <function set_operation_specific_signer at 0x103b6af20>
2024-08-08 05:29:08,577 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-08-08 05:29:08,577 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.us-east-1.amazonaws.com
x-amz-date:20240807T235908Z

content-type;host;x-amz-date
50aa65c4569e8c263612228ddd6c09828ca70c470154ae62c491a74ef7725d04
2024-08-08 05:29:08,577 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240807T235908Z
20240807/us-east-1/sts/aws4_request
e6eec51dddf586048528a6336c6024487bfcb69f45e36359c386957080d82a0d
2024-08-08 05:29:08,577 - MainThread - botocore.auth - DEBUG - Signature:
5130f3e985f80c1d71a9249e09eebb4f3baba1a6970153205b4748358a83dea8
2024-08-08 05:29:08,577 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.us-east-1.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.15.58 md/awscrt#0.19.19 ua/2.0 os/macos#23.5.0 md/arch#arm64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#sts.assume-role', 'X-Amz-Date': b'20240807T235908Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA5VAA23GQIPQLFGM3/20240807/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=5130f3e985f80c1d71a9249e09eebb4f3baba1a6970153205b4748358a83dea8', 'Content-Length': '118'}>
2024-08-08 05:29:08,577 - MainThread - botocore.httpsession - DEBUG - Certificate path: /opt/homebrew/Cellar/awscli/2.15.58/libexec/lib/python3.11/site-packages/awscli/botocore/cacert.pem
2024-08-08 05:29:08,577 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): sts.us-east-1.amazonaws.com:443
2024-08-08 05:29:09,625 - MainThread - urllib3.connectionpool - DEBUG - https://sts.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 200 1416
2024-08-08 05:29:09,626 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'UUID', 'Content-Type': 'text/xml', 'Content-Length': '1416', 'Date': 'Wed, 07 Aug 2024 23:59:09 GMT'}
2024-08-08 05:29:09,626 - MainThread - botocore.parsers - DEBUG - Response body:
b'<AssumeRoleResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">\n  <AssumeRoleResult>\n    <AssumedRoleUser>\n      <AssumedRoleId>ASSUMEDAWSROLEID:Session1</AssumedRoleId>\n      <Arn>arn:aws:sts::999999999999:assumed-role/RootRoleName/Session1</Arn>\n    </AssumedRoleUser>\n    <Credentials>\n      <AccessKeyId>ASIAASIAASIAASIAASIA</AccessKeyId>\n      <SecretAccessKey>SomeSECRETAccessKey</SecretAccessKey>\n      <SessionToken>SomeTokenValue</SessionToken>\n      <Expiration>2024-08-08T00:59:09Z</Expiration>\n    </Credentials>\n  </AssumeRoleResult>\n  <ResponseMetadata>\n    <RequestId>UUID</RequestId>\n  </ResponseMetadata>\n</AssumeRoleResponse>\n'
2024-08-08 05:29:09,628 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sts.AssumeRole: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x109968690>>
2024-08-08 05:29:09,628 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-08-08 05:29:09,628 - MainThread - botocore.hooks - DEBUG - Event after-call.sts.AssumeRole: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x109963f90>>
2024-08-08 05:29:09,629 - MainThread - awscli.formatter - DEBUG - RequestId: UUID
{
    "Credentials": {
        "AccessKeyId": "ASIAASIAASIAASIAASIA",
        "SecretAccessKey": "SomeSECRETAccessKey",
        "SessionToken": "SomeTokenValue",
        "Expiration": "2024-08-08T00:59:09+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "ASSUMEDAWSROLEID:Session1",
        "Arn": "arn:aws:sts::999999999999:assumed-role/RootRoleName/Session1"
    }
}

Debug Logs for SDK

(.venv) avanish@Avanish aws-sts-v2-signing % /Users/avanish/temp/aws-sts-v2-signing/.venv/bin/python /Users/avanish/temp/aws-sts-v2-signing/x.py
2024-08-08 05:29:22,237 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2024-08-08 05:29:22,238 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2024-08-08 05:29:22,238 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2024-08-08 05:29:22,239 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2024-08-08 05:29:22,239 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2024-08-08 05:29:22,239 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2024-08-08 05:29:22,239 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2024-08-08 05:29:22,240 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2024-08-08 05:29:22,240 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2024-08-08 05:29:22,240 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2024-08-08 05:29:22,240 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2024-08-08 05:29:22,242 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2024-08-08 05:29:22,245 botocore.credentials [DEBUG] Looking for credentials via: env
2024-08-08 05:29:22,245 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2024-08-08 05:29:22,245 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2024-08-08 05:29:22,245 botocore.credentials [DEBUG] Looking for credentials via: sso
2024-08-08 05:29:22,245 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2024-08-08 05:29:22,245 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2024-08-08 05:29:22,246 botocore.loaders [DEBUG] Loading JSON file: /Users/avanish/temp/aws-sts-v2-signing/.venv/lib/python3.12/site-packages/botocore/data/endpoints.json
2024-08-08 05:29:22,253 botocore.loaders [DEBUG] Loading JSON file: /Users/avanish/temp/aws-sts-v2-signing/.venv/lib/python3.12/site-packages/botocore/data/sdk-default-configuration.json
2024-08-08 05:29:22,254 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x106194220>
2024-08-08 05:29:22,293 botocore.loaders [DEBUG] Loading JSON file: /Users/avanish/temp/aws-sts-v2-signing/.venv/lib/python3.12/site-packages/botocore/data/sts/2011-06-15/service-2.json.gz
2024-08-08 05:29:22,301 botocore.loaders [DEBUG] Loading JSON file: /Users/avanish/temp/aws-sts-v2-signing/.venv/lib/python3.12/site-packages/botocore/data/sts/2011-06-15/endpoint-rule-set-1.json.gz
2024-08-08 05:29:22,301 botocore.loaders [DEBUG] Loading JSON file: /Users/avanish/temp/aws-sts-v2-signing/.venv/lib/python3.12/site-packages/botocore/data/partitions.json
2024-08-08 05:29:22,302 botocore.hooks [DEBUG] Event creating-client-class.sts: calling handler <function add_generate_presigned_url at 0x1060b62a0>
2024-08-08 05:29:22,302 botocore.configprovider [DEBUG] Looking for endpoint for sts via: environment_service
2024-08-08 05:29:22,302 botocore.configprovider [DEBUG] Looking for endpoint for sts via: environment_global
2024-08-08 05:29:22,302 botocore.configprovider [DEBUG] Looking for endpoint for sts via: config_service
2024-08-08 05:29:22,302 botocore.configprovider [DEBUG] Looking for endpoint for sts via: config_global
2024-08-08 05:29:22,302 botocore.configprovider [DEBUG] No configured endpoint found.
2024-08-08 05:29:22,302 botocore.endpoint [DEBUG] Setting sts timeout as (60, 60)
2024-08-08 05:29:22,303 botocore.loaders [DEBUG] Loading JSON file: /Users/avanish/temp/aws-sts-v2-signing/.venv/lib/python3.12/site-packages/botocore/data/_retry.json
2024-08-08 05:29:22,303 botocore.client [DEBUG] Registering retry handlers for service: sts
2024-08-08 05:29:22,304 botocore.hooks [DEBUG] Event before-parameter-build.sts.AssumeRole: calling handler <function generate_idempotent_uuid at 0x106195f80>
2024-08-08 05:29:22,304 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False, 'UseGlobalEndpoint': True}
2024-08-08 05:29:22,304 botocore.regions [DEBUG] Endpoint provider result: https://sts.amazonaws.com
2024-08-08 05:29:22,304 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-08-08 05:29:22,304 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 'sts'}
2024-08-08 05:29:22,304 botocore.hooks [DEBUG] Event before-call.sts.AssumeRole: calling handler <function add_recursion_detection_header at 0x106195440>
2024-08-08 05:29:22,304 botocore.hooks [DEBUG] Event before-call.sts.AssumeRole: calling handler <function inject_api_version_header_if_needed at 0x106197a60>
2024-08-08 05:29:22,304 botocore.endpoint [DEBUG] Making request for OperationModel(name=AssumeRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.141 md/Botocore#1.34.141 md/awscrt#0.21.0 ua/2.0 os/macos#23.5.0 md/arch#arm64 lang/python#3.12.3 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.141'}, 'body': {'Action': 'AssumeRole', 'Version': '2011-06-15', 'RoleArn': 'arn:aws:iam::999999999999:role/RootRoleName', 'RoleSessionName': 'Session1', 'ExternalId': 'test'}, 'url': 'https://sts.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x10622faa0>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'us-east-1', 'signing_name': 'sts'}, 'endpoint_properties': {'authSchemes': [{'name': 'sigv4', 'signingName': 'sts', 'signingRegion': 'us-east-1'}]}}}
2024-08-08 05:29:22,304 botocore.hooks [DEBUG] Event request-created.sts.AssumeRole: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x106756930>>
2024-08-08 05:29:22,304 botocore.hooks [DEBUG] Event choose-signer.sts.AssumeRole: calling handler <function set_operation_specific_signer at 0x106195e40>
2024-08-08 05:29:22,305 botocore.hooks [DEBUG] Event request-created.sts.AssumeRole: calling handler <function add_retry_headers at 0x1061b02c0>
2024-08-08 05:29:22,305 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'Boto3/1.34.141 md/Botocore#1.34.141 md/awscrt#0.21.0 ua/2.0 os/macos#23.5.0 md/arch#arm64 lang/python#3.12.3 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.141', 'host': b'sts.amazonaws.com', 'X-Amz-Date': b'20240807T235922Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA5VAA23GQIPQLFGM3/20240807/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=f83ce1940f571617724c5233197af36709407ef078ef5ae6dee42ed3c4bc6aba', 'amz-sdk-invocation-id': b'a96dbcf0-188f-45bf-9653-65cd19a1a0d2', 'amz-sdk-request': b'attempt=1', 'Content-Length': '134'}>
2024-08-08 05:29:22,310 botocore.httpsession [DEBUG] Certificate path: /Users/avanish/temp/aws-sts-v2-signing/.venv/lib/python3.12/site-packages/certifi/cacert.pem
2024-08-08 05:29:22,310 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): sts.amazonaws.com:443
2024-08-08 05:29:23,354 urllib3.connectionpool [DEBUG] https://sts.amazonaws.com:443 "POST / HTTP/11" 200 1028
2024-08-08 05:29:23,355 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'UUID', 'Content-Type': 'text/xml', 'Content-Length': '1028', 'Date': 'Wed, 07 Aug 2024 23:59:22 GMT'}
2024-08-08 05:29:23,355 botocore.parsers [DEBUG] Response body:
b'<AssumeRoleResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">\n  <AssumeRoleResult>\n    <AssumedRoleUser>\n      <AssumedRoleId>ASSUMEDAWSROLEID:Session1</AssumedRoleId>\n      <Arn>arn:aws:sts::999999999999:assumed-role/RootRoleName/Session1</Arn>\n    </AssumedRoleUser>\n    <Credentials>\n      <AccessKeyId>ASIAASIAASIAASIAASIA</AccessKeyId>\n      <SecretAccessKey>SomeSECRETAccessKey</SecretAccessKey>\n      <SessionToken>SomeTokenValue</SessionToken>\n      <Expiration>2024-08-08T00:59:23Z</Expiration>\n    </Credentials>\n  </AssumeRoleResult>\n  <ResponseMetadata>\n    <RequestId>UUID</RequestId>\n  </ResponseMetadata>\n</AssumeRoleResponse>\n'
2024-08-08 05:29:23,358 botocore.hooks [DEBUG] Event needs-retry.sts.AssumeRole: calling handler <botocore.retryhandler.RetryHandler object at 0x106757a70>
2024-08-08 05:29:23,358 botocore.retryhandler [DEBUG] No retry needed.
{
    "Credentials": {
        "AccessKeyId": "ASIAASIAASIAASIAASIA",
        "SecretAccessKey": "SomeSECRETAccessKey",
        "SessionToken": "SomeTokenValue",
        "Expiration": "2024-08-08 00:59:23+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "ASSUMEDAWSROLEID:Session1",
        "Arn": "arn:aws:sts::999999999999:assumed-role/RootRoleName/Session1"
    },
    "ResponseMetadata": {
        "RequestId": "UUID",
        "HTTPStatusCode": 200,
        "HTTPHeaders": {
            "x-amzn-requestid": "UUID",
            "content-type": "text/xml",
            "content-length": "1028",
            "date": "Wed, 07 Aug 2024 23:59:22 GMT"
        },
        "RetryAttempts": 0
    }
}
tim-finnigan commented 4 days ago

Hi thanks for following up. I found related documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/MrapOperations.html#use-presigned-url-mrap

As noted there:

To use SigV4A with temporary security credentials—for example, when using IAM roles—make sure that you request the temporary credentials from a Regional endpoint in AWS Security Token Service (AWS STS), instead of a global endpoint. If you use the global endpoint for AWS STS (sts.amazonaws.com), AWS STS will generate temporary credentials from a global endpoint, which isn't supported by Sig4A. As a result, you'll get an error. To resolve this issue, use any of the listed Regional endpoints for AWS STS.

It looks like you're using the global endpoint from the logs, can you try using a regional endpoint?

If still getting an error can you share the snippet you used to generate a presigned URL and reproduce the UnsupportedSignature error? Again here are the docs for that: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html. Also are you specifying the s3v4 config, for example:

from botocore.config import Config

my_config = Config(
    signature_version = 's3v4',
)

client = boto3.client('s3', config=my_config)