`--endpoint-url` parameter is not used when assuming a role given in a profile via `source_profile`

aronisstav commented 1 year ago

Describe the bug

I am using a privately hosted AWS-compatible platform and I am able to use aws-cli using the --endpoint parameter. I have configured a role and a user that can assume it without problem (aws --endpoint=... sts assume-role ... works fine).

I have also configured a profile that should automatically assume the role (via a role_arn field). However, when executing any aws-cli command using that profile and inspecting the --debug log I can see that the automatically generated AssumeRole request is targeting sts.amazonaws.com instead of the custom --endpoint.

Expected Behavior

Automatic requests to assume role should also go to an --endpoint if specified.

Current Behavior

Automatic requests to assume role go to sts.amazonaws.com, and fail.

Reproduction Steps

Setup a profile named withrole including the following info (it should also include a correct source_profile or equivalent)

[profile with-role]
role_arn=arn:aws:iam::123456789:role/myrole
source_profile=...

Run:

$ aws --endpoint=https://custom --profile with-role iam list-users --debug

Inspect the automatically generated AssumeRole request and verify that host is targetting the --endpoint and not sts.amazonaws.com.

Possible Solution

Not familiar enough with the code, but should be relatively easy to have automatic assumerole respect --endpoint, as its manual variant does.

Additional Information/Context

No response

CLI version used

$ aws --version
aws-cli/1.18.69 Python/3.8.10 Linux/5.10.16.3-microsoft-standard-WSL2 botocore/1.16.19

Environment details (OS name and version, etc.)

WSL2 with Ubuntu on Windows 10 (see above)

aBurmeseDev commented 1 year ago

Hi @aronisstav - thanks for reaching out. It looks like you're running older version of AWS CLI and it seems like version issue you're having. I would recommend updating it to later version and try it again. Here's docs on updating CLI for your reference: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#getting-started-install-instructions

aronisstav commented 1 year ago

Updating made this a little better, but not much. I verified that this behavious still exists in:

$ aws --version
aws-cli/2.9.17 Python/3.9.11 Linux/5.10.16.3-microsoft-standard-WSL2 exe/x86_64.ubuntu.20 prompt/off

... but now the request is towards:

host:sts.us-east-1.amazonaws.com

aBurmeseDev commented 1 year ago

@aronisstav - Thanks for following up. I just noticed that you have --endpoint parameter in your command instead of --endpoint-url which is the correct parameter name. That's probably the cause. Here's CLI docs for reference: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/list-users.html

aronisstav commented 1 year ago

@aBurmeseDev - Thank you for following up!

Nope, that is not it either, and logically so as the --endpoint specification, even if wrong, is recognized fine by the aws --endpoint=... sts assume-role ... command too...

The test I provided seems relatively easy to replicate:

$ aws --endpoint-url=https://custom --profile with-role iam list-users --debug

... and inspect the automatically generated AssumeRole request yo see that it is targetting sts.us-east-1.amazonaws.com and not http://custom.

Did you try it?

aBurmeseDev commented 1 year ago

Hi @aronisstav - thanks for following up. Unfortunately, I'm not seeing the behavior described on my end. Would you be able to share your debug logs here? That would give us more insight into the issue.

aronisstav commented 1 year ago

@aBurmeseDev - Of course. As I said, I expect the request to sts to be towards "http://custom", not sts.us-east-1.amazonaws.com.

$  aws --endpoint-url=https://custom --profile dev iam list-users --debug
2023-02-03 09:26:34,180 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.9.17 Python/3.9.11 Linux/5.10.16.3-microsoft-standard-WSL2 exe/x86_64.ubuntu.20
2023-02-03 09:26:34,180 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--endpoint-url=https://custom', '--profile', 'dev', 'iam', 'list-users', '--debug']
2023-02-03 09:26:34,543 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fc2cb808670>
2023-02-03 09:26:34,543 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fc2cb9621f0>
2023-02-03 09:26:34,543 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2023-02-03 09:26:34,543 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fc2cb9f3160>
2023-02-03 09:26:34,543 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fc2cb9f3f70>
2023-02-03 09:26:34,543 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fc2cb7a0040>
2023-02-03 09:26:34,543 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fc2cb928dc0>
2023-02-03 09:26:34,543 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2023-02-03 09:26:34,543 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fc2cb797280>
2023-02-03 09:26:34,544 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fc2cb742070>>
2023-02-03 09:26:34,544 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.9.17/dist/awscli/data/cli.json
2023-02-03 09:26:34,546 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fc2cb852f70>
2023-02-03 09:26:34,547 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fc2cb85daf0>
2023-02-03 09:26:34,547 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fc2cb85da60>
2023-02-03 09:26:34,547 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fc2cb85dc10>
2023-02-03 09:26:34,547 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fc2cb85db80>
2023-02-03 09:26:34,547 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fc2cb73c040>
2023-02-03 09:26:34,547 - MainThread - botocore.session - DEBUG - Setting config variable for profile to 'dev'
2023-02-03 09:26:34,548 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.9.17 Python/3.9.11 Linux/5.10.16.3-microsoft-standard-WSL2 exe/x86_64.ubuntu.20 prompt/off
2023-02-03 09:26:34,548 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--endpoint-url=https://custom', '--profile', 'dev', 'iam', 'list-users', '--debug']
2023-02-03 09:26:34,548 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fc2cb808ca0>
2023-02-03 09:26:34,548 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fc2cc231d30>
2023-02-03 09:26:34,548 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fc2cb774550>
2023-02-03 09:26:34,548 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fc2cc22e040>
2023-02-03 09:26:34,549 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fc2cc21ac10>
2023-02-03 09:26:34,550 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-02-03 09:26:34,551 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2023-02-03 09:26:34,551 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fc2cb928ca0>
2023-02-03 09:26:34,551 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fc2cb9cee50>
2023-02-03 09:26:34,564 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.9.17/dist/awscli/botocore/data/iam/2010-05-08/service-2.json
2023-02-03 09:26:34,579 - MainThread - botocore.hooks - DEBUG - Event building-command-table.iam: calling handler <function _add_wizard_command at 0x7fc2cb7744c0>
2023-02-03 09:26:34,579 - MainThread - botocore.hooks - DEBUG - Event building-command-table.iam: calling handler <function add_waiters at 0x7fc2cb797280>
2023-02-03 09:26:34,588 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.9.17/dist/awscli/botocore/data/iam/2010-05-08/waiters-2.json
2023-02-03 09:26:34,588 - MainThread - botocore.hooks - DEBUG - Event building-command-table.iam: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fc2cb742070>>
2023-02-03 09:26:34,589 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('path-prefix', <awscli.arguments.CLIArgument object at 0x7fc2cac2cf40>), ('marker', <awscli.arguments.CLIArgument object at 0x7fc2cac2cf10>), ('max-items', <awscli.arguments.CLIArgument object at 0x7fc2cac3e040>)])
2023-02-03 09:26:34,589 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.list-users: calling handler <function add_streaming_output_arg at 0x7fc2cb808f70>
2023-02-03 09:26:34,590 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.list-users: calling handler <function add_cli_input_json at 0x7fc2cc19e430>
2023-02-03 09:26:34,590 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.list-users: calling handler <function add_cli_input_yaml at 0x7fc2cc19e670>
2023-02-03 09:26:34,590 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.list-users: calling handler <function unify_paging_params at 0x7fc2cb962790>
2023-02-03 09:26:34,600 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.9.17/dist/awscli/botocore/data/iam/2010-05-08/paginators-1.json
2023-02-03 09:26:34,601 - MainThread - awscli.customizations.paginate - DEBUG - Modifying paging parameters for operation: ListUsers
2023-02-03 09:26:34,602 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.list-users: calling handler <function add_generate_skeleton at 0x7fc2cb852550>
2023-02-03 09:26:34,603 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.iam.list-users: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fc2cac3e130>>
2023-02-03 09:26:34,603 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.iam.list-users: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fc2cac3e160>>
2023-02-03 09:26:34,603 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.iam.list-users: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fc2cac3e310>>
2023-02-03 09:26:34,603 - MainThread - botocore.hooks - DEBUG - Event building-command-table.iam_list-users: calling handler <function add_waiters at 0x7fc2cb797280>
2023-02-03 09:26:34,603 - MainThread - botocore.hooks - DEBUG - Event building-command-table.iam_list-users: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fc2cb742070>>
2023-02-03 09:26:34,605 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.iam.list-users: calling handler functools.partial(<function check_should_enable_pagination at 0x7fc2cb9628b0>, ['marker', 'max-items'], {'max-items': <awscli.arguments.CLIArgument object at 0x7fc2cac3e040>}, OrderedDict([('path-prefix', <awscli.arguments.CLIArgument object at 0x7fc2cac2cf40>), ('marker', <awscli.arguments.CLIArgument object at 0x7fc2cac2cf10>), ('max-items', <awscli.customizations.paginate.PageArgument object at 0x7fc2cac3e2e0>), ('cli-input-json', <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fc2cac3e130>), ('cli-input-yaml', <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fc2cac3e160>), ('starting-token', <awscli.customizations.paginate.PageArgument object at 0x7fc2cac3e1f0>), ('page-size', <awscli.customizations.paginate.PageArgument object at 0x7fc2cac3e430>), ('generate-cli-skeleton', <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fc2cac3e310>)]))
2023-02-03 09:26:34,605 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.list-users.path-prefix: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc2caf0ab50>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.list-users.marker: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc2caf0ab50>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.list-users.max-items: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc2caf0ab50>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.list-users.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc2caf0ab50>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.list-users.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc2caf0ab50>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.list-users.starting-token: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc2caf0ab50>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.list-users.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc2caf0ab50>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.list-users.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc2caf0ab50>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event calling-command.iam.list-users: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fc2cac3e130>>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event calling-command.iam.list-users: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fc2cac3e160>>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event calling-command.iam.list-users: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fc2cac3e310>>
2023-02-03 09:26:34,606 - MainThread - botocore.hooks - DEBUG - Event calling-command.iam.list-users: calling handler functools.partial(<function check_should_enable_pagination_call_parameters at 0x7fc2cb962ca0>, ['Marker', 'MaxItems'])
2023-02-03 09:26:34,607 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2023-02-03 09:26:34,607 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2023-02-03 09:26:34,607 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2023-02-03 09:26:34,607 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2023-02-03 09:26:34,608 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2023-02-03 09:26:34,611 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.9.17/dist/awscli/botocore/data/endpoints.json
2023-02-03 09:26:34,627 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fc2cdbe2310>
2023-02-03 09:26:34,641 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.9.17/dist/awscli/botocore/data/iam/2010-05-08/endpoint-rule-set-1.json
2023-02-03 09:26:34,642 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.9.17/dist/awscli/botocore/data/partitions.json
2023-02-03 09:26:34,646 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.iam: calling handler <function add_generate_presigned_url at 0x7fc2cdebeaf0>
2023-02-03 09:26:34,646 - MainThread - botocore.regions - DEBUG - Using partition endpoint for iam, us-east-1: aws-global
2023-02-03 09:26:34,649 - MainThread - botocore.endpoint - DEBUG - Setting iam timeout as (60, 60)
2023-02-03 09:26:34,652 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'https://custom'}
2023-02-03 09:26:34,653 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://custom
2023-02-03 09:26:34,653 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.iam.ListUsers: calling handler <function base64_decode_input_blobs at 0x7fc2cb774ca0>
2023-02-03 09:26:34,653 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.iam.ListUsers: calling handler <function generate_idempotent_uuid at 0x7fc2cdbff280>
2023-02-03 09:26:34,654 - MainThread - botocore.hooks - DEBUG - Event before-call.iam.ListUsers: calling handler <function inject_api_version_header_if_needed at 0x7fc2cdc00af0>
2023-02-03 09:26:34,654 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListUsers) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.9.17 Python/3.9.11 Linux/5.10.16.3-microsoft-standard-WSL2 exe/x86_64.ubuntu.20 prompt/off command/iam.list-users'}, 'body': {'Action': 'ListUsers', 'Version': '2010-05-08'}, 'url': 'https://custom/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fc2ca6e3ee0>, 'has_streaming_input': False, 'auth_type': None}}
2023-02-03 09:26:34,654 - MainThread - botocore.hooks - DEBUG - Event request-created.iam.ListUsers: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fc2ca6e3f40>>
2023-02-03 09:26:34,654 - MainThread - botocore.hooks - DEBUG - Event choose-signer.iam.ListUsers: calling handler <function set_operation_specific_signer at 0x7fc2cdbff160>
2023-02-03 09:26:34,655 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fc2cdbe2310>
2023-02-03 09:26:34,656 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.9.17/dist/awscli/botocore/data/sts/2011-06-15/service-2.json
2023-02-03 09:26:34,658 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.9.17/dist/awscli/botocore/data/sts/2011-06-15/endpoint-rule-set-1.json
2023-02-03 09:26:34,659 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.sts: calling handler <function add_generate_presigned_url at 0x7fc2cdebeaf0>
2023-02-03 09:26:34,663 - MainThread - botocore.endpoint - DEBUG - Setting sts timeout as (60, 60)
2023-02-03 09:26:34,664 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False, 'UseGlobalEndpoint': False}
2023-02-03 09:26:34,665 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://sts.us-east-1.amazonaws.com
2023-02-03 09:26:34,665 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.sts.AssumeRole: calling handler <function base64_decode_input_blobs at 0x7fc2cb774ca0>
2023-02-03 09:26:34,665 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.sts.AssumeRole: calling handler <function generate_idempotent_uuid at 0x7fc2cdbff280>
2023-02-03 09:26:34,665 - MainThread - botocore.hooks - DEBUG - Event before-call.sts.AssumeRole: calling handler <function inject_api_version_header_if_needed at 0x7fc2cdc00af0>
2023-02-03 09:26:34,665 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=AssumeRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.9.17 Python/3.9.11 Linux/5.10.16.3-microsoft-standard-WSL2 exe/x86_64.ubuntu.20 prompt/off command/iam.list-users'}, 'body': {'Action': 'AssumeRole', 'Version': '2011-06-15', 'RoleArn': 'arn:aws:iam::824730019649:role/dev', 'RoleSessionName': 'botocore-session-1675412794'}, 'url': 'https://sts.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fc2ca6c2b50>, 'has_streaming_input': False, 'auth_type': None}}
2023-02-03 09:26:34,666 - MainThread - botocore.hooks - DEBUG - Event request-created.sts.AssumeRole: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fc2ca6c2b20>>
2023-02-03 09:26:34,666 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sts.AssumeRole: calling handler <function set_operation_specific_signer at 0x7fc2cdbff160>
2023-02-03 09:26:34,667 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2023-02-03 09:26:34,667 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.us-east-1.amazonaws.com
x-amz-date:20230203T082634Z

content-type;host;x-amz-date
4793aaf1098cf96ce0bdf31d60aff415f8527a425dc25b54e267a12472e21b7f
2023-02-03 09:26:34,667 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20230203T082634Z
20230203/us-east-1/sts/aws4_request
1bc4eb724be5d096d904c62e9c25e7931dcd7a1a4ced5b20569776e53a48fd51
2023-02-03 09:26:34,667 - MainThread - botocore.auth - DEBUG - Signature:
79e6a9361f53dc10ea0d89bed46e8019cdefa4db51705ceef041af2a30999730
2023-02-03 09:26:34,667 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.us-east-1.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.9.17 Python/3.9.11 Linux/5.10.16.3-microsoft-standard-WSL2 exe/x86_64.ubuntu.20 prompt/off command/iam.list-users', 'X-Amz-Date': b'20230203T082634Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=CMKSPZZCNZ3QH3BW1P3G/20230203/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=79e6a9361f53dc10ea0d89bed46e8019cdefa4db51705ceef041af2a30999730', 'Content-Length': '135'}>
2023-02-03 09:26:34,668 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.9.17/dist/awscli/botocore/cacert.pem
2023-02-03 09:26:34,669 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): sts.us-east-1.amazonaws.com:443
2023-02-03 09:26:40,398 - MainThread - urllib3.connectionpool - DEBUG - https://sts.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 403 306
2023-02-03 09:26:40,398 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '020e47d2-b348-424f-8e82-3d74d7c84da6', 'Content-Type': 'text/xml', 'Content-Length': '306', 'Date': 'Fri, 03 Feb 2023 08:26:39 GMT'}
2023-02-03 09:26:40,399 - MainThread - botocore.parsers - DEBUG - Response body:
b'<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>InvalidClientTokenId</Code>\n    <Message>The security token included in the request is invalid.</Message>\n  </Error>\n  <RequestId>020e47d2-b348-424f-8e82-3d74d7c84da6</RequestId>\n</ErrorResponse>\n'
2023-02-03 09:26:40,399 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sts.AssumeRole: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fc2ca5de7c0>>
2023-02-03 09:26:40,400 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-02-03 09:26:40,400 - MainThread - botocore.hooks - DEBUG - Event after-call.sts.AssumeRole: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fc2ca5de2b0>>
2023-02-03 09:26:40,400 - MainThread - botocore.credentials - WARNING - Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
  File "awscli/botocore/credentials.py", line 492, in _protected_refresh
  File "awscli/botocore/credentials.py", line 633, in fetch_credentials
  File "awscli/botocore/credentials.py", line 643, in _get_cached_credentials
  File "awscli/botocore/credentials.py", line 776, in _get_credentials
  File "awscli/botocore/client.py", line 341, in _api_call
  File "awscli/botocore/client.py", line 697, in _make_api_call
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.
2023-02-03 09:26:40,401 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 460, in main
  File "awscli/clidriver.py", line 595, in __call__
  File "awscli/clidriver.py", line 798, in __call__
  File "awscli/clidriver.py", line 931, in invoke
  File "awscli/clidriver.py", line 953, in _display_response
  File "awscli/formatter.py", line 82, in __call__
  File "awscli/botocore/paginate.py", line 446, in build_full_result
  File "awscli/botocore/paginate.py", line 252, in __iter__
  File "awscli/botocore/paginate.py", line 329, in _make_request
  File "awscli/botocore/client.py", line 341, in _api_call
  File "awscli/botocore/client.py", line 683, in _make_api_call
  File "awscli/botocore/client.py", line 703, in _make_request
  File "awscli/botocore/endpoint.py", line 101, in make_request
  File "awscli/botocore/endpoint.py", line 131, in _send_request
  File "awscli/botocore/endpoint.py", line 114, in create_request
  File "awscli/botocore/hooks.py", line 228, in emit
  File "awscli/botocore/hooks.py", line 211, in _emit
  File "awscli/botocore/signers.py", line 94, in handler
  File "awscli/botocore/signers.py", line 158, in sign
  File "awscli/botocore/signers.py", line 253, in get_auth_instance
  File "awscli/botocore/credentials.py", line 581, in get_frozen_credentials
  File "awscli/botocore/credentials.py", line 476, in _refresh
  File "awscli/botocore/credentials.py", line 492, in _protected_refresh
  File "awscli/botocore/credentials.py", line 633, in fetch_credentials
  File "awscli/botocore/credentials.py", line 643, in _get_cached_credentials
  File "awscli/botocore/credentials.py", line 776, in _get_credentials
  File "awscli/botocore/client.py", line 341, in _api_call
  File "awscli/botocore/client.py", line 697, in _make_api_call
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.

An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.

aronisstav commented 1 year ago

Also here is the content of my config:

$ cat ~/.aws/config
[profile admin]
region = us-east-1
[profile puc379]
region = us-east-1
[profile dev]
role_arn = arn:aws:iam::824730019649:role/dev
source_profile = puc379
region = us-east-1

aBurmeseDev commented 1 year ago

@aronisstav - thanks for these logs and additional information.

It seems like you're using profile dev in the config that probably does not have credentials for it in credentials because this error indicates there's a named profile in the ~/.aws/config that's missing credentials for the same named profile in the ~/.aws/credentials.

I'm not sure what your credential file looks like but you can try setting the default profile by running

set AWS_DEFAULT_PROFILE=<default_profile>

Also If you're using the CLI with MFA, you have to set the session token in addition to setting the access and secret keys. https://aws.amazon.com/premiumsupport/knowledge-center/authenticate-mfa-cli/

If the issue persists, I'd recommend removing ~/.aws/credentials file and re-run aws configure.

Here's docs on working with config variables for your reference: https://awscli.amazonaws.com/v2/documentation/api/latest/topic/config-vars.html

Hope that helps, John

aronisstav commented 1 year ago

@aBurmeseDev - thank you for following up (on a Sunday too)!

I don't think we are quite on track to the issue yet...

Indeed, as you wrote, "dev" does not have credentials configured, but:

the ~/.aws/config file shown in the last post shows that profile dev is sourcing credentials from profile puc379 to assume a given role via the source_profile parameter
The puc379 profile has correct credentials configured, as I explained in the original post (aws --endpoint-url=https://custom --profile puc379 sts assume-role arn:aws:iam::824730019649:role/dev works fine).
The problem is that the automatic call to assume the role during an operation using profile dev does not go towards the endpoint-url specified, but towards sts.us-east-1.amazonaws.com

Can you please confirm that the flow I am describing makes sense and endpoint-url should perhaps also cover the endpoint used for the automatic assume-role used when source_profile is specified?

aBurmeseDev commented 1 year ago

Hi @aronisstav - thank you for your patience while we investigate. I brought this up to the team discussion to verify and the team confirmed that this is expected behavior and not a bug but rather a feature request. Does this issue describes what you're seeing here?

Here's more on STS regionalized endpoints: https://docs.aws.amazon.com/sdkref/latest/guide/feature-sts-regionalized-endpoints.html

By default, AWS Security Token Service (AWS STS) is available as a global service, and all AWS STS requests go to a single endpoint at https://sts.amazonaws.com. Global requests map to the US East (N. Virginia) Region. AWS recommends using Regional AWS STS endpoints instead of the global endpoint. For more information on AWS STS endpoints, Endpoints in the AWS Security Token Service API Reference.

As far as the credential error you're seeing The security token included in the request is invalid., I would refer to the suggestions from my previous comment to troubleshoot.

aronisstav commented 1 year ago

Hi @aBurmeseDev ! I am indeed trying to use AWS CLI towards a privately hosted aws-like platform (I have perhaps incorrectly guessed that this was the only use of --endpoint-url). Given that the feature request exists from 2015, with no change happening, I am wondering if there are indeed other uses of --endpoint-url that require automatic role assumption to function in this way, or it should be considered to apply globally if given.

aBurmeseDev commented 2 months ago

Just wanted to check in here and share that you can now specify the endpoint to use for all service requests through the shared configuration file and environment variables, as well as specify the endpoint URL for individual AWS services.

Here's blogpost that was announced: https://aws.amazon.com/blogs/developer/new-improved-flexibility-when-configuring-endpoint-urls-with-the-aws-sdks-and-tools/

Closing this issue for now and if you have any other questions, please feel free to open a new issue.

github-actions[bot] commented 2 months ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

aws / aws-cli