Environment variables not interpolated in credentials

mgryszko commented 3 years ago

Confirm by changing [ ] to [x] below to ensure that it's a bug:

[x] I've gone though the User Guide and the API reference
[x] I've searched for previous similar issues and didn't find any solution

Describe the bug Credentials cannot be set using environment variable interpolation.

SDK version number aws-cli/2.1.34 Python/3.9.2 Darwin/20.3.0 source/x86_64 prompt/off

Platform/OS/Hardware/Device macOS BigSur

To Reproduce (observed behavior)

Define env variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Create ~/.aws/credentials with the following content:

[root]
aws_access_key_id = %(AWS_ACCESS_KEY_ID)
aws_secret_access_key = %(AWS_SECRET_ACCESS_KEY)

Assuming here Python configparser syntax.

aws configure list output:

  Name                    Value             Type    Location
  ----                    -----             ----    --------
profile                <not set>             None    None
access_key     ****************MXXV              env
secret_key     ****************pBQI              env
region             eu-central-1      config-file    ~/.aws/config

Credentials are read

aws --profile default --region eu-central-1 s3 ls output:

An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.

Expected behavior Environment variables are interpolated correctly by commands in the credentials file.

Logs/output

aws --profile default --region eu-central-1 --debug s3 ls                                                           254 ↵  11.0.10+9 ☕  INSERT
2021-04-09 07:49:05,975 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.1.34 Python/3.9.2 Darwin/20.3.0 source/x86_64
2021-04-09 07:49:05,975 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--profile', 'default', '--region', 'eu-central-1', '--debug', 's3', 'ls']
2021-04-09 07:49:06,004 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x105a373a0>
2021-04-09 07:49:06,004 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x105899b80>
2021-04-09 07:49:06,005 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2021-04-09 07:49:06,005 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x105831160>
2021-04-09 07:49:06,005 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x1058400d0>
2021-04-09 07:49:06,005 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x105a47d30>
2021-04-09 07:49:06,005 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x1058e8670>
2021-04-09 07:49:06,005 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2021-04-09 07:49:06,005 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x105a40f70>
2021-04-09 07:49:06,005 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/awscli/data/cli.json
2021-04-09 07:49:06,009 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x1059a3310>
2021-04-09 07:49:06,009 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x1059a3e50>
2021-04-09 07:49:06,009 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x1059a3dc0>
2021-04-09 07:49:06,009 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x1059a3f70>
2021-04-09 07:49:06,009 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x1059a3ee0>
2021-04-09 07:49:06,009 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x105ae3840>
2021-04-09 07:49:06,009 - MainThread - botocore.session - DEBUG - Setting config variable for profile to 'default'
2021-04-09 07:49:06,009 - MainThread - botocore.session - DEBUG - Setting config variable for region to 'eu-central-1'
2021-04-09 07:49:06,009 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.1.34 Python/3.9.2 Darwin/20.3.0 source/x86_64 prompt/off
2021-04-09 07:49:06,009 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--profile', 'default', '--region', 'eu-central-1', '--debug', 's3', 'ls']
2021-04-09 07:49:06,009 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x105a379d0>
2021-04-09 07:49:06,009 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x105530c10>
2021-04-09 07:49:06,010 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x105a9e430>
2021-04-09 07:49:06,010 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x10552adc0>
2021-04-09 07:49:06,010 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x1055915e0>
2021-04-09 07:49:06,012 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-04-09 07:49:06,019 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2021-04-09 07:49:06,019 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x1058e8550>
2021-04-09 07:49:06,019 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x105892a60>
2021-04-09 07:49:06,019 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x105a40f70>
2021-04-09 07:49:06,020 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x105b2dc10>
2021-04-09 07:49:06,020 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <function add_waiters at 0x105a40f70>
2021-04-09 07:49:06,021 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x105b2dc10>
2021-04-09 07:49:06,021 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x105b2dc10>
2021-04-09 07:49:06,021 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x105b2dc10>
2021-04-09 07:49:06,021 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x105b2dc10>
2021-04-09 07:49:06,021 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10555b250>
2021-04-09 07:49:06,021 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x105b2dc10>
2021-04-09 07:49:06,021 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10555b250>
2021-04-09 07:49:06,021 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x105b2dc10>
2021-04-09 07:49:06,022 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2021-04-09 07:49:06,022 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2021-04-09 07:49:06,022 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2021-04-09 07:49:06,022 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2021-04-09 07:49:06,022 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2021-04-09 07:49:06,023 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/data/endpoints.json
2021-04-09 07:49:06,030 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x104b92ee0>
2021-04-09 07:49:06,059 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/data/s3/2006-03-01/service-2.json
2021-04-09 07:49:06,070 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x104b3e940>
2021-04-09 07:49:06,070 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x104b3e700>
2021-04-09 07:49:06,075 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2021-04-09 07:49:06,076 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.ListBuckets: calling handler <function base64_decode_input_blobs at 0x105a9eb80>
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler <function validate_bucket_name at 0x104bb81f0>
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x10a02f2e0>>
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x10a02f340>>
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler <function generate_idempotent_uuid at 0x104bb8040>
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListBuckets: calling handler <function add_expect_header at 0x104bb8550>
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListBuckets: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x10a02f2e0>>
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListBuckets: calling handler <function inject_api_version_header_if_needed at 0x104bb98b0>
2021-04-09 07:49:06,077 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListBuckets) with params: {'url_path': '/', 'query_string': '', 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.1.34 Python/3.9.2 Darwin/20.3.0 source/x86_64 prompt/off command/s3.ls'}, 'body': b'', 'url': 'https://s3.eu-central-1.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x105fb9760>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': None}}}
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListBuckets: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x105fb9730>>
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListBuckets: calling handler <function set_operation_specific_signer at 0x104bb3ee0>
2021-04-09 07:49:06,077 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListBuckets: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x10a02f400>>
2021-04-09 07:49:06,077 - MainThread - botocore.utils - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2021-04-09 07:49:06,078 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x104b92ee0>
2021-04-09 07:49:06,078 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/data/sts/2011-06-15/service-2.json
2021-04-09 07:49:06,080 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.sts: calling handler <function add_generate_presigned_url at 0x104b3e700>
2021-04-09 07:49:06,083 - MainThread - botocore.endpoint - DEBUG - Setting sts timeout as (60, 60)
2021-04-09 07:49:06,083 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.sts.AssumeRole: calling handler <function base64_decode_input_blobs at 0x105a9eb80>
2021-04-09 07:49:06,084 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.sts.AssumeRole: calling handler <function generate_idempotent_uuid at 0x104bb8040>
2021-04-09 07:49:06,084 - MainThread - botocore.hooks - DEBUG - Event before-call.sts.AssumeRole: calling handler <function inject_api_version_header_if_needed at 0x104bb98b0>
2021-04-09 07:49:06,084 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=AssumeRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.1.34 Python/3.9.2 Darwin/20.3.0 source/x86_64 prompt/off command/s3.ls'}, 'body': {'Action': 'AssumeRole', 'Version': '2011-06-15', 'RoleArn': 'arn:aws:iam::064436394451:role/federateclarity', 'RoleSessionName': 'botocore-session-1617947346'}, 'url': 'https://sts.eu-central-1.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x10a02feb0>, 'has_streaming_input': False, 'auth_type': None}}
2021-04-09 07:49:06,084 - MainThread - botocore.hooks - DEBUG - Event request-created.sts.AssumeRole: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x10a02fe80>>
2021-04-09 07:49:06,084 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sts.AssumeRole: calling handler <function set_operation_specific_signer at 0x104bb3ee0>
2021-04-09 07:49:06,084 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-04-09 07:49:06,085 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.eu-central-1.amazonaws.com
x-amz-date:20210409T054906Z

content-type;host;x-amz-date
c0e3f7cafac0b154bb914b8784587a5b49e4c4d8c11b9e1eab218c222f6d1ce7
2021-04-09 07:49:06,085 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210409T054906Z
20210409/eu-central-1/sts/aws4_request
11a77f9f11ff207852088d3d355515d29f6d35d2be58b390036e5d5c671c61a1
2021-04-09 07:49:06,085 - MainThread - botocore.auth - DEBUG - Signature:
c74dc998dd40683e4ed034a38e1bc794d507e3591b9a760b71b55e8bda9cbcac
2021-04-09 07:49:06,085 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.eu-central-1.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.1.34 Python/3.9.2 Darwin/20.3.0 source/x86_64 prompt/off command/s3.ls', 'X-Amz-Date': b'20210409T054906Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=%(AWS_ACCESS_KEY_ID)/20210409/eu-central-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=c74dc998dd40683e4ed034a38e1bc794d507e3591b9a760b71b55e8bda9cbcac', 'Content-Length': '147'}>
2021-04-09 07:49:06,085 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/cacert.pem
2021-04-09 07:49:06,086 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): sts.eu-central-1.amazonaws.com:443
2021-04-09 07:49:06,229 - MainThread - urllib3.connectionpool - DEBUG - https://sts.eu-central-1.amazonaws.com:443 "POST / HTTP/1.1" 403 306
2021-04-09 07:49:06,230 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '615b8504-4816-4a7a-b08d-494648d76130', 'Content-Type': 'text/xml', 'Content-Length': '306', 'Date': 'Fri, 09 Apr 2021 05:49:06 GMT'}
2021-04-09 07:49:06,230 - MainThread - botocore.parsers - DEBUG - Response body:
b'<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>InvalidClientTokenId</Code>\n    <Message>The security token included in the request is invalid.</Message>\n  </Error>\n  <RequestId>615b8504-4816-4a7a-b08d-494648d76130</RequestId>\n</ErrorResponse>\n'
2021-04-09 07:49:06,231 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sts.AssumeRole: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x10a098910>>
2021-04-09 07:49:06,231 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2021-04-09 07:49:06,231 - MainThread - botocore.hooks - DEBUG - Event after-call.sts.AssumeRole: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x10a098490>>
2021-04-09 07:49:06,232 - MainThread - botocore.credentials - WARNING - Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 529, in _protected_refresh
    metadata = self._refresh_using()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 670, in fetch_credentials
    return self._get_cached_credentials()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 680, in _get_cached_credentials
    response = self._get_credentials()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 813, in _get_credentials
    return client.assume_role(**kwargs)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/client.py", line 249, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/client.py", line 568, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.
2021-04-09 07:49:06,234 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/awscli/clidriver.py", line 457, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/awscli/customizations/commands.py", line 197, in __call__
    return self.subcommand_table[parsed_args.subcommand](remaining,
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/awscli/customizations/commands.py", line 191, in __call__
    rc = self._run_main(parsed_args, parsed_globals)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/awscli/customizations/s3/subcommands.py", line 505, in _run_main
    self._list_all_buckets()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/awscli/customizations/s3/subcommands.py", line 570, in _list_all_buckets
    response_data = self.client.list_buckets()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/client.py", line 249, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/client.py", line 554, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/client.py", line 574, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 115, in create_request
    self._event_emitter.emit(event_name, request=request,
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/signers.py", line 154, in sign
    auth = self.get_auth_instance(**kwargs)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/signers.py", line 234, in get_auth_instance
    frozen_credentials = self._credentials.get_frozen_credentials()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 618, in get_frozen_credentials
    self._refresh()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 513, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 529, in _protected_refresh
    metadata = self._refresh_using()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 670, in fetch_credentials
    return self._get_cached_credentials()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 680, in _get_cached_credentials
    response = self._get_credentials()
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/credentials.py", line 813, in _get_credentials
    return client.assume_role(**kwargs)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/client.py", line 249, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/awscli/2.1.34/libexec/lib/python3.9/site-packages/botocore/client.py", line 568, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.

An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.

Additional context Profiles are defined in ~/.aws/config inheriting from root:

[profile root]
region=eu-central-1
output=json

[profile default]
region=eu-central-1
source_profile=root
role_arn=arn:aws:iam::XXX:role/YYY
output=json

kdaily commented 3 years ago

Hi @mgryszko,

Thanks for opening. Based on your example, it looks like you're trying to work around not being able to use credential_source without assuming a role (since you're chaining here), or alongside source_profile. This is by design as explicitly using a profile is intended to override the precedence of using environment variables:

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-precedence

Your request is noted though, and is similar to another recent issue: https://github.com/aws/aws-cli/issues/6042#issuecomment-807073577

If you can provide some more background as to what you are limited in doing based on the current behavior, that would be useful feedback.

As to the technical behavior, this functionality is part of botocore, and a configparser.RawConfigParser object is used:

https://github.com/boto/botocore/blob/bf94065aa55a347f0dba60138dab0b2da30fb507/botocore/configloader.py#L147

Interpolation is not used by default in a RawConfigParser:

https://docs.python.org/3/library/configparser.html#rawconfigparser-objects

Legacy variant of the ConfigParser. It has interpolation disabled by default and allows for non-string section names, option names, and values via its unsafe add_section and set methods, as well as the legacy defaults= keyword argument handling.

mgryszko commented 3 years ago

Hi @kdaily,

Huge thanks for so detailed response.

Regarding the configuration precedence - according to the documentation link you provided, it looks like CLI options trump over env variables, env variables trump over CLI credentials file, etc. Even if my default profile inherits from the root profile, if I define the environment variables (AWS_...), they should override what's in the config/credentials file. At least this is my understanding when reading the documentation. Maybe the doc be updated to reflect this behaviour?

Regarding my use case: I was trying to execute aws-cli in a Docker container. The Docker image has a generic .aws/credentials file with aws_access_key_id and aws_secret_access_key pointing to env variables. The env variables are passed through the docker-compose.yml.

tim-finnigan commented 2 years ago

The documentation linked earlier does describe how environment variables take precedence over profile configurations. I'm not sure if I understand what the discrepancy is in the documentation. Could you please elaborate on which part you think should be updated/clarified?

github-actions[bot] commented 2 years ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

aws / aws-cli

Environment variables not interpolated in credentials #6082