search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
9.07k stars 1.87k forks source link

`--ssl-no-verify` / `--ca-bundle` / `AWS_CA_BUNDLE` is not honored for `AWS_CONTAINER_CREDENTIALS_FULL_URI` #4188

Closed synfinatic closed 4 months ago

synfinatic commented 4 months ago

Describe the bug

I've created my own service to implement the AWS_CONTAINER_CREDENTIALS_FULL_URI protocol to securely manage AWS Identity Center IAM Role credentials.

The problem is that my service must use a self-signed certificate since it listens on https://localhost:4144. In a perfect world, I would be able to add the self-signed cert (in PEM format) to the embedded AWS truststore, but this doesn't seem to work.

Expected Behavior

SSL validation passes and the AWS Boto3 SDK makes the necessary HTTPS request to fetch the credentials stored at the HTTP endpoint at AWS_CONTAINER_CREDENTIALS_FULL_URI.

Current Behavior

SSL validation does not pass. AWS Boto3 SDK is unable to fetch the necessary credentials and errors out.

Reproduction Steps

Testing with AWS_CA_BUNDLE:

$ export AWS_CONTAINER_CREDENTIALS_FULL_URI=https://localhost:4144/
$ export AWS_CA_BUNDLE=`pwd`/localhost.crt
$ aws sts get-caller-identity

SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)

--ca-bundle doesn't work either:

$ unset AWS_CA_BUNDLE
$ aws --ca-bundle=`pwd`/localhost.crt sts get-caller-identity

SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)

Trying to disable SSL validation all together doesn't seem to work either:

$ aws --ssl-no-verify sts get-caller-identity

SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)

Possible Solution

Honor the --ca-bundle, AWS_CA_BUNDLE and --no-ssl-verify flags/environment variables when connecting to the endpoint defined at AWS_CONTAINER_CREDENTIALS_FULL_URI

Additional Information/Context

No response

SDK version used

aws-cli/2.17.5 Python/3.11.9 Darwin/23.5.0 source/arm64

Environment details (OS name and version, etc.)

macOS 14.5

synfinatic commented 4 months ago

So I thought the issue might be the use of a self-signed certificate, but creating a server certificate signed by my internal CA and then passing that CA into aws by both methods does not solve the issue.

$ openssl s_client -CAfile CA.crt localhost:4144
Connecting to 127.0.0.1
CONNECTED(00000005)
Can't use SSL_get_servername
depth=1 CN=SynFindotNetCA, ST=California, L=San Jose, O=Syn Fin dot Net
verify return:1
depth=0 CN=localhost, C=US, ST=California, L=San Jose, O=Syn Fin dot Net, OU=aws-sso-cli
verify return:1
---
Certificate chain
 0 s:CN=localhost, C=US, ST=California, L=San Jose, O=Syn Fin dot Net, OU=aws-sso-cli
   i:CN=SynFindotNetCA, ST=California, L=San Jose, O=Syn Fin dot Net
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jul  6 15:30:28 2024 GMT; NotAfter: Jul  4 15:30:28 2034 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFnzCCA4egAwIBAgIBDDANBgkqhkiG9w0BAQsFADBbMRcwFQYDVQQDEw5TeW5G
aW5kb3ROZXRDQTETMBEGA1UECBMKQ2FsaWZvcm5pYTERMA8GA1UEBxMIU2FuIEpv
c2UxGDAWBgNVBAoTD1N5biBGaW4gZG90IE5ldDAeFw0yNDA3MDYxNTMwMjhaFw0z
NDA3MDQxNTMwMjhaMHkxEjAQBgNVBAMTCWxvY2FsaG9zdDELMAkGA1UEBhMCVVMx
EzARBgNVBAgTCkNhbGlmb3JuaWExETAPBgNVBAcTCFNhbiBKb3NlMRgwFgYDVQQK
Ew9TeW4gRmluIGRvdCBOZXQxFDASBgNVBAsTC2F3cy1zc28tY2xpMIIBIjANBgkq
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuA8jrATxYTcmRanYIzcGj4HWDU7SgWiM
b7CseCgt8H2FS9Yki/7P+5UaUKnRAfGNOqdl+upez/oP+cO3X+eYz1Yfn854BMCU
e8yiGk/wQAyzLx6AKGib+d3t1NHdAmKpQRqfBkekmVcI/6ba2E0IpyVT7A+gYZ19
ayfJrIwbnjc/U/DdnhXcrZ5jfnLDHKYenDGfnLon9axrtVYycb8+BQ00/RTv+nbS
Gi+Ny48e/B1HDlAK93ZMV1kYMV6QWTtiQy6dz314zN7FvPpJQfHA90CNasrFGoyD
xKcSOoXwi5YYT/mb/oAERkt1DTnjCPujgOiZCN21gQ5KAbOcWFQCzwIDAQABo4IB
TjCCAUowCQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBkAwCwYDVR0PBAQDAgWg
MDMGCWCGSAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBTZXJ2ZXIgQ2VydGlm
aWNhdGUwHQYDVR0OBBYEFBZuaKHF2jYOkNWT0UTifBKUWPYhMIGDBgNVHSMEfDB6
gBTXvRSzQDmHEiU4NXhDGwGVnuohZaFfpF0wWzEXMBUGA1UEAxMOU3luRmluZG90
TmV0Q0ExEzARBgNVBAgTCkNhbGlmb3JuaWExETAPBgNVBAcTCFNhbiBKb3NlMRgw
FgYDVQQKEw9TeW4gRmluIGRvdCBOZXSCAQAwJwYDVR0lBCAwHgYIKwYBBQUHAwEG
CCsGAQUFBwMCBggrBgEFBQgCAjAaBgNVHREEEzARgglsb2NhbGhvc3SHBH8AAAEw
DQYJKoZIhvcNAQELBQADggIBAIOrYd42F92dvclp8XS+k/cNOB3z02UL5R5PhEK6
PnHVkPNhlFouFjadREBIaUEl/WQaZ3wR2jmO7grKdlRBRE2pNkJMH33PhWC2iO8t
l24I5QeQTmVsmrADdDNil/Rb6iS3ItVtNwPYBZ0RHY3IKHXSZBlDAUefH+rm5YnS
8DV56vjx6tf4Jc9iJxD/NRCl9TUBiMiMsVXEAPLpYFqnLvpPT2PJMTnYM9YdilIu
+kI+1kaOurrznnGAhWNJCm/wfH9DRH2ar5ajuzLJKkWvPd7No3YEBdO3Y4odPHfP
M6ktyjQXl65LaJgatlLkIPY5I6DoKtXBVHqUhP2XTOC4xO4SUMT7l8+NqQAWTMI6
twRGZHqID0qCatCNu6ezeUzMKW0R9asVckwCJcZPzlU0DMbUsE7z+8IBL5fGZhDo
ZkqceZe1MSey511QtJX5y+/1ofggZsQkZtW0IdxU67Kn5VFcLMP4r/xc1DAyqvl0
MaMsA+GdaZh2NV9Gt8ANvB1j1WrOocHW6HIsCOU9rXwHHn6bYEFfMEQa4akwsfxj
D9wQcT5Go5RXcvPhERIc7xw2Ojsz6L0rg2XZA6tgxe+q24bv2+4BwYcjNj6VVfIW
kj3uTGyXf/MZ1VQD7zihKCr3jNza87LidXBnksawwT27eQHqQ+9NtPJTv1Oeh9M+
iPkd
-----END CERTIFICATE-----
subject=CN=localhost, C=US, ST=California, L=San Jose, O=Syn Fin dot Net, OU=aws-sso-cli
issuer=CN=SynFindotNetCA, ST=California, L=San Jose, O=Syn Fin dot Net
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1983 bytes and written 363 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: A74AC870CE2D4AD3DE7E3A273F8EC43EC9E8CDE1318CE19C71F8C5B242825DD0
    Session-ID-ctx:
    Resumption PSK: B4DA1DAB2D2D495B6FF489C6D58509750FE94A32B4BC9FD9AFD24681402C47C2
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 6e d3 12 d5 03 e8 61 e5-0d 62 25 a2 41 09 e7 21   n.....a..b%.A..!
    0010 - 67 77 87 cc ad 80 92 6a-b1 e8 eb fd ec d4 a1 12   gw.....j........
    0020 - b8 6b fe 00 7e ca 1b 0b-8a 4d 4e 38 dc 6a 09 b4   .k..~....MN8.j..
    0030 - 1e 7c 6a f0 73 a0 2a d0-7f d2 09 1f 02 5b 38 d5   .|j.s.*......[8.
    0040 - d4 88 2f 86 89 ce 41 58-81 a0 88 10 3c 23 fc ea   ../...AX....<#..
    0050 - f2 d4 a3 b0 9d 01 f1 79-0f b7 2f ab 74 f3 2b 8b   .......y../.t.+.
    0060 - 1b b8 6b 75 45 03 c8 ba-79                        ..kuE...y

    Start Time: 1720280214
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

output from aws-cli:

$  aws sts get-caller-identity --ca-bundle CA.crt --no-verify-ssl --debug
2024-07-06 08:40:11,120 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.5 Python/3.11.9 Darwin/23.5.0 source/arm64
2024-07-06 08:40:11,120 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sts', 'get-caller-identity', '--ca-bundle', 'CA.crt', '--no-verify-ssl', '--debug']
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x103a544a0>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x10386ade0>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x1034a7560>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x1034c8c20>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x103a6c540>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x1038c2200>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x103a574c0>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x103b2ded0>>
2024-07-06 08:40:11,127 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/data/cli.json
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x103982160>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x103982480>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x1039823e0>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x1039825c0>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x103982520>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x103b27280>
2024-07-06 08:40:11,128 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.5 Python/3.11.9 Darwin/23.5.0 source/arm64
2024-07-06 08:40:11,128 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sts', 'get-caller-identity', '--ca-bundle', 'CA.crt', '--no-verify-ssl', '--debug']
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x103a54ea0>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x1032c3e20>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x103af0fe0>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x103163600>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x1032cbce0>
2024-07-06 08:40:11,129 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-07-06 08:40:11,133 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x1038c19e0>
2024-07-06 08:40:11,133 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x10384ae80>
2024-07-06 08:40:11,139 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/data/sts/2011-06-15/service-2.json
2024-07-06 08:40:11,140 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts: calling handler <function add_waiters at 0x103a574c0>
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x103b2ded0>>
2024-07-06 08:40:11,146 - MainThread - awscli.clidriver - DEBUG - OrderedDict()
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_streaming_output_arg at 0x103a554e0>
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_cli_input_json at 0x1032eca40>
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_cli_input_yaml at 0x1032ecae0>
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function unify_paging_params at 0x10386b560>
2024-07-06 08:40:11,152 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/data/sts/2011-06-15/paginators-1.json
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_generate_skeleton at 0x103980860>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.get-caller-identity: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x103babed0>>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.get-caller-identity: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x1031ec2d0>>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.get-caller-identity: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x103ba4250>>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts_get-caller-identity: calling handler <function add_waiters at 0x103a574c0>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts_get-caller-identity: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x103b2ded0>>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.get-caller-identity.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x100f2e150>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.get-caller-identity.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x100f2e150>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.get-caller-identity.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x100f2e150>
2024-07-06 08:40:11,153 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.get-caller-identity: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x103babed0>>
2024-07-06 08:40:11,153 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.get-caller-identity: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x1031ec2d0>>
2024-07-06 08:40:11,153 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.get-caller-identity: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x103ba4250>>
2024-07-06 08:40:11,153 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-07-06 08:40:11,153 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2024-07-06 08:40:11,154 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Could not connect to the endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
OSError: [Errno 64] Host is down

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 448, in send
    urllib_response = conn.urlopen(
                      ^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 801, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/retry.py", line 527, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1303, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 94, in _send_request
    rval = super(AWSConnection, self)._send_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1349, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1298, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 122, in _send_output
    self.send(msg)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 206, in send
    return super(AWSConnection, self).send(str)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 996, in send
    self.connect()
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x103bbff50>: Failed to establish a new connection: [Errno 64] Host is down

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 387, in _fetch_metadata_token
    response = self._session.send(request.prepare())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 477, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://169.254.169.254/latest/api/token"
2024-07-06 08:40:11,160 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (2): 169.254.169.254:80
2024-07-06 08:40:11,160 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/placement/availability-zone/: Could not connect to the endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
OSError: [Errno 64] Host is down

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 448, in send
    urllib_response = conn.urlopen(
                      ^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 801, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/retry.py", line 527, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1303, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 94, in _send_request
    rval = super(AWSConnection, self)._send_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1349, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1298, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 122, in _send_output
    self.send(msg)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 206, in send
    return super(AWSConnection, self).send(str)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 996, in send
    self.connect()
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x103c6a2d0>: Failed to establish a new connection: [Errno 64] Host is down

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 437, in _get_request
    response = self._session.send(request.prepare())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 477, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
2024-07-06 08:40:11,161 - MainThread - awscli.utils - DEBUG - Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: custom-process
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2024-07-06 08:40:11,164 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2024-07-06 08:40:11,164 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2024-07-06 08:40:11,164 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2024-07-06 08:40:11,164 - MainThread - botocore.httpsession - DEBUG - Certificate path: /opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/cacert.pem
2024-07-06 08:40:11,164 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): localhost:4144
2024-07-06 08:40:11,176 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn
    conn.connect()
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 448, in send
    urllib_response = conn.urlopen(
                      ^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 801, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/retry.py", line 527, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn
    conn.connect()
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 499, in main
    return command_table[parsed_args.command](remaining, parsed_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 634, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 837, in __call__
    return self._operation_caller.invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 959, in invoke
    client = self._session.create_client(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/session.py", line 850, in create_client
    credentials = self.get_credentials()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/session.py", line 456, in get_credentials
    'credential_provider').load_credentials()
                           ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/credentials.py", line 1974, in load_credentials
    creds = provider.load()
            ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/credentials.py", line 1829, in load
    return self._retrieve_or_fail()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/credentials.py", line 1838, in _retrieve_or_fail
    creds = fetcher()
            ^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/credentials.py", line 1867, in fetch_creds
    response = self._fetcher.retrieve_full_uri(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 2796, in retrieve_full_uri
    return self._retrieve_credentials(full_url, headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 2841, in _retrieve_credentials
    return self._get_response(
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 2855, in _get_response
    response = self._session.send(request.prepare())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 475, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
synfinatic commented 4 months ago

So I tried using the Go SDK v2 and looks like it's having the same issue, so this seems more of a general AWS SDK issue, and not specific to the Python/boto3 SDK:

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/sts"
)

func main() {
    ca, err := os.Open("./CA.crt")
    if err != nil {
        panic(fmt.Sprintf("Unable to open CA.crt: %v", err))
    }
    cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithCustomCABundle(ca))
    if err != nil {
        panic(err)
    }

    client := sts.NewFromConfig(cfg)
    output, err := client.GetCallerIdentity(context.TODO(), nil)
    if err != nil {
        panic(err)
    }
    fmt.Printf("identity: %v\n", output)
}
$ ./aws-test
panic: operation error STS: GetCallerIdentity, get identity: get credentials: failed to refresh cached credentials, failed to 
load credentials, exceeded maximum number of attempts, 3, request send failed, Get "https://localhost:4144/": tls: failed 
to verify certificate: x509: “localhost” certificate is not trusted

goroutine 1 [running]:
main.main()
    /Users/aturner/go/src/github.com/synfinatic/aws-sso-cli/cmd/aws-test/main.go:25 +0x194
tim-finnigan commented 4 months ago

Thanks for reaching out. Here is where AWS_CONTAINER_CREDENTIALS_FULL_URI is documented in the SDKs & Tools Reference Guide: https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html. Also referenced there:

If you use Amazon ECS, we recommend you use a task IAM Role for improved credential isolation, authorization, and auditability. When configured, Amazon ECS sets the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable that the SDKs and tools use to obtain credentials.

Have you tried that approach for ECS?

Can you also confirm that your certificate is valid, and no proxy/network configurations are causing the error here? This troubleshooting guide on SSL validation errors highlights common causes: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-troubleshooting.html#tshoot-certificate-verify-failed

Otherwise since this issue applies to behavior across SDKs (as environment variables such as that one are used across AWS SDKs), could you create an issue for this in our cross-SDK repository sharing your findings up to this point?

synfinatic commented 4 months ago

Thanks. Opened https://github.com/aws/aws-cli/issues/9016

github-actions[bot] commented 4 months ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.