search

aws / aws-cli

Universal Command Line Interface for Amazon Web Services
Other
15.56k stars 4.13k forks source link

sqs commands seem to ignore `AWS_ENDPOINT_URL` #8737

Open thrau opened 5 months ago

thrau commented 5 months ago

Describe the bug

When using AWS_ENDPOINT_URL together with aws sqs, it seems that specifically SQS commands are resolving queue.amazonaws.com as endpoint provider instead of what is specified in AWS_ENDPOINT_URL

Expected Behavior

The following commands should resolve to localhost:4566:

export AWS_ENDPOINT_URL=http://localhost:4566
aws --debug sqs list-queues

Current Behavior

However, I can see they're being directed to queue.amazonaws.com:

2024-06-12 00:59:12,132 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://queue.amazonaws.com/, headers={'X-Amz-Target': b'AmazonSQS.ListQueues', 'Content-Type': b'application/x-amz-json-1.0', 'User-Agent': b'aws-cli/1.33.6 md/Botocore#1.34.124 ua/2.0 os/linux#5.15.0-107-generic md/arch#x86_64 lang/python#3.11.5 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.34.124', 'X-Amz-Date': b'20240611T225912Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=test/20240611/us-east-1/sqs/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=df4bfb5f9ad0bbfd3f0bc5b444982a124b48c19842fa51b72af920f18aaec8d4', 'amz-sdk-invocation-id': b'823975a9-2302-4f56-a997-a2732a15baf3', 'amz-sdk-request': b'attempt=1', 'Content-Length': '2'}>

Full debug output:

export AWS_ENDPOINT_URL=http://localhost:4566
thomas@ninox ~ % aws --debug sqs list-queues                                    
2024-06-12 00:57:03,434 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.33.6 Python/3.11.5 Linux/5.15.0-107-generic botocore/1.34.124
2024-06-12 00:57:03,434 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 'sqs', 'list-queues']
2024-06-12 00:57:03,434 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f397ea5fe20>
2024-06-12 00:57:03,434 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7f397ef87600>
2024-06-12 00:57:03,435 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f397efd5260>
2024-06-12 00:57:03,436 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-06-12 00:57:03,437 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f397ec87380>
2024-06-12 00:57:03,438 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/sqs/2012-11-05/service-2.json.gz
2024-06-12 00:57:03,439 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sqs: calling handler <function add_waiters at 0x7f397ea6a480>
2024-06-12 00:57:03,444 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('queue-name-prefix', <awscli.arguments.CLIArgument object at 0x7f397e9ae150>), ('next-token', <awscli.arguments.CLIArgument object at 0x7f397e9ae210>), ('max-results', <awscli.arguments.CLIArgument object at 0x7f397e9ae3d0>)])
2024-06-12 00:57:03,444 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sqs.list-queues: calling handler <function add_streaming_output_arg at 0x7f397ea68860>
2024-06-12 00:57:03,444 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sqs.list-queues: calling handler <function add_cli_input_json at 0x7f397efd5f80>
2024-06-12 00:57:03,444 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sqs.list-queues: calling handler <function unify_paging_params at 0x7f397ebd5ee0>
2024-06-12 00:57:03,449 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/sqs/2012-11-05/paginators-1.json
2024-06-12 00:57:03,449 - MainThread - awscli.customizations.paginate - DEBUG - Modifying paging parameters for operation: ListQueues
2024-06-12 00:57:03,449 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sqs.list-queues: calling handler <function add_generate_skeleton at 0x7f397ebb99e0>
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sqs.list-queues: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7f397e9ae890>>
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sqs.list-queues: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f397e9bca90>>
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sqs.list-queues: calling handler <function update_endpoint_url at 0x7f397ea85bc0>
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.sqs.list-queues: calling handler functools.partial(<function check_should_enable_pagination at 0x7f397ebd6020>, ['next-token', 'max-results'], {}, OrderedDict([('queue-name-prefix', <awscli.arguments.CLIArgument object at 0x7f397e9ae150>), ('next-token', <awscli.arguments.CLIArgument object at 0x7f397e9ae210>), ('max-results', <awscli.arguments.CLIArgument object at 0x7f397e9ae3d0>), ('cli-input-json', <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7f397e9ae890>), ('starting-token', <awscli.customizations.paginate.PageArgument object at 0x7f397f279110>), ('page-size', <awscli.customizations.paginate.PageArgument object at 0x7f397edd0f50>), ('max-items', <awscli.customizations.paginate.PageArgument object at 0x7f397efa4690>), ('generate-cli-skeleton', <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f397e9bca90>)]))
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.queue-name-prefix: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f397ef22150>
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.next-token: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f397ef22150>
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.max-results: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f397ef22150>
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f397ef22150>
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.starting-token: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f397ef22150>
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f397ef22150>
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.max-items: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f397ef22150>
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f397ef22150>
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event calling-command.sqs.list-queues: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7f397e9ae890>>
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event calling-command.sqs.list-queues: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f397e9bca90>>
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2024-06-12 00:57:03,452 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/endpoints.json
2024-06-12 00:57:03,478 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2024-06-12 00:57:03,478 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f397f62eca0>
2024-06-12 00:57:03,484 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/sqs/2012-11-05/endpoint-rule-set-1.json.gz
2024-06-12 00:57:03,484 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/partitions.json
2024-06-12 00:57:03,485 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f397f764900>
2024-06-12 00:57:03,486 - MainThread - botocore.endpoint - DEBUG - Setting sqs timeout as (60, 60)
2024-06-12 00:57:03,487 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/_retry.json
2024-06-12 00:57:03,487 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: sqs
2024-06-12 00:57:03,488 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.sqs.ListQueues: calling handler <function generate_idempotent_uuid at 0x7f397f6509a0>
2024-06-12 00:57:03,488 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'https://queue.amazonaws.com'}
2024-06-12 00:57:03,488 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://queue.amazonaws.com
2024-06-12 00:57:03,488 - MainThread - botocore.hooks - DEBUG - Event before-call.sqs.ListQueues: calling handler <function add_recursion_detection_header at 0x7f397f62f920>
2024-06-12 00:57:03,489 - MainThread - botocore.hooks - DEBUG - Event before-call.sqs.ListQueues: calling handler <function inject_api_version_header_if_needed at 0x7f397f652480>
2024-06-12 00:57:03,489 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListQueues) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonSQS.ListQueues', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'aws-cli/1.33.6 md/Botocore#1.34.124 ua/2.0 os/linux#5.15.0-107-generic md/arch#x86_64 lang/python#3.11.5 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.34.124'}, 'body': b'{}', 'url': 'https://queue.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f397e9beb50>, 'has_streaming_input': False, 'auth_type': None}}
2024-06-12 00:57:03,489 - MainThread - botocore.hooks - DEBUG - Event request-created.sqs.ListQueues: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f397ece6a50>>
2024-06-12 00:57:03,489 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sqs.ListQueues: calling handler <function set_operation_specific_signer at 0x7f397f650860>
2024-06-12 00:57:03,489 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-06-12 00:57:03,489 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.0
host:queue.amazonaws.com
x-amz-date:20240611T225703Z
x-amz-target:AmazonSQS.ListQueues

content-type;host;x-amz-date;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
2024-06-12 00:57:03,489 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240611T225703Z
20240611/us-east-1/sqs/aws4_request
e4bbdedc4cf6facf495b206eec8ddb2eebcc5dc8dcecdf25f8735622bba7ad94
2024-06-12 00:57:03,489 - MainThread - botocore.auth - DEBUG - Signature:
4b582078a1798d994b58538e4997fe332bbc6905eaa53df8d9f5316792921df2
2024-06-12 00:57:03,489 - MainThread - botocore.hooks - DEBUG - Event request-created.sqs.ListQueues: calling handler <function add_retry_headers at 0x7f397f652ca0>
2024-06-12 00:57:03,490 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://queue.amazonaws.com/, headers={'X-Amz-Target': b'AmazonSQS.ListQueues', 'Content-Type': b'application/x-amz-json-1.0', 'User-Agent': b'aws-cli/1.33.6 md/Botocore#1.34.124 ua/2.0 os/linux#5.15.0-107-generic md/arch#x86_64 lang/python#3.11.5 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.34.124', 'X-Amz-Date': b'20240611T225703Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=test/20240611/us-east-1/sqs/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=4b582078a1798d994b58538e4997fe332bbc6905eaa53df8d9f5316792921df2', 'amz-sdk-invocation-id': b'bd45b163-4df4-47a2-a64d-876a0661a32c', 'amz-sdk-request': b'attempt=1', 'Content-Length': '2'}>
2024-06-12 00:57:03,491 - MainThread - botocore.httpsession - DEBUG - Certificate path: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/certifi/cacert.pem
2024-06-12 00:57:03,491 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): queue.amazonaws.com:443
2024-06-12 00:57:04,045 - MainThread - urllib3.connectionpool - DEBUG - https://queue.amazonaws.com:443 "POST / HTTP/1.1" 403 132
2024-06-12 00:57:04,045 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '847208d5-fde7-5275-bc9c-563725983489', 'x-amzn-query-error': 'InvalidClientTokenId;Sender', 'Connection': 'close', 'Date': 'Tue, 11 Jun 2024 22:57:03 GMT', 'Content-Type': 'application/x-amz-json-1.0', 'Content-Length': '132'}
2024-06-12 00:57:04,045 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"com.amazon.coral.service#UnrecognizedClientException","message":"The security token included in the request is invalid."}'
2024-06-12 00:57:04,047 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sqs.ListQueues: calling handler <botocore.retryhandler.RetryHandler object at 0x7f397e99ff90>
2024-06-12 00:57:04,047 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2024-06-12 00:57:04,048 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 217, in main
    return command_table[parsed_args.command](remaining, parsed_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 361, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 530, in __call__
    return self._operation_caller.invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 654, in invoke
    self._display_response(operation_name, response, parsed_globals)
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 674, in _display_response
    formatter(command_name, response)
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/formatter.py", line 64, in __call__
    response_data = response.build_full_result()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/paginate.py", line 479, in build_full_result
    for response in self:
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/paginate.py", line 269, in __iter__
    response = self._make_request(current_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/paginate.py", line 357, in _make_request
    return self._method(**current_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the ListQueues operation: The security token included in the request is invalid.
2024-06-12 00:57:04,051 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

An error occurred (InvalidClientTokenId) when calling the ListQueues operation: The security token included in the request is invalid.

Reproduction Steps

export AWS_ENDPOINT_URL=http://localhost:4566
aws --debug sqs list-queues  # <-- doesn't work
aws --debug sns list-topics # <-- works fine

Possible Solution

No response

Additional Information/Context

It seems specific to the CLI. When using boto3 directly, this works fine:

import os

import boto3

def main():
    os.environ["AWS_ENDPOINT_URL"] = "http://localhost:4566"
    sqs = boto3.client("sqs")
    print(sqs.list_queues())

if __name__ == "__main__":
    main()

CLI version used

aws-cli/1.33.6 Python/3.11.5 Linux/5.15.0-107-generic botocore/1.34.124

Environment details (OS name and version, etc.)

Linux 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

tim-finnigan commented 5 months ago

Thanks for reaching out. I could reproduce this issue. The AWS_ENDPOINT_URL environment variable documented here was not used when running SQS commands in version 1.33.6, even though other services like SNS and S3 used that variable. (Also AWS_ENDPOINT_URL_SQS appears to not work when testing on v1, even though AWS_ENDPOINT_URL_SNS did.)

I also tested in v2, and it appears that the environment variable does work as expected for SQS in 2.16.1. I recommend migrating to v2 if you're able to. Otherwise you can also specify --endpoint-url in your command. In the meantime I'll mark this for further review and bring it up with the team.

tim-finnigan commented 5 months ago

Confirmed the issue with the team and created a backlog item for addressing it. A few other services are affected due to the changes that were introduced via https://github.com/aws/aws-cli/pull/7203. In the meantime as mentioned above please use --endpoint-url or migrate to v2 in order to work around this.

thrau commented 5 months ago

thanks for the prompt and clear communication @tim-finnigan :+1:

joe4dev commented 3 months ago

This bug affects the service SQS when using custom profiles in the AWS CLI v1, for both environment variable and parameter configuration:

AWS_PROFILE=localstack aws sqs list-queues  # Fails (custom profile environment variable)
aws --profile localstack sqs list-queues  # Fails (custom profile CLI parameter)
aws --endpoint-url=http://localhost:4566/ sqs list-queues  # Works (using endpoint URL CLI parameter)
aws --profile localstack lambda list-functions  # Works  (using another service than SQS)

# AWS CLI version (latest as of 2024-07-29)
aws --version
aws-cli/1.33.31 Python/3.11.7 Darwin/23.5.0 botocore/1.34.149

Beyond LocalStack, this issue also affects any enterprise reverse proxy configurations that rely on endpoint configuration using custom profiles.