search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.94k stars 1.86k forks source link

elbv2 describe_rules for GWLB returns ListenerNotFound error #3591

Closed daftkid closed 2 weeks ago

daftkid commented 1 year ago

Describe the bug

I tried to find all of the rules for a listener of the GWLB (Please the test code in Reproduction Steps) and it failed on describe_rules with the error saying that the specified Listener is not found. But it actually exists! As I can describe it via describe_listeners and the same listener ARN passed as a parameter.

awscli for elbv2 describe-listeners --listener-arns <my_listener_arn> works fine.

Expected Behavior

I'm able to get the listener rules for the GWLB listener by specifying listener's ARN.

Current Behavior

It's possible to describe the listener by its ARN without any issues however it's not possible to get the rules for this Listener ARN because Listener is not found in this case.

Traceback (most recent call last):
  File "/Users/xxxxx/github/xxxxxxx/test.py", line 5, in <module>
    res = client.describe_rules(
  File "/Users/xxxxxx/Library/Python/3.9/lib/python/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/xxxxxxx/Library/Python/3.9/lib/python/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ListenerNotFoundException: An error occurred (ListenerNotFound) when calling the DescribeRules operation: One or more listeners not found

Reproduction Steps

Run the following code:

import boto3

client = boto3.client("elbv2", region_name="cn-northwest-1")

res = client.describe_listeners(
        ListenerArns=["<my_existing_arn_copied_from_aws_console>"]
        )

print(res)

res = client.describe_rules(
        ListenerArn="<my_existing_arn_copied_from_aws_console>"
        )

print(res)

Possible Solution

No response

Additional Information/Context

I tried to run this code within the Chinese region, I'm not able to test for global regions.

boto3                    1.26.34
botocore                 1.29.34

SDK version used

1.26.34

Environment details (OS name and version, etc.)

MacOS Ventura 13.2

aBurmeseDev commented 1 year ago

Hi @daftkid - thanks for reaching out and sorry to hear that you're having an issue.

I wasn't able to reproduce it on my end in us-west-2, however I find it strange that listener shows up when you called describe_listeners.

response = client.describe_rules( RuleArns=[ '', ] ) print(response)

- Could you also try running `describe_rules` CLI command like this?

aws elbv2 describe-rules --listener-arn arn:aws:elasticloadbalancing:us-west-2:123456789012:listener/app/my-load-balancer/50dc6c495c0c9188/f2f7dc8efc522ab2



If issue persists, please share your debug logs by adding `boto3.set_stream_logger('')` to your code (without any sensitive info).
Best,
John
daftkid commented 1 year ago

Hi @aBurmeseDev . Please note that it works fine for all ALB and NLB LBs but it DOES NOT work for Gateway LB. (Please also note that I'm running it in the Chinese regions so there might be some differences in how AWS API works there, not sure about that). I suppose it might be related to the fact that GWLB is a kinda different one and you can only configure a listener for GENEVE protocol on port 6081. I cannot test boto3 describe_rules with a RuleArns filter as I cannot find these Rule ARNs, please see below.

Please see the debug output for the awscli command below (TLDR - It also failed with the same error):

aws elbv2 describe-rules --listener-arn arn:aws-cn:elasticloadbalancing:cn-northwest-1:xxxxxxxxx:listener/gwy/xxxxxxx/xxxxxx/xxxxx --debug

2023-02-08 10:30:15,298 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): elasticloadbalancing.cn-northwest-1.amazonaws.com.cn:443
2023-02-08 10:30:16,281 - MainThread - urllib3.connectionpool - DEBUG - https://elasticloadbalancing.cn-northwest-1.amazonaws.com.cn:443 "POST / HTTP/1.1" 400 295
2023-02-08 10:30:16,282 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'xxxxxxxxxx', 'Content-Type': 'text/xml', 'Content-Length': '295', 'Date': 'Wed, 08 Feb 2023 08:30:15 GMT', 'Connection': 'close'}
2023-02-08 10:30:16,282 - MainThread - botocore.parsers - DEBUG - Response body:
b'<ErrorResponse xmlns="http://elasticloadbalancing.amazonaws.com/doc/2015-12-01/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>ListenerNotFound</Code>\n    <Message>One or more listeners not found</Message>\n  </Error>\n  <RequestId>xxxxxxxxx</RequestId>\n</ErrorResponse>\n'
2023-02-08 10:30:16,283 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'xxxxx', 'Content-Type': 'text/xml', 'Content-Length': '295', 'Date': 'Wed, 08 Feb 2023 08:30:15 GMT', 'Connection': 'close'}
2023-02-08 10:30:16,283 - MainThread - botocore.parsers - DEBUG - Response body:
b'<ErrorResponse xmlns="http://elasticloadbalancing.amazonaws.com/doc/2015-12-01/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>ListenerNotFound</Code>\n    <Message>One or more listeners not found</Message>\n  </Error>\n  <RequestId>xxxxxxxx</RequestId>\n</ErrorResponse>\n'
2023-02-08 10:30:16,283 - MainThread - botocore.hooks - DEBUG - Event needs-retry.elastic-load-balancing-v2.DescribeRules: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x1115afe80>>
2023-02-08 10:30:16,284 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-02-08 10:30:16,284 - MainThread - botocore.hooks - DEBUG - Event after-call.elastic-load-balancing-v2.DescribeRules: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x1115afa30>>
2023-02-08 10:30:16,284 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/clidriver.py", line 459, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/clidriver.py", line 594, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/clidriver.py", line 770, in __call__
    return self._operation_caller.invoke(
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/clidriver.py", line 903, in invoke
    self._display_response(operation_name, response, parsed_globals)
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/clidriver.py", line 925, in _display_response
    formatter(command_name, response, stream)
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/formatter.py", line 82, in __call__
    response_data = response.build_full_result()
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/botocore/paginate.py", line 446, in build_full_result
    for response in self:
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/botocore/paginate.py", line 252, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/botocore/paginate.py", line 329, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/botocore/client.py", line 304, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/awscli/2.7.2/libexec/lib/python3.9/site-packages/awscli/botocore/client.py", line 634, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ListenerNotFoundException: An error occurred (ListenerNotFound) when calling the DescribeRules operation: One or more listeners not found

An error occurred (ListenerNotFound) when calling the DescribeRules operation: One or more listeners not found

For me, it seems as AWS API is not able to return a list of rules for the GWLB because you are not able to create more than one rule. So I suppose it should be somehow reflected in the boto3.

Please let me know if you need any further testing or information.

aBurmeseDev commented 1 year ago

@daftkid - thanks for providing these additional information. As you mentioned, the region you're in cn-northwest-1 is probably the play here and from what I've found, there are some caveats under CN regions. I'll have to check and test under one of those regions and will follow up. Thank you for your patience.

daftkid commented 1 year ago

@aBurmeseDev Hello, I was able to reproduce it for the ap-southeast-2 region too. Please see boto output below:

2023-02-15 11:01:00,295 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'ap-southeast-2', 'UseDualStack': False, 'UseFIPS': False}
2023-02-15 11:01:00,296 botocore.regions [DEBUG] Endpoint provider result: https://elasticloadbalancing.ap-southeast-2.amazonaws.com
2023-02-15 11:01:00,296 botocore.hooks [DEBUG] Event before-parameter-build.elastic-load-balancing-v2.DescribeRules: calling handler <function generate_idempotent_uuid at 0x10bdd79d0>
2023-02-15 11:01:00,296 botocore.hooks [DEBUG] Event before-call.elastic-load-balancing-v2.DescribeRules: calling handler <function add_recursion_detection_header at 0x10bdd7670>
2023-02-15 11:01:00,297 botocore.hooks [DEBUG] Event before-call.elastic-load-balancing-v2.DescribeRules: calling handler <function inject_api_version_header_if_needed at 0x10bddd280>
2023-02-15 11:01:00,297 botocore.endpoint [DEBUG] Making request for OperationModel(name=DescribeRules) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.26.34 Python/3.9.12 Darwin/22.3.0 Botocore/1.29.34'}, 'body': {'Action': 'DescribeRules', 'Version': '2015-12-01', 'ListenerArn': 'arn:aws:elasticloadbalancing:ap-southeast-2:xxxxxxxxxx:listener/gwy/xxxxxxxxxx/xxxx/xxxx'}, 'url': 'https://elasticloadbalancing.ap-southeast-2.amazonaws.com/', 'context': {'client_region': 'ap-southeast-2', 'client_config': <botocore.config.Config object at 0x10c577580>, 'has_streaming_input': False, 'auth_type': None}}
2023-02-15 11:01:00,297 botocore.hooks [DEBUG] Event request-created.elastic-load-balancing-v2.DescribeRules: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x10c577460>>
2023-02-15 11:01:00,298 botocore.hooks [DEBUG] Event choose-signer.elastic-load-balancing-v2.DescribeRules: calling handler <function set_operation_specific_signer at 0x10bdd78b0>
2023-02-15 11:01:00,298 botocore.auth [DEBUG] Calculating signature using v4 auth.
2023-02-15 11:01:00,298 botocore.auth [DEBUG] CanonicalRequest:
....
2023-02-15 11:01:00,300 botocore.httpsession [DEBUG] Certificate path: /usr/local/lib/python3.9/site-packages/certifi/cacert.pem
2023-02-15 11:01:00,617 urllib3.connectionpool [DEBUG] https://elasticloadbalancing.ap-southeast-2.amazonaws.com:443 "POST / HTTP/1.1" 400 295
2023-02-15 11:01:00,619 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'xxxxx', 'Content-Type': 'text/xml', 'Content-Length': '295', 'Date': 'Wed, 15 Feb 2023 09:00:59 GMT', 'Connection': 'close'}
2023-02-15 11:01:00,619 botocore.parsers [DEBUG] Response body:
b'<ErrorResponse xmlns="http://elasticloadbalancing.amazonaws.com/doc/2015-12-01/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>ListenerNotFound</Code>\n    <Message>One or more listeners not found</Message>\n  </Error>\n  <RequestId>xxxxxx</RequestId>\n</ErrorResponse>\n'
2023-02-15 11:01:00,622 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'xxxxx, 'Content-Type': 'text/xml', 'Content-Length': '295', 'Date': 'Wed, 15 Feb 2023 09:00:59 GMT', 'Connection': 'close'}
2023-02-15 11:01:00,622 botocore.parsers [DEBUG] Response body:
b'<ErrorResponse xmlns="http://elasticloadbalancing.amazonaws.com/doc/2015-12-01/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>ListenerNotFound</Code>\n    <Message>One or more listeners not found</Message>\n  </Error>\n  <RequestId>xxxxx</RequestId>\n</ErrorResponse>\n'
2023-02-15 11:01:00,622 botocore.hooks [DEBUG] Event needs-retry.elastic-load-balancing-v2.DescribeRules: calling handler <botocore.retryhandler.RetryHandler object at 0x10c59e8e0>
2023-02-15 11:01:00,622 botocore.retryhandler [DEBUG] No retry needed.
Traceback (most recent call last):
  File "/Users/xxxxx/Documents/work/xxxxxx/test.py", line 11, in <module>
    res = client.describe_rules(
  File "/Users/xxx/Library/Python/3.9/lib/python/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/xxxx/Library/Python/3.9/lib/python/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ListenerNotFoundException: An error occurred (ListenerNotFound) when calling the DescribeRules operation: One or more listeners not found

The same for awscli:

> aws elbv2 describe-rules --listener-arn arn:aws:elasticloadbalancing:ap-southeast-2:xxxxx:listener/gwy/xxxx/xxx/xxx --region ap-southeast-2

An error occurred (ListenerNotFound) when calling the DescribeRules operation: One or more listeners not found
daftkid commented 1 year ago

@aBurmeseDev In my understanding, this issue is being reproduced for the Gateway LoadBalancer only.

aBurmeseDev commented 1 year ago

@daftkid - thanks for your patience and confirming that it's not region specific. However, I suspect that it might be service related due to the fact that it was reproducible with both CLI and boto3. Unfortunately, I'm still not able to reproduce it on my end. Could you double check a few things here:

If you'd like, I could try reaching out to service team for this specific issue or you could reach out to AWS Support where they could look at your requestID to troubleshoot.

daftkid commented 1 year ago

@aBurmeseDev Hi, I confirm that target GWLB is in the ap-southeast-2 region and I have the region set to ap-southeast-2 in the ~/.aws/config file.

For the second one, I can see the following info for the GWLB resource:

IP listener
A Gateway Load Balancer consists of an IP listener that receives all connection requests and routes them to the target group you specify.

Listener ARN
arn:aws:elasticloadbalancing:ap-southeast-2:xxx:listener/gwy/xxxxxxxx/55b083a88f927021/3814037877268e76

Protocol:Port
[GENEVE:6081]

Forwarding to target group
xxxx-xxx-xxxx

Also, please see the output of awscli for the describe-listeners:

🌱 > aws elbv2 describe-listeners --load-balancer-arn arn:aws:elasticloadbalancing:ap-southeast-2:xxxxxx:loadbalancer/gwy/xxxxxx/55b083a88f927021
{
    "Listeners": [
        {
            "ListenerArn": "arn:aws:elasticloadbalancing:ap-southeast-2:xxxxxxxx:listener/gwy/xxxxxxxxxxx/55b083a88f927021/3814037877268e76",
            "LoadBalancerArn": "arn:aws:elasticloadbalancing:ap-southeast-2:xxxxxxxxx:loadbalancer/gwy/xxxxxxxxx/55b083a88f927021",
            "DefaultActions": [
                {
                    "Type": "forward",
                    "TargetGroupArn": "arn:aws:elasticloadbalancing:ap-southeast-2:xxxxxxx:targetgroup/xxxxxxxx/0070c3e5cdd68269f7",
                    "ForwardConfig": {
                        "TargetGroups": [
                            {
                                "TargetGroupArn": "arn:aws:elasticloadbalancing:ap-southeast-2:xxxxxxx:targetgroup/xxxxxxx/0070c3e5cdd68269f7"
                            }
                        ]
                    }
                }
            ]
        }
    ]
}

Moreover, based on this doc https://docs.aws.amazon.com/elasticloadbalancing/latest/gateway/gateway-listeners.html I suppose that GWLB supports only one rule and that may be the reason why rules are not returned in boto and awscli.

You said you cannot reproduce this issue - may I ask you to confirm that you are able to list rules for the Gateway Load Balancer via specifying listener-arn?

Also, I'd appreciate it if you can talk to the service team as I cannot raise a support ticket because of a low support plan.

aBurmeseDev commented 1 month ago

Apologies this fell off my radar. I'm not sure if you're still working on this but I'd like to address that I was able to list_rules for Gateway Load Balancer by specifying listener-arn. As previously mentioned, the error may be due to

  1. Incorrect Listener ARN: Double-check the listener ARN you're using and ensure that it matches the actual listener associated with your GWLB.
  2. Listener Deletion: The listener associated with the rules you're trying to describe has been deleted, but the rules still exist. In this case, the AWS CLI cannot find the listener and returns the ListenerNotFound error.
  3. GWLB Resource Ownership: Ensure that you're using the correct AWS credentials and have the necessary permissions to access the GWLB resources.

Hope it helps! John