Botocore 1.35.15 breaks the ELBv2 waiter 'load_balancers_deleted'

[bblommers]

Describe the bug

With botocore 1.15.35, the following code fails:

waiter = conn.get_waiter("load_balancers_deleted")
waiter.wait(LoadBalancerArns=[load_balancer_arn])

Expected Behavior

The same test passes against botocore==1.35.14, so I would expect this to continue working.

Current Behavior

    def wait(self, **kwargs):
        acceptors = list(self.config.acceptors)
        current_state = 'waiting'
        # pop the invocation specific config
        config = kwargs.pop('WaiterConfig', {})
        sleep_amount = config.get('Delay', self.config.delay)
        max_attempts = config.get('MaxAttempts', self.config.max_attempts)
        last_matched_acceptor = None
        num_attempts = 0

        while True:
            response = self._operation_method(**kwargs)
            num_attempts += 1
            for acceptor in acceptors:
                if acceptor.matcher_func(response):
                    last_matched_acceptor = acceptor
                    current_state = acceptor.state
                    break
            else:
                # If none of the acceptors matched, we should
                # transition to the failure state if an error
                # response was received.
                if is_valid_waiter_error(response):
                    # Transition to a failure state, which we
                    # can just handle here by raising an exception.
>                   raise WaiterError(
                        name=self.name,
                        reason='An error occurred ({}): {}'.format(
                            response['Error'].get('Code', 'Unknown'),
                            response['Error'].get('Message', 'Unknown'),
                        ),
                        last_response=response,
                    )
E                   botocore.exceptions.WaiterError: Waiter LoadBalancersDeleted failed: An error occurred (LoadBalancerNotFound): One or more load balancers not found

Reproduction Steps

Self-contained test:

def test_describe_unknown_load_balancers():
    sts = boto3.client("sts", "us-east-1")
    account_id = sts.get_caller_identity()["Account"]

    load_balancer_arn = f"arn:aws:elasticloadbalancing:us-east-1:{account_id}:loadbalancer/app/mylb/50dc6c495c0c9188"

    conn = boto3.client("elbv2", "us-east-1")
    with pytest.raises(ClientError) as exc:
        conn.describe_load_balancers(LoadBalancerArns=[load_balancer_arn])
    err = exc.value.response["Error"]
    assert err["Code"] == "LoadBalancerNotFound"

    waiter = conn.get_waiter("load_balancers_deleted")
    waiter.wait(LoadBalancerArns=[load_balancer_arn])

Note that this reproduces the problem with an unknown ARN, but it is the same for a LoadBalancer that used to exist, but has just been deleted.

Possible Solution

This commit changes the expected error code, but (as can be seen in the attached test), the error code that AWS sends is not been changed.

https://github.com/boto/botocore/commit/d6d8e1a5db7234625a9f16d48e1f51c232666846#diff-898e5d7a1a88b482c13dfb8a9a0ded7f577101ff9f256427f9e24bcc82fa8f63

Additional Information/Context

No response

SDK version used

1.35.15

Environment details (OS name and version, etc.)

Ubuntu

[nateprewitt]

Thanks for the report @bblommers, could you clarify if the returned exeception is coming from the live ELBv2 service or is this being generated from moto (or another 3p implementation)? I'm taking a look into the issue but there seems to data suggesting the AWS service was not previously returning LoadBalancerNotFound.

[bblommers]

Hi @nateprewitt, that is coming from the live ELBv2 service.

I did indeed first notice it while using Moto, but was able to reproduce it against AWS.

[nateprewitt]

Ok, thanks for the quick response. I'm taking a look at what's happening with the service, this change went out to all AWS SDKs on Monday. I'll follow up as soon as we have more information.

[nateprewitt]

Hi @bblommers, wanted to follow up that this should be fixed in https://github.com/boto/botocore/commit/828be0de750a2ec8fbd429b55380953b81d836b1#diff-898e5d7a1a88b482c13dfb8a9a0ded7f577101ff9f256427f9e24bcc82fa8f63 with today's release of 1.35.18.

There was some miscommunication between the service team and the Javascript SDK that led to the regression. We should be putting in place better controls to handle this going forward. Thanks for your patience!

I'll close this out tomorrow if we don't hear back on any further issues you're seeing.

[bblommers]

Can confirm this is fixed in 1.35.18. Thank you for the quick resolution @nateprewitt!

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.