search

boto / botocore

The low-level, core functionality of boto3 and the AWS CLI.
Apache License 2.0
1.44k stars 1.06k forks source link

RecursionError through botocore/endpoint_provider.py:572 #3184

Closed ilg closed 1 week ago

ilg commented 1 month ago

Describe the bug

Unfortunately, I don't have much in the way of detail nor consistent repro steps, but the stack trace seems to point at botocore. This most likely happened when AWS Secrets Manager rotated a credential, leading to my code's call to create a secrets manager client to get the new secret, and that led into RecursionError: maximum recursion depth exceeded in comparison (though the code works fine most of the time). This is running inside a docker container on an EC2 instance, all managed with Elastic Beanstalk.

Expected Behavior

No infinite recursion

Current Behavior

[…]
  File "/code/[my file].py", line [##], in [function name]
    client = boto3.client("secretsmanager", "us-west-2")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/boto3/__init__.py", line 92, in client
    return _get_default_session().client(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/boto3/session.py", line 299, in client
    return self._session.create_client(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 997, in create_client
    client = client_creator.create_client(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 161, in create_client
    client_args = self._get_client_args(
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 520, in _get_client_args
    return args_creator.get_client_args(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/args.py", line 156, in get_client_args
    ruleset_resolver = self._build_endpoint_resolver(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/args.py", line 660, in _build_endpoint_resolver
    return EndpointRulesetResolver(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/regions.py", line 469, in __init__
    self._provider = EndpointProvider(
                     ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 704, in __init__
    self.ruleset = RuleSet(**ruleset_data, partitions=partition_data)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 658, in __init__
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 658, in <listcomp>
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
    return rule_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in __init__
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in <listcomp>
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
    return rule_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in __init__
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in <listcomp>
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
    return rule_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in __init__
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in <listcomp>
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
    return rule_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in __init__
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in <listcomp>
    self.rules = [RuleCreator.create(**rule) for rule in rules]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
    return rule_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 445, in __init__
    super().__init__(**kwargs)
RecursionError: maximum recursion depth exceeded in comparison

Reproduction Steps

boto3.client("secretsmanager", "us-west-2") seems pretty self-contained, but works fine the vast majority of the time, so I'm not sure what happened here.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

boto3==1.34.100

Environment details (OS name and version, etc.)

Linux [hostname] 5.10.214-202.855.amzn2.x86_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

ilg commented 3 weeks ago

FWIW, this does seem to occur every time Secrets Manager rotates the credentials, but only within the already-running web application server. As soon as the worker thread/process restarts, the boto3 client creation call works fine. Given this, I should be able to debug or test further by manually triggering credential rotation.

tim-finnigan commented 3 weeks ago

Thanks for reaching out. When you're able to, please provide a code snippet for reproducing the issue, as well as debug logs (with sensitive info redacted) by adding boto3.set_stream_logger('') to your script.

github-actions[bot] commented 1 week ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.