search

boto / botocore

The low-level, core functionality of boto3 and the AWS CLI.
Apache License 2.0
1.51k stars 1.09k forks source link

Boto3/botocore v1.35.24 cause credstash lookup to error out because of AWS::Auth::AccountId unimplemented variable #3263

Open xtreme-bozhidar-lenchov opened 2 months ago

xtreme-bozhidar-lenchov commented 2 months ago

Describe the bug

Boto3/botocore 1.35.24 cause credstash plugin lookups to error out

Expected Behavior

Credstash lookup succeeds

Current Behavior

Our automation relies on being able to fetch secrets from credstash/DynamoDB, and after upgrading to boto3/botocore 1.35.24 released 2:10pm ET Fri Sep 20 2024 our ansible automation fails to complete across fleet of VMs.

"msg": "An unhandled exception occurred while running the lookup plugin 'credstash'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Encountered exception while fetching <credstash secret path>: Unknown builtin variable name: AWS::Auth::AccountId. Encountered exception while fetching <credstash secret path>: Unknown builtin variable name: AWS::Auth::AccountId"

Reproduction Steps

Pin boto3/botocore to latest (1.35.24) and run task with credstash plugin lookup (using latest credstash v1.17.1)

Possible Solution

Pin back to 1.35.23 which does not introduce unimplemented variables: https://github.com/boto/botocore/commit/caa3fde03514a2308a7bfcac05b2bb3fbec6797a#diff-2094bc30bf64ab572d05afda026a85f8dc4e551fac8994c8d72d74855452ea98L453-R455

Additional Information/Context

Had P1 Support case with AWS, no bug filed overnight so filing one in case this is a breaking change

SDK version used

1.35.24

Environment details (OS name and version, etc.)

Ubuntu 22.04, ansible v2.9.13 (but we tested across multiple versions)

SamRemis commented 2 months ago

Hi @xtreme-bozhidar-lenchov,

Can you provide any more information about what commands you are calling that are causing this error and how you're installing botocore? I have been unable to reproduce this without directly modifying the regions.py file to remove the lines you linked above.

Removing those lines causes this error for all dynamodb calls. Including those lines causes the requests succeed without issue.

xtreme-bozhidar-lenchov commented 2 months ago

I guess it's an issue with the credstash implementation/breaking change in botocore? I'll share an ansible snippet later today, but I guess DynamoDB APIs were a backwards compatible change (if this field was added there) but botocore was a breaking change for botocore consumers?

Downgrading to boto3/botocore 1.35.23 worked with our existing credstash version and backing DynamoDB (with no changes there). I am not sure if that was the only line needed to remove the implementation/go back to 1.35.23 state). I wasn't able to find this class mentioned on the web from a Google search, so it caught my eye in the recent commit is all.

SamRemis commented 2 months ago

The lines that you linked to aren't the source of the problem, they're what should be preventing it. Version 1.35.24 released an update to dynamodb's modeled endpoints that introduced two new required parameters which need to be passed in to the endpoint resolver. This error is raised when we encounter an unknown endpoint parameter.

The lines that you linked are a pass-through that tell the SDK to ignore these new parameters when we find them in a service's endpoint model. Since this pass-through was released at the same time as the new parameters, I'm not sure how you are encountering it.

I would expect to see this error if your copy of the SDK has the latest endpoint updates for dynamodb but is missing this pass-through.

ivan-sam commented 2 months ago

I also see the same error after upgrade to 1.35.24 in python lambda that is created from a docker image. We don't use any hacky setup of boto libraries. Looked at the code changes in boto and so far I can't understand how that can happen. But that happens. And downgrade to 1.35.23 has solved the issue. I will try to investigate further.