Boto3/botocore v1.35.24 cause credstash lookup to error out because of AWS::Auth::AccountId unimplemented variable

[xtreme-bozhidar-lenchov]

Describe the bug

Boto3/botocore 1.35.24 cause credstash plugin lookups to error out

Expected Behavior

Credstash lookup succeeds

Current Behavior

Our automation relies on being able to fetch secrets from credstash/DynamoDB, and after upgrading to boto3/botocore 1.35.24 released 2:10pm ET Fri Sep 20 2024 our ansible automation fails to complete across fleet of VMs.

"msg": "An unhandled exception occurred while running the lookup plugin 'credstash'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Encountered exception while fetching <credstash secret path>: Unknown builtin variable name: AWS::Auth::AccountId. Encountered exception while fetching <credstash secret path>: Unknown builtin variable name: AWS::Auth::AccountId"

Reproduction Steps

Pin boto3/botocore to latest (1.35.24) and run task with credstash plugin lookup (using latest credstash v1.17.1)

Possible Solution

Pin back to 1.35.23 which does not introduce unimplemented variables: https://github.com/boto/botocore/commit/caa3fde03514a2308a7bfcac05b2bb3fbec6797a#diff-2094bc30bf64ab572d05afda026a85f8dc4e551fac8994c8d72d74855452ea98L453-R455

Additional Information/Context

Had P1 Support case with AWS, no bug filed overnight so filing one in case this is a breaking change

SDK version used

1.35.24

Environment details (OS name and version, etc.)

Ubuntu 22.04, ansible v2.9.13 (but we tested across multiple versions)

[SamRemis]

Hi @xtreme-bozhidar-lenchov,

Can you provide any more information about what commands you are calling that are causing this error and how you're installing botocore? I have been unable to reproduce this without directly modifying the regions.py file to remove the lines you linked above.

Removing those lines causes this error for all dynamodb calls. Including those lines causes the requests succeed without issue.

[xtreme-bozhidar-lenchov]

I guess it's an issue with the credstash implementation/breaking change in botocore? I'll share an ansible snippet later today, but I guess DynamoDB APIs were a backwards compatible change (if this field was added there) but botocore was a breaking change for botocore consumers?

Downgrading to boto3/botocore 1.35.23 worked with our existing credstash version and backing DynamoDB (with no changes there). I am not sure if that was the only line needed to remove the implementation/go back to 1.35.23 state). I wasn't able to find this class mentioned on the web from a Google search, so it caught my eye in the recent commit is all.

[SamRemis]

The lines that you linked to aren't the source of the problem, they're what should be preventing it. Version 1.35.24 released an update to dynamodb's modeled endpoints that introduced two new required parameters which need to be passed in to the endpoint resolver. This error is raised when we encounter an unknown endpoint parameter.

The lines that you linked are a pass-through that tell the SDK to ignore these new parameters when we find them in a service's endpoint model. Since this pass-through was released at the same time as the new parameters, I'm not sure how you are encountering it.

I would expect to see this error if your copy of the SDK has the latest endpoint updates for dynamodb but is missing this pass-through.

[ivan-sam]

I also see the same error after upgrade to 1.35.24 in python lambda that is created from a docker image. We don't use any hacky setup of boto libraries. Looked at the code changes in boto and so far I can't understand how that can happen. But that happens. And downgrade to 1.35.23 has solved the issue. I will try to investigate further.