Creating AWS4Auth instances with STS tokens leaks memory, slows down over time

[miikka]

If you continuously create AWS4Auth instances in with the security_token argument set, it will slowly leak memory and make request signing slower.

Our production service creates a new AWS4Auth instance for every request to AWS S3 (possibly we should just re-use them) and we noticed that after tens of thousands of requests, the requests were getting slower and slower. Restarting the service makes them fast again. Looks like the code below is causing the issue:

https://github.com/Colin-b/httpx_auth/blob/e3bd73981be62a32ce8dd4c2cf00e7e62cbfb9d8/httpx_auth/aws.py#L50-L53

Every time you create a new AWS4Auth instance, one more copy of x-amz-security-token gets appended to default_include_headers. Here's a Python REPL example demonstrating the problem:

>>> from httpx_auth.aws import AWS4Auth
>>> AWS4Auth("test", "test", "us-east-1", "s3", security_token="token").default_include_headers
['host', 'content-type', 'date', 'x-amz-*', 'x-amz-security-token']
>>> AWS4Auth("test", "test", "us-east-1", "s3", security_token="token").default_include_headers
['host', 'content-type', 'date', 'x-amz-*', 'x-amz-security-token', 'x-amz-security-token']
>>> AWS4Auth("test", "test", "us-east-1", "s3", security_token="token").default_include_headers
['host', 'content-type', 'date', 'x-amz-*', 'x-amz-security-token', 'x-amz-security-token', 'x-amz-security-token']

[Colin-b]

Thanks for reporting this issue. As you pointed out, the auth instances should be reused as much as possible if you do not change any parameter to it. In any case this is an issue that will be tackled of course !

[Colin-b]

The fix is now on the develop branch and will be part of the next httpx-auth release.