Boto3 Performance degradation after ugprade from python 3.8 to 3.12

[jricardooliveira]

Describe the bug

We have been experiencing significant performance degradation in our services after upgrading our environment from Python 3.8 with boto3 1.28 (container running python:3.8.16-slim-buster) to Python 3.12 with any version of boto3 greater than or equal to 1.28 (container running python:3.12.2-slim-bookworm).

Expected Behavior

The performance should be consistent across requests, similar to the behavior observed in the Python 3.8 environment.

Current Behavior

Reproduction Steps

1- Set up a container with Python 3.8.16-slim-buster and boto3 1.28. 2 - Set up a container with Python 3.12.2-slim-bookworm and boto3 1.28 or higher. 3 - Send multiple burst requests from a single user to a service using boto3.

In the screenshot you may see the performance degradation before and after and the culprit... boto3 kinesis POST request.

Possible Solution

No response

Additional Information/Context

We suspect that the connection initialization process in the newer environment might be causing this issue. Any insights or suggestions on how to address this performance degradation would be greatly appreciated.

SDK version used

1.28.x 1.34.x

Environment details (OS name and version, etc.)

Container Debian slim-bookworm

[tim-finnigan]

Thanks for reaching out. We need a bit more information here in order to investigate this further. Can you provide a minimal code snippet to reproduce the issue? It's not clear what your code involves and what might be affecting the performance.

Also could you share debug logs, which you can get by adding boto3.set_stream_logger('') to your script? That may help with understanding what is going on here. And per the CHANGELOG the latest Boto3 version is 1.34.146, so please try updating to test this in the latest (or a recent) version.

[jricardooliveira]

Hi,

We found out that the issue is probably related with OpenSSL 3.x vs OpenSSL 1.x ... but we aren't sure. What we're sure is that using Debian or Ubuntu that comes with OpenSSL 1.x is fast and the problem arises on Debian 12 based distros.

Are you aware of issues related to OpenSSL?

Thanks Joao

[tim-finnigan]

Thanks for following up - I'm not aware of issues involving OpenSSL related to performance degradation. But if you're able to provide logs/memory profiles comparing both then that could with further investigation.

[nateprewitt]

There is an as-of-yet unresolved performance decrease in Python 3.12+ with OpenSSL 3.x because of how OpenSSL 3 does certificate path lookups (https://github.com/python/cpython/issues/95031). This has lead to up to 5x decreases in performance in some instances depending on the number of certs and locations searched on the host.

We've done some initial investigations to help minimize this but it's still an unresolved issue for most of Python's networking infrastructure. I'm a bit surprised you're seeing this on Debian/Ubuntu as most reports have been more significant on Windows (up to 55x reported performance decrease). We have yet to see information on a safe solution for addressing this. Using a single SSLContext has unfortunately proven to be problematic in Requests and other projects that have tried addressing this.