[Package Update Request] - OpenSSL 3.2.x

[rowanbeentje]

What package is missing from Amazon Linux 2023? Please describe and include package name.

openssl 3.2+ (or openssl 3.1+)

Is this an update to existing package or new package request?

This is an update to the existing openssl 3.0.8.

Is this package available in Amazon Linux 2? If it is available via external sources such as EPEL, please specify.

No

Any additional information you'd like to include. (use-cases, etc)

You might want to merge this in as a duplicate of https://github.com/amazonlinux/amazon-linux-2023/issues/512 , but I wanted to raise this for 3.1+ in particular, for a particular use-case: a significant performance regression compared to Amazon Linux 2.

We recently upgraded some servers from AL2 to AL2023 so that we could upgrade to PHP 8.2 on elastic beanstalk. During testing we found that some of our servers which perform lots of network requests to other servers over an HTTPS REST API were performing way worse - something of the order of a four or five times or more increase in CPU load, it seems. We had to scale some t3.medium servers with lots of headroom up to t3.2xlarge, and others even further - which can add up!

I managed to mostly reproduce this on base AL2 and AL2023 servers, and using a simple curl command: curl -w "tcp_established: %{time_connect}, ssl_handshake_done: %{time_appconnect}, TTFB: %{time_starttransfer}\n" -o /dev/null -s --head "https://aws.amazon.com/". On a t2.micro, AL2 shows the SSL stage taking ~9-10ms, and on AL2023 it takes ~37-40ms. This comes with an accompanying increase in CPU usage; I assume that other SSL tasks also vary, as our PHP servers seem to show a larger hit than this (and initial SSL negotiation would only be a tiny part of their tasks!).

Long story short, I tried various things like disabling SELinux and ensuring TLS1.2, but in the course of a still-in-progress support case (170690787201370) a helpful Linux Team member pointed out the OpenSSL version differences. This led me to the epic https://github.com/openssl/openssl/issues/17064 which is still open but does show that some work has happened in OpenSSL 3.1 and 3.2 to address some of the regressions.

I tried building OpenSSL from source on a clean AL2023 box and ended up with these numbers:

• AL2 (out-of-the-box OpenSSL 1.0.2k-fips): ~9-10ms SSL negotiation
• AL2023 (out-of-the-box OpenSSL 3.0.8): ~37-40ms SSL negotiation
• AL2023 (OpenSSL 3.0.8, build with default make): ~38-42ms SSL negotiation
• AL2023 (OpenSSL 3.2.1, build with default make): ~12-13ms SSL negotiation

So still an overhead compared to AL2, but OpenSSL 3.2 improves the performance regression significantly.

(I'm sort of surprised I haven't seen more about this when trying to work out why our AL2023 instances were performing so much worse!)

[rowanbeentje]

I saw the comment on https://github.com/amazonlinux/amazon-linux-2023/issues/512#issuecomment-1757791113 indicating that there are likely complex reasons openssl generally doesn't get updated - but maybe an alternate version package could be offered, like some of the older versions left available on AL2? (I see FIPS may be one of those reasons but the openssl source page mentions 3.2 can be built with the 3.0.8 FIPS provider...)

[rowanbeentje]

Wonder if this is related at all to https://github.com/amazonlinux/amazon-linux-2023/issues/434 🤔

[stewartsmith]

There's FIPS implications for bumping OpenSSL, as well as various compatibility things to ensure we address. It does look like OpenSSL is making this kind of thing more feasible, but it's still a big step for us to evaluate the possibility of any update.

Just setting expectations that this is less likely than other package update requests we have.

[rowanbeentje]

Understood 😬 Is there a chance of offering more recent versions as an alternate package, or does that not really align with FIPS compliance, or be too much of a pain to maintain?

[ccassidy-maxar]

Ran into a similar issue when I bumped a Lambda function from Python 3.11 -> 3.12 (and therefore AL2/OpenSSL 1.x -> AL2023/OpenSSL 3.0.x). A simple Lambda that posted an SNS message took 2x longer with 3.12. When I build a custom layer with OpenSSL 3.2.x on Python 3.12 it brought it back in-line with performance on 3.11.

[krk]

FIPS provider module (fips.so) can be used from the FIPS-certified 3.0.x version, in a 3.1/3.2/3.3 series.

This enables e.g. OpenSSL 3.3.1 to support FIPS by loading fips.so compiled from 3.0.x.