OpenSSL legacy provider isn't always installed on Debian unstable

[cjwatson]

This is an attempt to get a conversation started. I wasn't responsible for the original change here, but I've been cleaning up after it in Debian's Python team and I'm not convinced my current approach is really ideal.

The OpenSSL packages in Debian unstable recently changed to move the legacy provider into a separate package, openssl-provider-legacy. There's a Recommends relationship from the main library package (libssl3t64) which means that it's installed on typical user systems, but it's not a Depends so it's now possible for users not to have the legacy provider installed. In particular, Recommends are not normally installed in package builds and automatic test environments (deliberately, to ensure that declared dependencies are sufficient), and as a result we find ourselves having to apply workarounds to a fairly large number of Python packages in Debian to get their tests working again.

I'd like to find out what approach you'd recommend here. I can think of a bunch of possibilities:

• Just set CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1 in the Debian build rules for lots of Python packages (this is pretty much what I'm doing at the moment, but it's repetitive)
• Make affected Python packages in Debian depend/build-depend on openssl-provider-legacy (also repetitive and feels heavy-handed)
• Make Debian's python3-cryptography package depend on openssl-provider-legacy
• Effectively revert the Debian OpenSSL packaging change (perhaps just by upgrading the Recommends to Depends)
• Change the logic in cryptography/src/rust/src/lib.rs in some way
• Something else I haven't thought of

Do you have any opinions on this? Ideally I wouldn't be passing messages back and forward, so I'll see if I can get Debian's OpenSSL maintainers to show up on this issue.

[alex]

I guess as a starting point: What would your desired behavior be :-)

I'm guessing it's something like: if openssl-provider-legacy is installed, use it, if it's not, ignore it. Never bother users about this.

My suggestion would probably be to just change cryptography to silently allow the legacy provider to fail to load. This would look something like (untested):

diff --git a/src/rust/src/lib.rs b/src/rust/src/lib.rs
index cd7b99f15..dce24b100 100644
--- a/src/rust/src/lib.rs
+++ b/src/rust/src/lib.rs
@@ -60,9 +60,7 @@ fn _initialize_providers() -> CryptographyResult<LoadedProviders> {
         .map(|v| v.is_empty() || v == "0")
         .unwrap_or(true);
     let legacy = if load_legacy {
-        let legacy_result = provider::Provider::load(None, "legacy");
-        _legacy_provider_error(legacy_result.is_ok())?;
-        Some(legacy_result?)
+        provider::Provider::load(None, "legacy").ok()
     } else {
         None
     };

[alex]

It occurs to me that this will also impact our non-wheel users. If distros are going to be switching legacy to not be installed by default, I guess we'll probably also need to handle legacy failing to load silently.

[stefanor]

Yes, graceful failure sounds like the best possible option here.

[cjwatson]

I'm not completely sure what my desired behaviour would be! It does include not having cryptography be weirdly different on Debian than elsewhere, though I appreciate that's a bit vague.

Silently allowing the legacy provider to fail to load is probably OK, although it does mean that users of cryptography-based applications who don't have openssl-provider-legacy installed may have an interesting debugging experience. A thought: would it make sense for it to still raise a non-fatal warning of some kind, unless CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1 is set? That may still have some downstream test suite fallout due to test suites being picky about what shows up on stderr and such, but the fallout would probably be less extensive and easier to deal with.

[stefanor]

I assume that the overwealming majority of cryptography's use will not need the legacy provider, and wouldn't want to be showing warnings. It's going to be things like exhaustive test suites that need it.

[alex]

I would not assume that. AFAIK legacy provider is needed for several algorithms that are widely used in PEM and PKCS#12 encryption. (These algorithms are garbage are people shouldn't use them, but here we are.)

FWIW the original motivation for making it noisy for the legacy provider to fail to load was concern that users would basically bork their installs and it'd be impossible for them to debug.

Do you know if any other distros are planning to put the legacy provider in a separate package?

[cjwatson]

Do you know if any other distros are planning to put the legacy provider in a separate package?

I'm afraid I have no idea. If any of the Debian OpenSSL folks show up here then maybe they will.

[reaperhulk]

Without the legacy provider RC2 PKCS12 encryption (which is, sadly, incredibly common) will fail with an opaque error: https://github.com/pyca/cryptography/blob/45f0c8d274d3f2d6cbefdd8bebfb568cf16efbf7/src/rust/src/pkcs12.rs#L719-L721

As Alex noted there are also other scenarios where this is likely to happen. I'm generally supportive of silently not loading the legacy provider, but I think we need to make the error messages where these failures can occur actionable. This probably requires error stack parsing though, and error stacks aren't guaranteed stable (nor are they always consistent across OpenSSL, LibreSSL, and BoringSSL, sigh).

I suppose one other approach would be to simply track whether we've loaded legacy or not and just add some alternate text to the error, e.g. " or possibly due to lack of legacy provider"

[cjwatson]

Also, I wonder if "widely used in PEM and PKCS#12" may be a reason why Debian's OpenSSL team should reconsider their packaging structure? The original motivation in @xnox's bug report said "None of the algorithms it provides are useful, or needed at all", which is strongly at odds with what you're saying here. Perhaps they were unaware of how widespread the uses actually are?

[xnox]

I have been hit by this in Wolfi OS / Chainguard too.

IMHO the variable as is doesn't make sense (opt into secure behaviour).

Standards compliance and certification require to block this. Thus I am literally shipping docker containers that set said variable, because with strict FIPS mode enabled it just crashes. Even if legacy provider is present.

I am trying to go even further than just splitting. IMHO all distros should start shipping opensslconf.h setting / indicating that none of these things are available anymore. Such that without breaking runtime ABI, we can make all runtime applications to stop using these things.

RC2 PKCS12 is museum cryptography. Upstream calls it pathetically weak.

Please convert your private keys to anything stronger or unencrypted and protected by other means.

[alex]

@xnox You should re-consider your approach, both maintainers of this library had extremely negative reactions to your comment.

And the reason for that is that you've failed to understand how ecosystems works, and instead loudly demanded things from people who cannot give you them.

We support RC2 in order to parse things emitted by other projects. We don't ourselves encrypt anything with RC2, and haven't for a long time (if ever). We're not at all confused about how cryptographically garbage it is.

However, RC2 remains widely used. The macOS still uses RC2-40 for encrypting the certificate container in PKCS#12. It's also what OpenSSL<3.0 used, most examples for libraries like bouncycastle continue to show, Windows uses it in some circumstances, etc. Looking around at the current state of this ecosystem you'll find numerous examples using the -legacy flag on OpenSSL's CLI specifically to force the use of RC2.

Most people don't use PKCS#12 encryption as a security boundary, they use it as an interchange format.

So we're as interested in anyone else in getting rid of RC2, however we also need to manage our breakage budget (see https://alexgaynor.net/2021/oct/07/whats-in-a-version-number/), and breaking common workflows like "parse a PKCS#12 from macOS" for limited security gain.

If you want to get rid of PKCS#12, the place to start is with people who emit it, not people who parse it.

[xnox]

Supporting converting RC2-40-CBC files to AES-256-CBC makes sense on Linux as a standalone tool.

Requiring runtime cryptographic libraries to support direct usage of RC2-40-CBC certificates without requiring to upgrade them first does not make sense.

The risk of having legacy provider available and loaded is too large causing to unintentionally use weak cryptography.

Ditto all secret scanning and security scanners should flag up weak private keys.

The common workflows must change to add a new step of converting private keys to strong encryption prior to usage.

We must protect user privacy, even if Mac OS chooses not to.

I will also email Apple to address their defaults.

[kroeckx]

I would like to come to a situation where the legacy provider is only loaded in cases where the maintainer knows about it, and that it's needed, for instance the application that needs to do PKCS#12 with RC2. I do not want something RC2 to be available in all applications. So loading it when it's available is also not a preferred solution for me, I would like to push it application.

[alex]

As a general purpose cryptography library, we don't really know what our users are going to use us for, which makes it more challenging to only load it when required. (If Python packaging had some sort of "features" indicator, ala Rust's Cargo, we could use it to opt-in, but it doesn't.)

[xnox]

As a general purpose cryptography library, we don't really know what our users are going to use us for, which makes it more challenging to only load it when required. (If Python packaging had some sort of "features" indicator, ala Rust's Cargo, we could use it to opt-in, but it doesn't.)

Assume that OpenSSL on some distributions no longer provides RC2. Meaning try not to go via OpenSSL to gain RC2. Most of the time it should not be needed. And when that fails, fallback to a vendored in implementation of RC2 to on-the-fly convert to unencrypted or AES-256-CBC to then continue back via OpenSSL or anything else.

This is similar to how other pieces of software have adapted to continue using obsolete cryptography, as required to maintain interoperability with whatever you want to support. I.e. I have seen projects vendor in MD4, MD5, RC2, RC4 to continue supporting NTLM auth and similar things.

Crashing upon importing your library, because legacy provider is not available, is not optimal for all the users who are on modern linux distributions, and are not planning on using weak cryptography (which obviously is impossible to know ahead of time). As the current variable is "opt out of legacy algorithms support", instead of "opt into legacy algorithms support", and thus forcing systems to carry additional runtime dependencies which depending on use case will never be used (with a declining probability over time).

[stefanor]

Vendoring legacy cryptography functions in 2nd-level libraries to allow core cryptographic libraries to drop support for them, seems.... not particularly useful

[alex]

It also simply wouldn't work. RC2 decryption is handled by OpenSSL as part of the PKCS#12 parse-and-decrypt process, there's not an entrypoint for us to only do that RC2 encryption.

[stefanor]

Debian has reverted this change for now, to figure out next steps without breaking everything.

[kroeckx]

Would it be possible to only load the legacy provider if some function is called?

[xnox]

About Mac OS X: As per https://openradar.appspot.com/FB8988319 it seems that Mac OS Catalina 10.15.x appear to lack support for AES-256-CBC In either macOS 11 Big Sur or macOS 12 Monterey AES-256-CBC support was added, with default message digest set to MD5. macOS 13 Ventura switch default message digest to SHA256. Based on this discussion https://discussions.apple.com/thread/254328280?sortBy=rank

Version history is available at https://en.wikipedia.org/wiki/MacOS_version_history

Note that Catalina release went EOL in 2022, but macOS releases can be very sticky even when hardware supports newer releases, as one may simply not have enough disk space to upgrade, nor a compelling reason to upgrade.

Supported releases of Mac OS do support AES-256-CBC.

Obviously keys generated in previous releases, are not converted automatically.

However, this incompatibility with Mac OS X should be diminishing over time, as people rotate keys / generate new keys on the supported Mac OS X releases.

[reaperhulk]

I just exported a cert+key from Keychain Access on macOS 14.6.1 (latest release) and it used RC2-40 and 3DES. I also tested on a machine running the beta release of Sequoia that was released today -- RC2-40 and 3DES. So while they may support newer algorithms on import, they are still choosing to export legacy cryptography (with no UI options to change it).

[xnox]

I just exported a cert+key from Keychain Access on macOS 14.6.1 (latest release) and it used RC2-40 and 3DES. I also tested on a machine running the beta release of Sequoia that was released today -- RC2-40 and 3DES. So while they may support newer algorithms on import, they are still choosing to export legacy cryptography (with no UI options to change it).

Absolutely fabulous. Probably likely due to appleTV and iPhone support - as I think those devices have longer support time-frames and thus more of them are behind the curve, requiring RC2-40.

How broken is RC-40 and 3DES and how quickly those keys can be bruteforce decrypted, to effectively render them plain text? Sort of pondering if a javascript web-app can do it.

[xnox]

@reaperhulk if you can make a gist of a freshly generated & exported key, that would help with cracking it or providing instructions on how to convert it. As I think at the very least OpenSSL documentation should direct people at upgrading those.

@reaperhulk are you able to test if import of strong pkcs12 file works via gui in the keychain app? (one can generate one with mac os terminal).

[alex]

Let's try to keep this thread on topic. For the purposes of this thread, PKCS#12 + RC2 is a fact of life, we can focus on changing life elsehwere.

@kroeckx to answer your question, I think theoretically we could lazily load it, but it'd be anytime PKCS#12 was used, not just when RC2 was used (and anything else that might require these algorithms, which is maybe all PEM loading?). We'd still have to decide what the behavior is (allow it to silently fail or not). And finally, I think the fact that OpenSSL allows lazy-loading of algorithms is a very poor design decision on their part, and the shared mutable state they have has led to tons of performance regressions, so I'm very reticent to rely on it.