search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.12k stars 4.7k forks source link

openssl 3.0 support #46526

Open satya-rajesh opened 3 years ago

satya-rajesh commented 3 years ago

In continuation to https://github.com/PowerShell/PowerShell/issues/14410

PowerShell-7.0.3$ grep -Rns "ERR_put_error" ./ Binary file ./src/powershell-unix/bin/Linux/netcoreapp3.1/linux-x64/System.Security.Cryptography.Native.OpenSsl.a matches Binary file ./src/powershell-unix/bin/Linux/netcoreapp3.1/linux-x64/System.Security.Cryptography.Native.OpenSsl.so matches Binary file ./bin/System.Security.Cryptography.Native.OpenSsl.a matches Binary file ./bin/System.Security.Cryptography.Native.OpenSsl.so matches

System.Security.Cryptography.Native.OpenSsl.so contains deprecated APIs of Openssl 3.0 Reference: https://www.openssl.org/news/changelog.html

This bug is created for support of openssl 3.0 in dotnet

Project Plan (updated as data is known, checkmarks mean done locally, not necessarily merged)

ghost commented 3 years ago

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq See info in area-owners.md if you want to be subscribed.

Issue Details
In continuation to https://github.com/PowerShell/PowerShell/issues/14410 PowerShell-7.0.3$ grep -Rns "ERR_put_error" ./ Binary file ./src/powershell-unix/bin/Linux/netcoreapp3.1/linux-x64/System.Security.Cryptography.Native.OpenSsl.a matches Binary file ./src/powershell-unix/bin/Linux/netcoreapp3.1/linux-x64/System.Security.Cryptography.Native.OpenSsl.so matches Binary file ./bin/System.Security.Cryptography.Native.OpenSsl.a matches Binary file ./bin/System.Security.Cryptography.Native.OpenSsl.so matches System.Security.Cryptography.Native.OpenSsl.so contains deprecated APIs of Openssl 3.0 Reference: https://www.openssl.org/news/changelog.html This bug is created for support of openssl 3.0 in dotnet
Author: satya-rajesh
Assignees: -
Labels: `area-System.Security`, `untriaged`
Milestone: -
bartonjs commented 3 years ago

I thought we already had an issue tracking this, but apparently it was just on my implicit TODO. So now this is the issue tracking working with OpenSSL 3. (Which we may have to service... at least, I think we did for 1.0/1.1 hybridization)

jeffhandley commented 3 years ago

@bartonjs I just found that we simply had a note card on our project board for it (hence you thinking we had something tracking it). I've updated the board to use this issue instead of the note card.

omajid commented 3 years ago

Fedora 34, tracked via https://github.com/dotnet/core/issues/5851, will have OpenSSL 3.0: https://fedoraproject.org/wiki/Changes/OpenSSL3.0, though there are plans for a OpenSSL 1.1 package for compatibility.

Lxiamail commented 3 years ago

Sorry for accidently closed the issue.

omajid commented 3 years ago

Hey folks! Is there some feature branch or personal repo that I can try out to see what the latest status of this implementation is? I was trying out building runtime with (a pre-release of) OpenSSL 3.0 today and I get a couple of pages worth of build errors. I would like to help out with the port/testing if possible.

bartonjs commented 3 years ago

@omajid I have a tinkering branch at https://github.com/bartonjs/runtime/tree/support_openssl3

My recollection is that https://github.com/bartonjs/runtime/commit/459d2746cbb9b8aecfc63709a26d8b0b4ece1508#diff-9f438d89e563743defcabf3a647a5be42683be66ad8d381683c9967d85d4f214 was the only part required to compile against 3.0, but since I commented out a static assert and haven't written code to deal with it yet there are obviously runtime errors expected.

Moving asymmetric operations off of primitives and onto EVP_PKEY got too big, so I pulled that off as its own thing, but I keep getting distracted by other things.

omajid commented 3 years ago

Thanks! Unfortunately, that doesn't seem to address the compiler errors I am seeing.

Building runtime's main branch leads me to errors like https://gist.github.com/omajid/03106c961a4cc39a2e456aea13302943. I tried support_openssl3 branch, but that leads me to essentially the same set of errors.

The first few errors seem to be fixed by something like this in opensslshim.h:

+#if OPENSSL_VERSION_NUMBER >= OPENSSL_VERSION_3_0_0_RTM
+
+// Remove problematic #defines
+#undef CRYPTO_num_locks
+#undef ERR_load_crypto_strings
+#undef OPENSSL_add_all_algorithms_conf
+#undef SSL_load_error_strings
+#undef SSL_library_init
+#undef CRYPTO_set_locking_callback
+
+#endif

I will keep working away at this.

bartonjs commented 3 years ago

Hmmm.. I probably only tried building non-portable when building against 3.0

omajid commented 3 years ago

Thanks, non-portable worked for me too using support_openssl3 branch, once I had bypassed the non-portable build errors.

omajid commented 3 years ago

Moving asymmetric operations off of primitives and onto EVP_PKEY got too big, so I pulled that off as its own thing, but I keep getting distracted by other things.

Not sure what your schedule or the scope of this implementation change looks like, but I wonder if it we should get runtime building/running against OpenSSL 3.0 first (even if that means using deprecated (but not removed) primitives) and then look at moving to EVP_KEY methods?

bartonjs commented 3 years ago

but I wonder if it we should get runtime building/running against OpenSSL 3.0 first

If that not being done is blocking you from downstream work, I can shift it around. Largely it was paying down some debt and making sure I started off in a good place. (e.g. if some tests using RSA primitives fail are they worth fixing?)

omajid commented 3 years ago

If that not being done is blocking you from downstream work, I can shift it around.

Thanks for the very kind offer. I am not blocked today/right now, but I expect I am going to be blocked in a few weeks. I am not sure how long you expect the pay-down-debt phase to take, but if it's only a few more days/weeks, then I think there's no need to change you plans at all.

bartonjs commented 3 years ago

With direct compilation (build_native.sh -portablebuild=false) there are a few compile errors around const/const-cast due to some const-ness changing between 1.1 and 3, otherwise there are three (and a half) major errors:

The bigger hurdle is making portable work across all 3 versions, because some former functions are now macros, and that makes things hard.

bartonjs commented 3 years ago

Just wanted to give an update:

My current work-in-progress branch (https://github.com/bartonjs/runtime/tree/openssl3_portable_stage) successfully builds on Ubuntu 18.04 (OpenSSL 1.1.1 headers) and runs on an Ubuntu 16.04 (no, I don't remember why I picked the old one) with OpenSSL 3.0 alpha 13 installed into default library paths as well as building non-portable against the OpenSSL 3 headers does on that same machine.

Needs a little bit more love before it builds portable with the OpenSSL 3 headers, and then to make sure that it stays working for portable-against-1.1.1 and that they are both interchangeable.

bartonjs commented 3 years ago

Ah, complications.

1.1.1:

evp.h:# define EVP_PKEY_OP_SIGN                (1<<3)
evp.h:# define EVP_PKEY_OP_VERIFY              (1<<4)

3.0:

evp.h:# define EVP_PKEY_OP_FROMDATA            (1<<3)
evp.h:# define EVP_PKEY_OP_SIGN                (1<<4)
evp.h:# define EVP_PKEY_OP_VERIFY              (1<<5)
satya-rajesh commented 3 years ago

which release is the openssl 3.0.0 support targeted ?

bartonjs commented 3 years ago

which release is the openssl 3.0.0 support targeted ?

.NET 6

omajid commented 3 years ago

I am interested in getting this support merged into 3.1 as well. I have a draft PR here: https://github.com/dotnet/corefx/pull/43078

bartonjs commented 3 years ago

@omajid Can you describe the need for supporting OpenSSL 3.0 in 3.1? 6.0 will come out in ~November, and then 3.1 will go out of support about a year later.

At minimum, I don't think we'd accept the PR into release/3.1 while OpenSSL 3.0 is still calling their builds alpha, so it's unclear how much time this would grant if the problem space is just ".NET LTS using OpenSSL 3".

It's also worth noting that I'm/we're not done with the OpenSSL 3.0 project... we want to get the rest of the runtime onto EVP_PKEY and off of the primitives to avoid dependencies on now-deprecated API.

omajid commented 3 years ago

@omajid Can you describe the need for supporting OpenSSL 3.0 in 3.1? 6.0 will come out in ~November, and then 3.1 will go out of support about a year later.

Happy to! I am interested in this because a couple of Linux distributions that I care about will be moving to OpenSSL 3.0 soon.

Fedora is planning to use it as soon as 3.0 is declared GA. https://fedoraproject.org/wiki/Changes/OpenSSL3.0 was initially for Fedora 34, but got pushed to Fedora 35 due to OpenSSL upstream being late. There is a plan to have a compatibility package but I am not sure how long that will be supported or if it would be usable for building.

It's also looking like RHEL 9 will include (only?) OpenSSL 3.0: https://bugzilla.redhat.com/show_bug.cgi?id=1955873

At minimum, I don't think we'd accept the PR into release/3.1 while OpenSSL 3.0 is still calling their builds alpha

Yeah, that's completely fair. I was motivated by the distros above and figured it's better to share my work than keep it to myself.

it's unclear how much time this would grant if the problem space is just ".NET LTS using OpenSSL 3".

True. Maybe the better move for these distributions would be waiting for the next LTS (6.0) and not shipping/supporting .NET Core 3.1?

It's also worth noting that I'm/we're not done with the OpenSSL 3.0 project... we want to get the rest of the runtime onto EVP_PKEY and off of the primitives to avoid dependencies on now-deprecated API.

Thanks! I didn't realize that - even though I now see that this issue is still open. I built portable and non portable builds against OpenSSL 3.0 alpha 15 and assumed this was fully completed.

satya-rajesh commented 3 years ago

$ grep -r "EVP_MD_size" . dotnet-sdk-6.0.100/shared/Microsoft.NETCore.App/6.0.0-preview.5.21301.5/libSystem.Security.Cryptography.Native.OpenSsl.so

EVP_MD_size API seems to be causing an issue for powershell build.

As per https://www.openssl.org/docs/manmaster/man3/EVP_MD_size.html, this function is renamed in openssl 3.0.0

Error observed with https://dotnet.microsoft.com/download/dotnet/6.0, https://github.com/PowerShell/PowerShell/releases/tag/v7.2.0-preview.7 and openssl 3.0.0 beta1:

Welcome to .NET 6.0!^M ---------------------^M SDK Version: 6.0.100-preview.5.21302.13

Telemetry^M ---------^M The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.^M ^M Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

----------------^M Installed an ASP.NET Core HTTPS development certificate.^M To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).^M Learn about HTTPS: https://aka.ms/dotnet-https^M ----------------^M Write your first app: https://aka.ms/dotnet-hello-world^M Find out what's new: https://aka.ms/dotnet-whats-new^M Explore documentation: https://aka.ms/dotnet-docs^M Report issues and find source on GitHub: https://github.com/dotnet/core^M Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli^M

Cannot get required symbol EVP_MD_size from libssl ./build.sh: line 6: 17097 Aborted (core dumped) dotnet restore $f

Any suggestions ?

bartonjs commented 3 years ago

Looks like EVP_MD_size got renamed to EVP_MD_get_size for Beta1, and our support was based on Alpha....14? So we'll need to do another sweep.

bartonjs commented 3 years ago

.NET 6 Preview 7 will work with OpenSSL 3.0 Beta 1. Hopefully there aren't any more renames now that they got to Beta :smile:.

satya-rajesh commented 3 years ago

.NET 6 Preview 7 will work with OpenSSL 3.0 Beta 1. Hopefully there aren't any more renames now that they got to Beta 😄.

Thanks for the update. When will be the preview 7, i didnt find it yet at https://dotnet.microsoft.com/download/dotnet/6.0 ?

bartonjs commented 3 years ago

Preview 7 will come out in about a month. (Preview 6 came out two days ago)

bartonjs commented 3 years ago

Moving the issue to 7.0 to track that we still have some plumbing/infrastructure work to do to get off of deprecated APIs. While doing the DSA* => EVP_PKEY* change the tests revealed that we probably want to clean up how we interact with the error queue (#55973), which is too concerning to do with the limited available time in 6.0.

6.0 will still ship with OSSL3 support, it's just not as clean as I'd like.

adamijak commented 2 years ago

My Razor server fails to start on Fedora 36 and VsCode, with error message Cannot get required symbol EVP_MD_size from libssl

> dotnet --list-sdks
3.1.417 [/usr/lib64/dotnet/sdk]
6.0.103 [/usr/lib64/dotnet/sdk]
> openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

I should have openssl1.1 installed, is there any way I can fix it?

omajid commented 2 years ago

Hey, @adamijak do you have a set of steps I can follow to reproduce the issue you are seeing?

I ran the following steps and I couldn't see a problem:

podman run -it registry.fedoraproject.org/fedora:36 /bin/bash
dnf install dotnet-sdk-3.1 dotnet-sdk-6.0 openssl1.1
mkdir -p foobar
cd foobar
dotnet new razor
dotnet run

Are you seeing errors running via dotnet run? Or is it the vscode instance the one showing errors? Does the error say anything about OmniSharp? Can you share the error messages (preferably the complete set of logs and/or stack trace) here?

adamijak commented 2 years ago

Hey, @adamijak do you have a set of steps I can follow to reproduce the issue you are seeing?

I ran the following steps and I couldn't see a problem:

podman run -it registry.fedoraproject.org/fedora:36 /bin/bash
dnf install dotnet-sdk-3.1 dotnet-sdk-6.0 openssl1.1
mkdir -p foobar
cd foobar
dotnet new razor
dotnet run

Are you seeing errors running via dotnet run? Or is it the vscode instance the one showing errors? Does the error say anything about OmniSharp? Can you share the error messages (preferably the complete set of logs and/or stack trace) here?

Hi I opened separate issue #67375 for this

jeffhandley commented 1 year ago

@bartonjs Do you have any objections to us punting this out of .NET 8 (into Future for the time-being), and revisiting the remaining work as a candidate for .NET 9 planning?