search

NuGet / Home

Repo for NuGet Client issues
Other
1.5k stars 252 forks source link

Package signature verification is slow on Linux #12672

Open marcin-krystianc opened 1 year ago

marcin-krystianc commented 1 year ago

NuGet Product Used

dotnet.exe

Product Version

7.0.107, 8.0.100-preview.6.23319.41

Worked before?

No response

Impact

It bothers me. A fix would be nice

Repro Steps & Context

https://github.com/marcin-krystianc/TestNuGetSignatureVerification

TestNuGetSignatureVerification on Linux

> apt-get update && apt-get install -y git dotnet7
> dotnet --info
> git clone https://github.com/marcin-krystianc/TestNuGetSignatureVerification.git
> git clone https://github.com/dotnet/orleans.git
> dotnet restore orleans/Orleans.sln
> export DOTNET_NUGET_SIGNATURE_VERIFICATION=True
> dotnet run -c Release --project TestNuGetSignatureVerification/TestNuGetSignatureVerification/TestNuGetSignatureVerification.csproj -- 1
Using MSBuild:/usr/lib/dotnet/sdk/7.0.107
Found 479 packages in '/root/.nuget/packages/
Verified 479 packages in '31.7100083 seconds with degree of parallelism=1

> dotnet nuget locals --list global-packages
> time dotnet nuget verify --all ~/.nuget/packages/**/*/*.nupkg -v diag > out.txt
real    0m47.733s
user    0m48.614s
sys     0m0.983s

TestNuGetSignatureVerification on Windows (PowerShell)

> dotnet --info
> git clone https://github.com/marcin-krystianc/TestNuGetSignatureVerification.git
> git clone https://github.com/dotnet/orleans.git
> dotnet restore orleans/Orleans.sln
> dotnet run -c Release --project TestNuGetSignatureVerification/TestNuGetSignatureVerification/TestNuGetSignatureVerification.csproj -- 1
Using MSBuild:C:\Program Files\dotnet\sdk\7.0.304
Found 493 packages in 'C:\Users\Marcin\.nuget\packages\
Verified 493 packages in '4.1218048 seconds with degree of parallelism=1

> dotnet nuget locals --list global-packages
> Measure-Command {dotnet nuget verify --all $env:USERPROFILE\.nuget\packages\**\*\*.nupkg -v diag > out.txt}
...
Seconds           : 4
Milliseconds      : 638
...

I would expect that signature verification on Linux and on Windows takes a similar time. Unfortunately, signature verification on Linux is about an order of magnitude slower. It affects the "clean" restore scenarios, where the NuGet packages need to be installed into global-packages folder as the signature verification was recently enabled by default on Linux.

It is probably not even a problem with NuGet itself but with how the signature verification is implemented on Windows and on Linux. I'm reporting it here as I've discovered it with NuGet and also it provides an easy way to reproduce it.

Verbose Logs

No response

nkolev92 commented 1 year ago

479 vs 493 package probably makes the comparison not 100% equal, but the difference shouldn't be 10x.

cc @dtivel are you familiar with any issues here?

erdembayar commented 1 year ago

Thank you for reporting this issue. Package signature validation depends on many ambient factors, such as the certificate revocation list. Could you please try turning off the validation by using an environmental variable and then try again?

Happy coding! 🧑‍💻

marcin-krystianc commented 1 year ago

Hi @erdembayar ,

I've tried different values for NUGET_CERT_REVOCATION_MODE (Offline and Online) but it didn't make any significant difference. Regarding the DOTNET_NUGET_SIGNATURE_VERIFICATION, it really affects the restore operation. (PS. I'm using --disable-parallel because the results are less noise than without it.)

root@4db7832c41c4:/orleans# dotnet --version
7.0.305
root@4db7832c41c4:/orleans# for i in {1..10};do dotnet nuget locals -c global-packages && DOTNET_NUGET_SIGNATURE_VERIFICATION=False dotnet restore -clp:summary /p:RestoreUseStaticGraphEvaluation=true --disable-parallel; done | grep -i Elapsed
Time Elapsed 00:00:14.57
Time Elapsed 00:00:10.55
Time Elapsed 00:00:12.46
Time Elapsed 00:00:11.85
Time Elapsed 00:00:12.03
Time Elapsed 00:00:10.02
Time Elapsed 00:00:09.83
Time Elapsed 00:00:11.21
Time Elapsed 00:00:09.76
Time Elapsed 00:00:18.09
root@4db7832c41c4:/orleans# for i in {1..10};do dotnet nuget locals -c global-packages && DOTNET_NUGET_SIGNATURE_VERIFICATION=True dotnet restore -clp:summary /p:RestoreUseStaticGraphEvaluation=true --disable-parallel; done | grep -i Elapsed
Time Elapsed 00:00:35.81
Time Elapsed 00:00:29.04
Time Elapsed 00:00:29.67
Time Elapsed 00:00:32.48
Time Elapsed 00:00:29.65
Time Elapsed 00:00:33.99
Time Elapsed 00:00:29.56
Time Elapsed 00:00:28.82
Time Elapsed 00:00:28.37
Time Elapsed 00:00:31.43
dtivel commented 1 year ago

First, thank you, @marcin-krystianc, for the report and the excellent repro steps. I was able to reproduce the behavior you describe.

I get 475 packages. I don't think the difference (475 vs. 479 vs. 493) is significant.

All 475 packages are from https://nuget.org and have a repository signature and timestamp. Looking at all 475 repository signature timestamps, there are only 3 distinct timestamp certificate chains. NuGet client builds a certificate chain every time it validates a timestamp, so that's 475 chain builds for only 3 distinct certificate chains. The same idea applies to repository signatures; there are 475 chain builds for only 2 distinct certificate chains. To a much lesser extent, this phenomenon also applies to author signatures and their timestamps. As an experiment, I added in NuGet caching of successful chain building results, such that chain building was performed only once per leaf certificate. The overall time dropped by ~24%. It's obvious from experimentation that Windows has some additional per-process caching which memoizes certificate chain building results, and we get this optimization for free on the Windows platform. The Linux platform doesn't provide an equivalent optimization, so we'll have to look into if/where a similar optimization makes sense on Linux.

That experiment accounted for ~24% of the overall time. As for the rest of the difference between Windows and Linux, I don't have clear culprits yet. One trace suggested that ~33% of the overall time was spent in regular expression classes, but I can't corroborate this.

.NET already provides some caching of intermediate chain building results (CRL's, OCSP responses, and certificates fetched via AIA), which is great. This is because the platform (Linux) doesn't provide equivalent behaviors by default. I agree with your sentiment that this is "probably not even a problem with NuGet itself but with how the signature verification is implemented on Windows and on Linux."

For now, if the performance hit is unacceptable, you have the option of disabling verification using the DOTNET_NUGET_SIGNATURE_VERIFICATION environment variable.

marcin-krystianc commented 1 year ago

@dtivel Thanks for your detailed response. I think, that another factor for slow certificate verification on Linux is OpenSSL itself. There are one or more performance regressions in it (e.g. https://github.com/openssl/openssl/issues/20051). I've confirmed it by running my tests on an older version of Ubuntu, and the results were much better. I'll keep digging and will post a detailed response once I've precise data.

PS: Here are some test results:

dtivel commented 1 year ago

@marcin-krystianc, thank you for the additional information. I tested on Debian 12 "Bookworm". openssl version prints OpenSSL 3.0.9 30 May 2023 (Library: OpenSSL 3.0.9 30 May 2023).

adrien-n commented 1 year ago

Hi, I am finishing preparing a SRU (stable release update) for openssl in Ubuntu 22.04. It includes a series of patch that improves performance and could help your usecase.

There is a PPA at https://launchpad.net/~adrien-n/+archive/ubuntu/openssl-jammy-sru which you can test and I would welcome performance numbers if you do so.

marcin-krystianc commented 11 months ago

Hi @adrien-n,

I've run another round of benchmarks. I've followed https://launchpad.net/~adrien-n/+archive/ubuntu/openssl-jammy-sru, but it didn't make a massive difference (only a few percent of improvement). Performance in Ubuntu 24.04 (OpenSSL 3.0.10 1 Aug 2023) is visibly faster, but still far from Ubuntu 20.04 (OpenSSL 1.1.1f 31 Mar 2020).

Updated test script:

export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 \
&& apt-get update && apt-get install -y git wget \
&& mkdir dotnet \
&& wget https://download.visualstudio.microsoft.com/download/pr/5226a5fa-8c0b-474f-b79a-8984ad7c5beb/3113ccbf789c9fd29972835f0f334b7a/dotnet-sdk-8.0.100-linux-x64.tar.gz \
&& tar -C dotnet -xvzf dotnet-sdk-8.0.100-linux-x64.tar.gz \
&& rm dotnet-sdk-8.0.100-linux-x64.tar.gz \
&& echo "export DOTNET_ROOT=/dotnet" >> $HOME/.bashrc \
&& echo "export PATH=$PATH:/dotnet:$HOME/.dotnet/tools" >> $HOME/.bashrc \
&& source ~/.bashrc \
&& git clone https://github.com/dotnet/orleans.git \
&& git --git-dir=/orleans/.git checkout 2e454ae58197ade4855392bdeffaa86bf3b588fd \
&& dotnet --info \
&& openssl version \
&& export DOTNET_NUGET_SIGNATURE_VERIFICATION=False \
&& echo DOTNET_NUGET_SIGNATURE_VERIFICATION=False \
&& for i in {1..5};do dotnet nuget locals -c global-packages && dotnet restore -clp:summary /p:RestoreUseStaticGraphEvaluation=true orleans/Orleans.sln --disable-parallel; done | grep -i Elapsed \
&& export DOTNET_NUGET_SIGNATURE_VERIFICATION=True \
&& echo DOTNET_NUGET_SIGNATURE_VERIFICATION=True \
&& for i in {1..5};do dotnet nuget locals -c global-packages && dotnet restore -clp:summary /p:RestoreUseStaticGraphEvaluation=true orleans/Orleans.sln --disable-parallel; done | grep -i Elapsed

Results:

# ubuntu:22.04
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
DOTNET_NUGET_SIGNATURE_VERIFICATION=False
Time Elapsed 00:01:26.04
Time Elapsed 00:00:09.83
Time Elapsed 00:00:09.46
Time Elapsed 00:00:09.53
Time Elapsed 00:00:09.66
DOTNET_NUGET_SIGNATURE_VERIFICATION=True
Time Elapsed 00:00:30.28
Time Elapsed 00:00:29.51
Time Elapsed 00:00:29.30
Time Elapsed 00:00:29.42
Time Elapsed 00:00:29.69

# ubuntu:22.04 + https://launchpad.net/~adrien-n/+archive/ubuntu/openssl-jammy-sru:
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
DOTNET_NUGET_SIGNATURE_VERIFICATION=False
Time Elapsed 00:00:09.90
Time Elapsed 00:00:09.28
Time Elapsed 00:00:10.51
Time Elapsed 00:00:10.08
Time Elapsed 00:00:10.55
DOTNET_NUGET_SIGNATURE_VERIFICATION=True
Time Elapsed 00:00:29.47
Time Elapsed 00:00:28.36
Time Elapsed 00:00:28.50
Time Elapsed 00:00:28.46
Time Elapsed 00:00:28.67

# ubuntu:24.04
OpenSSL 3.0.10 1 Aug 2023 (Library: OpenSSL 3.0.10 1 Aug 2023)
DOTNET_NUGET_SIGNATURE_VERIFICATION=False
Time Elapsed 00:01:27.11
Time Elapsed 00:00:09.55
Time Elapsed 00:00:09.12
Time Elapsed 00:00:10.31
Time Elapsed 00:00:09.44
DOTNET_NUGET_SIGNATURE_VERIFICATION=True
Time Elapsed 00:00:26.43
Time Elapsed 00:00:25.42
Time Elapsed 00:00:24.93
Time Elapsed 00:00:25.23
Time Elapsed 00:00:24.64

# ubuntu:20.04
OpenSSL 1.1.1f  31 Mar 2020
DOTNET_NUGET_SIGNATURE_VERIFICATION=False
Time Elapsed 00:01:56.35
Time Elapsed 00:00:10.94
Time Elapsed 00:00:09.55
Time Elapsed 00:00:09.43
Time Elapsed 00:00:09.68
DOTNET_NUGET_SIGNATURE_VERIFICATION=True
Time Elapsed 00:00:13.84
Time Elapsed 00:00:12.48
Time Elapsed 00:00:13.10
Time Elapsed 00:00:12.72
Time Elapsed 00:00:13.92
adrien-n commented 10 months ago

Thanks a lot for testing. This seems to validate the patch is useful (if a single patch series brings 20% of the improvement of 1000 patch series, it seems interesting to me).

Unfortunately the performance is still way worse. It's possible that openssl 3.1 or 3.2 would be much faster. The issue with openssl for Ubuntu is that there is no more recent openssl LTS version and we want openssl LTS for our own LTS. Since there is also no calendar, this also prevents us from moving forward in interim releases since we don't know if there will be an LTS in time for our next LTS.