search

AzureAD / azure-activedirectory-identitymodel-extensions-for-dotnet

IdentityModel extensions for .Net
MIT License
1.06k stars 400 forks source link

Run performance benchmarks to compare IdentityModel on .NET 8 and .NET 9 #2828

Closed pmaytak closed 1 month ago

pmaytak commented 1 month ago

Run benchmarks (probably just the common token validation since it's closer to user's E2E scenarios) for comparison.

Compile Runtime
.NET 8 .NET 8
.NET 8 .NET 9
.NET 9 .NET 9
pmaytak commented 1 month ago

The benchmarks were run on the feature branch https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/pull/2817 which includes improvements to Base64Url related logic (for both, .NET 9 and < .NET 8 via Microsoft.Bcl.Memory version 9.0.0-rc.1.24431.7). The benchmarks ran were chosen to show typical end-to-end scenarios.

Method Compile Runtime Mean Ratio Error StdDev P90 P95 P100 Gen0 Gen1 Allocated Alloc Ratio
JsonWebTokenHandler_ValidateTokenAsyncWithTVP .NET 8 .NET 8 29.80 μs 1.00 0.078 μs 0.169 μs 29.96 μs 30.05 μs 30.38 μs 0.4272 - 7.23 KB 1.00
JsonWebTokenHandler_ValidateTokenAsyncWithTVP .NET 8 .NET 9 27.91 μs 0.937 0.065 μs 0.142 μs 28.16 μs 28.17 μs 28.19 μs 0.4272 - 7.2 KB 0.996
JsonWebTokenHandler_ValidateTokenAsyncWithTVP .NET 9 .NET 9 27.86 μs 0.935 0.072 μs 0.156 μs 28.08 μs 28.11 μs 28.13 μs 0.4272 - 7.2 KB 0.996
Method Compile Runtime Mean Ratio Error StdDev P90 P95 P100 Gen0 Gen1 Allocated Alloc Ratio
JsonWebTokenHandler_ValidateJWEAsync .NET 8 .NET 8 36.19 μs 1.00 0.091 μs 0.191 μs 36.41 μs 36.47 μs 36.82 μs 0.8545 0.1221 15.83 KB 1.00
JsonWebTokenHandler_ValidateJWEAsync .NET 8 .NET 9 34.26 μs 0.947 0.115 μs 0.247 μs 34.54 μs 34.72 μs 34.84 μs 0.8545 0.1221 15.84 KB 1.001
JsonWebTokenHandler_ValidateJWEAsync .NET 9 .NET 9 34.19 μs 0.945 0.114 μs 0.249 μs 34.50 μs 34.54 μs 34.76 μs 0.8545 0.1221 15.84 KB 1.001
Method Compile Runtime Mean Ratio Error StdDev P90 P95 P100 Gen0 Gen1 Allocated Alloc Ratio
SHRHandler_ValidateSignedHttpRequestAsync .NET 8 .NET 8 72.61 μs 1.00 0.091 μs 0.202 μs 72.86 μs 72.92 μs 73.15 μs 0.7324 - 15.45 KB 1.00
SHRHandler_ValidateSignedHttpRequestAsync .NET 8 .NET 9 70.10 μs 0.965 0.091 μs 0.201 μs 70.31 μs 70.40 μs 70.72 μs 0.7324 - 15.4 KB 0.997
SHRHandler_ValidateSignedHttpRequestAsync .NET 9 .NET 9 69.87 μs 0.962 0.097 μs 0.200 μs 70.13 μs 70.17 μs 70.38 μs 0.7324 - 15.4 KB 0.997

Same results but as an image (easier to read). image

Benchmark info for all the runs:

BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.4169/23H2/2023Update/SunValley3) (Hyper-V)
AMD EPYC 7763, 1 CPU, 16 logical and 8 physical cores
.NET SDK 9.0.100-rc.1.24452.12
  [Host]    : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
  MediumRun : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2

Job=MediumRun  MaxAbsoluteError=10.0000 ms  Runtime=.NET 8.0  
IterationCount=15  LaunchCount=4  WarmupCount=10  
----
BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.4169/23H2/2023Update/SunValley3) (Hyper-V)
AMD EPYC 7763, 1 CPU, 16 logical and 8 physical cores
.NET SDK 9.0.100-rc.1.24452.12
  [Host]    : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
  MediumRun : .NET 9.0.0 (9.0.24.43107), X64 RyuJIT AVX2

Job=MediumRun  MaxAbsoluteError=10.0000 ms  Runtime=.NET 9.0  
IterationCount=15  LaunchCount=4  WarmupCount=10
----
BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.4169/23H2/2023Update/SunValley3) (Hyper-V)
AMD EPYC 7763, 1 CPU, 16 logical and 8 physical cores
.NET SDK 9.0.100-rc.1.24452.12
  [Host]    : .NET 9.0.0 (9.0.24.43107), X64 RyuJIT AVX2
  MediumRun : .NET 9.0.0 (9.0.24.43107), X64 RyuJIT AVX2

Job=MediumRun  MaxAbsoluteError=10.0000 ms  Runtime=.NET 9.0  
IterationCount=15  LaunchCount=4  WarmupCount=10
pmaytak commented 1 month ago

These benchmarks were run on the dev branch (without Base64Url related improvements).

Method Compile Runtime Mean Ratio Error StdDev P90 P95 P100 Gen0 Gen1 Allocated Alloc Ratio
JsonWebTokenHandler_ValidateTokenAsyncWithTVP .NET 8 .NET 8 31.52 μs 1.00 0.062 μs 0.136 μs 31.68 μs 31.75 μs 31.85 μs 0.4272 - 7.23 KB 1.00
JsonWebTokenHandler_ValidateTokenAsyncWithTVP .NET 8 .NET 9 29.78 μs 0.945 0.088 μs 0.192 μs 30.03 μs 30.13 μs 30.47 μs 0.3662 - 7.2 KB 0.996
JsonWebTokenHandler_ValidateTokenAsyncWithTVP .NET 9 .NET 9 30.20 μs 0.958 0.462 μs 0.986 μs 31.85 μs 31.94 μs 32.10 μs 0.3662 - 7.2 KB 0.996
Method Compile Runtime Mean Ratio Error StdDev P90 P95 P100 Gen0 Gen1 Allocated Alloc Ratio
JsonWebTokenHandler_ValidateJWEAsync .NET 8 .NET 8 38.58 μs 1.00 0.137 μs 0.303 μs 38.98 μs 39.11 μs 39.42 μs 0.9766 0.1221 16.97 KB 1.00
JsonWebTokenHandler_ValidateJWEAsync .NET 8 .NET 9 37.05 μs 0.960 0.160 μs 0.344 μs 37.54 μs 37.68 μs 38.10 μs 0.9766 0.1221 16.98 KB 1.001
JsonWebTokenHandler_ValidateJWEAsync .NET 9 .NET 9 36.73 μs 0.952 0.124 μs 0.267 μs 37.07 μs 37.17 μs 37.51 μs 0.9766 0.1221 16.98 KB 1.001
Method Compile Runtime Mean Ratio Error StdDev P90 P95 P100 Gen0 Gen1 Allocated Alloc Ratio
SHRHandler_ValidateSignedHttpRequestAsync .NET 8 .NET 8 77.03 μs 1.00 0.122 μs 0.265 μs 77.36 μs 77.44 μs 77.86 μs 0.7324 - 15.84 KB 1.00
SHRHandler_ValidateSignedHttpRequestAsync .NET 8 .NET 9 74.38 μs 0.966 0.164 μs 0.350 μs 74.83 μs 74.93 μs 75.03 μs 0.7324 - 15.79 KB 0.997
SHRHandler_ValidateSignedHttpRequestAsync .NET 9 .NET 9 74.74 μs 0.970 0.209 μs 0.454 μs 75.47 μs 75.56 μs 75.92 μs 0.7324 - 15.79 KB 0.997

Same results but as an image (easier to read). image

pmaytak commented 1 month ago

These are the combined results of the above two sets of runs where .NET 8 does not include Base64Url improvement and .NET 9 does. The ratio was readjusted.

Method Compile Runtime Base64Url Mean Ratio Error StdDev P90 P95 P100 Gen0 Gen1 Allocated Alloc Ratio
JsonWebTokenHandler_ValidateTokenAsyncWithTVP .NET 8 .NET 8 Not changed 31.52 μs 1.00 0.062 μs 0.136 μs 31.68 μs 31.75 μs 31.85 μs 0.4272 - 7.23 KB 1.00
JsonWebTokenHandler_ValidateTokenAsyncWithTVP .NET 8 .NET 9 Not changed 29.78 μs 0.945 0.088 μs 0.192 μs 30.03 μs 30.13 μs 30.47 μs 0.3662 - 7.2 KB 0.996
JsonWebTokenHandler_ValidateTokenAsyncWithTVP .NET 9 .NET 9 Changed 27.86 μs 0.884 0.072 μs 0.156 μs 28.08 μs 28.11 μs 28.13 μs 0.4272 - 7.2 KB 0.996
Method Compile Runtime Base64Url Mean Ratio Error StdDev P90 P95 P100 Gen0 Gen1 Allocated Alloc Ratio
JsonWebTokenHandler_ValidateJWEAsync .NET 8 .NET 8 Not changed 38.58 μs 1.00 0.137 μs 0.303 μs 38.98 μs 39.11 μs 39.42 μs 0.9766 0.1221 16.97 KB 1.00
JsonWebTokenHandler_ValidateJWEAsync .NET 8 .NET 9 Not changed 37.05 μs 0.960 0.160 μs 0.344 μs 37.54 μs 37.68 μs 38.10 μs 0.9766 0.1221 16.98 KB 1.001
JsonWebTokenHandler_ValidateJWEAsync .NET 9 .NET 9 Changed 34.19 μs 0.886 0.114 μs 0.249 μs 34.50 μs 34.54 μs 34.76 μs 0.8545 0.1221 15.84 KB 0.933
Method Compile Runtime Base64Url Mean Ratio Error StdDev P90 P95 P100 Gen0 Gen1 Allocated Alloc Ratio
SHRHandler_ValidateSignedHttpRequestAsync .NET 8 .NET 8 Not changed 77.03 μs 1.00 0.122 μs 0.265 μs 77.36 μs 77.44 μs 77.86 μs 0.7324 - 15.84 KB 1.00
SHRHandler_ValidateSignedHttpRequestAsync .NET 8 .NET 9 Not changed 74.38 μs 0.966 0.164 μs 0.350 μs 74.83 μs 74.93 μs 75.03 μs 0.7324 - 15.79 KB 0.997
SHRHandler_ValidateSignedHttpRequestAsync .NET 9 .NET 9 Changed 69.87 μs 0.907 0.097 μs 0.200 μs 70.13 μs 70.17 μs 70.38 μs 0.7324 - 15.4 KB 0.972

Same results but as an image (easier to read). image