dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 663 Improvements on 9/26/2023 6:21:18 AM #22587

Closed performanceautofiler[bot] closed 9 months ago

performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Perf_Convert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.45 μs 5.78 μs 0.78 0.08 True
306.99 ns 258.96 ns 0.84 0.07 False
7.07 μs 5.39 μs 0.76 0.06 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Perf_Convert*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Perf_Convert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Perf_Convert*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Perf_Convert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Perf_Convert.ToBase64String(formattingOptions: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.783993342790387 < 7.0798763999962615. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 132.71187821189702 (T) = (0 -5819.507416390688) / Math.Sqrt((16330.750606319774 / (141)) + (1299.4745983487126 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.22340919781462912 = (7493.659981568494 - 5819.507416390688) / 7493.659981568494 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Perf_Convert.ToHexString #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 258.95648483803495 < 294.54012023224016. IsChangePoint: Marked as a change because one of 8/21/2023 4:59:56 PM, 9/13/2023 5:14:37 AM, 9/27/2023 6:04:31 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 23.096549382019585 (T) = (0 -260.2172532585397) / Math.Sqrt((314.5617937377487 / (153)) + (2.437695535397992 / (22))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (153) + (22) - 2, .975) and 0.11555473775402784 = (294.2152153065307 - 260.2172532585397) / 294.2152153065307 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Perf_Convert.ToBase64CharArray(binaryDataSize: 1024, formattingOptions: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.386436442818381 < 6.731364182731632. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 55.63991339123654 (T) = (0 -5452.9299377696) / Math.Sqrt((7890.881351045745 / (140)) + (24634.787753663586 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.23210628569453748 = (7101.151938327319 - 5452.9299377696) / 7101.151938327319 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Perf_Ascii

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
31.39 ns 29.12 ns 0.93 0.09 True
41.91 ns 22.48 ns 0.54 0.01 True
480.26 ns 301.79 ns 0.63 0.04 True
634.39 ns 432.87 ns 0.68 0.01 True
605.41 ns 441.28 ns 0.73 0.01 True
328.81 ns 100.26 ns 0.30 0.01 True
198.33 ns 86.10 ns 0.43 0.01 True
55.56 ns 48.49 ns 0.87 0.02 True
738.33 ns 530.93 ns 0.72 0.01 True
161.32 ns 24.02 ns 0.15 0.01 True
289.76 ns 104.94 ns 0.36 0.01 True
55.13 ns 47.70 ns 0.87 0.01 True
178.96 ns 68.06 ns 0.38 0.02 True
87.84 ns 73.96 ns 0.84 0.01 True
165.62 ns 75.85 ns 0.46 0.03 False
137.61 ns 56.35 ns 0.41 0.07 False
289.64 ns 112.36 ns 0.39 0.05 True
164.91 ns 77.93 ns 0.47 0.02 False
87.80 ns 74.66 ns 0.85 0.01 True
72.15 ns 52.01 ns 0.72 0.01 True
17.97 ns 10.00 ns 0.56 0.03 True
139.81 ns 55.08 ns 0.39 0.07 False
72.21 ns 51.93 ns 0.72 0.01 True
705.90 ns 494.15 ns 0.70 0.01 True
31.03 ns 26.90 ns 0.87 0.12 True
606.55 ns 440.30 ns 0.73 0.01 True
67.43 ns 13.86 ns 0.21 0.03 True
15.49 ns 8.54 ns 0.55 0.01 True
42.15 ns 22.63 ns 0.54 0.02 True
304.21 ns 124.36 ns 0.41 0.01 True
149.10 ns 39.88 ns 0.27 0.02 False
360.42 ns 137.77 ns 0.38 0.01 True
29.62 ns 17.91 ns 0.60 0.02 True
540.31 ns 403.77 ns 0.75 0.02 True
305.97 ns 127.35 ns 0.42 0.01 True
292.73 ns 80.04 ns 0.27 0.09 True
58.26 ns 45.32 ns 0.78 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Perf_Ascii*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Perf_Ascii* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Perf_Ascii*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Perf_Ascii* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Perf_Ascii.ToUpperInPlace_Bytes(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 29.12351548321747 < 29.66247356575345. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 13.330622911176988 (T) = (0 -27.71341395773129) / Math.Sqrt((0.17052674720783872 / (141)) + (2.334726869579388 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.11911962978006332 = (31.461041583673676 - 27.71341395773129) / 31.461041583673676 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpperInPlace_Chars(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 22.48034306689948 < 39.85918054414459. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 393.65212066688235 (T) = (0 -22.532556758743855) / Math.Sqrt((0.023882532845311995 / (143)) + (0.06708652303891263 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.46133613115913 = (41.83045877428311 - 22.532556758743855) / 41.83045877428311 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.FromUtf16(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 301.7876027595479 < 452.3723081946115. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 247.6652062118725 (T) = (0 -302.8570833415415) / Math.Sqrt((59.0463679376222 / (142)) + (3.0521079853827695 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.37039992275757483 = (481.030886571711 - 302.8570833415415) / 481.030886571711 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.Equals_Bytes_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 432.8687518852873 < 602.331077499167. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 821.2068523725181 (T) = (0 -433.9404984044475) / Math.Sqrt((1.9111323376275013 / (143)) + (1.3846715611760796 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3158625963520356 = (634.2885158603895 - 433.9404984044475) / 634.2885158603895 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpper_Bytes_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 441.27790137349365 < 575.0262260811378. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 79.87827429771664 (T) = (0 -440.96597721844574) / Math.Sqrt((639.8314287620589 / (144)) + (1.1810720953706595 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.27720718764420715 = (610.0862787791274 - 440.96597721844574) / 610.0862787791274 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.EqualsIgnoreCase_ExactlyTheSame_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 100.26400525696208 < 312.6101960138341. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 846.5067309213783 (T) = (0 -100.59946438630864) / Math.Sqrt((7.618859167404108 / (142)) + (0.5786866283602674 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.6944350141807943 = (329.22444996963935 - 100.59946438630864) / 329.22444996963935 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.EqualsIgnoreCase_DifferentCase_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 86.10054616186794 < 187.94272279547175. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 55.70127551996495 (T) = (0 -85.41679502129998) / Math.Sqrt((613.3650478320664 / (141)) + (0.8848952007265777 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.577114533789986 = (201.98564823431448 - 85.41679502129998) / 201.98564823431448 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpper_Bytes(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 48.49358108154514 < 52.75191757260281. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 51.40262044290707 (T) = (0 -48.57552578934023) / Math.Sqrt((0.045223898887100517 / (145)) + (0.537678129910901 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.125028370327438 = (55.5166866467642 - 48.57552578934023) / 55.5166866467642 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.EqualsIgnoreCase_DifferentCase_Bytes_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 530.927499886575 < 701.7645599350237. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 517.6322956250782 (T) = (0 -532.9045628577907) / Math.Sqrt((3.3283772451531637 / (143)) + (4.0561456141934675 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2788566753082106 = (738.9717752508485 - 532.9045628577907) / 738.9717752508485 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.IsValid_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.021919756015077 < 153.25728485279987. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1217.5123920197288 (T) = (0 -24.23798433570977) / Math.Sqrt((1.6136011715457537 / (144)) + (0.04552205542712421 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.8499833822043813 = (161.5686628046193 - 24.23798433570977) / 161.5686628046193 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpperInPlace_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 104.94098675796064 < 276.4735240073503. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 702.709845362842 (T) = (0 -102.03260299536159) / Math.Sqrt((5.26465071810548 / (142)) + (1.069109598404389 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.6499972655988501 = (291.519445326443 - 102.03260299536159) / 291.519445326443 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLower_Bytes(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 47.703276019067985 < 52.43699755907364. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 67.60453425189456 (T) = (0 -48.41773795676228) / Math.Sqrt((0.07865469095110003 / (144)) + (0.30845976453151475 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.12686223097823027 = (55.452575383387284 - 48.41773795676228) / 55.452575383387284 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.EqualsIgnoreCase_ExactlyTheSame_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 68.06496554571761 < 173.1257846366433. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.11071528075709 (T) = (0 -67.73375704200483) / Math.Sqrt((611.7726178177809 / (142)) + (0.6274611715486589 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.636937229298588 = (186.56211131520845 - 67.73375704200483) / 186.56211131520845 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpper_Bytes_Chars(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 73.96369941087919 < 83.43032982122112. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 158.79641554332912 (T) = (0 -74.47380983553782) / Math.Sqrt((0.10717183605921522 / (145)) + (0.19072338869805439 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.15227129034341597 = (87.85099405882721 - 74.47380983553782) / 87.85099405882721 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLower_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 75.85203786082698 < 157.4423111674019. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 44.23845748416586 (T) = (0 -76.89788705317278) / Math.Sqrt((629.3225327747289 / (143)) + (1.1260589000083252 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5479180302266804 = (170.09722173111768 - 76.89788705317278) / 170.09722173111768 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpperInPlace_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 56.352450677021935 < 131.31878750991402. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 41.07475822629763 (T) = (0 -56.09083606096519) / Math.Sqrt((637.8426091792468 / (143)) + (1.1488512902186705 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6083350906421947 = (143.2112878147157 - 56.09083606096519) / 143.2112878147157 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLowerInPlace_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 112.35865890196537 < 277.21697804286157. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 334.9229760619907 (T) = (0 -109.85151254812634) / Math.Sqrt((6.666172411681253 / (143)) + (7.4457211860150405 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6234116399381769 = (291.7018267109807 - 109.85151254812634) / 291.7018267109807 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpper_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 77.93371896754194 < 156.60312309278902. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 44.53807653849407 (T) = (0 -77.0334847988893) / Math.Sqrt((610.1568270087586 / (143)) + (1.1052312466720466 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5453347982176345 = (169.4290315091299 - 77.0334847988893) / 169.4290315091299 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLower_Bytes_Chars(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 74.66203788873301 < 83.39612686772423. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 128.10785169021165 (T) = (0 -74.36201577434818) / Math.Sqrt((0.18891213354059722 / (144)) + (0.2841197455910445 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.15174420948496756 = (87.66461320493676 - 74.36201577434818) / 87.66461320493676 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLower_Chars(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 52.01066834528202 < 68.52082428370622. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 222.18063712629336 (T) = (0 -52.35512310589293) / Math.Sqrt((0.03776989854072325 / (144)) + (0.22749440736324697 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.27319535490210045 = (72.0346567114204 - 52.35512310589293) / 72.0346567114204 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.IsValid_Chars(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.995680674495663 < 17.05696318538647. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 188.52038223450901 (T) = (0 -10.01028235891264) / Math.Sqrt((0.00339941774903863 / (145)) + (0.052577915927341456 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.4424830330258908 = (17.955116977413017 - 10.01028235891264) / 17.955116977413017 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLowerInPlace_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 55.08055968757557 < 132.76409757309688. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 41.81570295679978 (T) = (0 -56.42034793374464) / Math.Sqrt((626.246478251672 / (144)) + (1.4710741607228492 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.6085005164007907 = (144.11346706016087 - 56.42034793374464) / 144.11346706016087 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpper_Chars(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 51.93479081730784 < 68.50626573970595. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 216.23739045736465 (T) = (0 -52.19953721996468) / Math.Sqrt((0.10309541909620502 / (144)) + (0.23612809300355023 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.27738765666746673 = (72.2372620694958 - 52.19953721996468) / 72.2372620694958 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.EqualsIgnoreCase_ExactlyTheSame_Bytes_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 494.14680664567294 < 670.657493983062. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 892.6819844181531 (T) = (0 -493.0865808713479) / Math.Sqrt((3.0247416974935133 / (142)) + (1.0571938964959429 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.30093470823723656 = (705.3512549993442 - 493.0865808713479) / 705.3512549993442 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLowerInPlace_Bytes(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 26.898051138283783 < 29.494777817679203. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 22.882137410051044 (T) = (0 -27.69262923065708) / Math.Sqrt((0.5627794222701272 / (143)) + (0.5744464676385085 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.11153833143206156 = (31.16918850904764 - 27.69262923065708) / 31.16918850904764 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLower_Bytes_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 440.30260379766196 < 575.5778358341657. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 80.20660873862509 (T) = (0 -440.78595444821457) / Math.Sqrt((633.8454946940175 / (143)) + (1.1224181683141254 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2778261241691898 = (610.3598720476026 - 440.78595444821457) / 610.3598720476026 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.IsValid_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.861825453160902 < 63.97692065625021. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 69.52334377705611 (T) = (0 -12.780137067582967) / Math.Sqrt((92.43422574108729 / (143)) + (0.11451913681450859 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8143520814534502 = (68.8407237077557 - 12.780137067582967) / 68.8407237077557 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.IsValid_Bytes(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.542196549274768 < 14.733218283756855. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 357.3800248543881 (T) = (0 -8.541021661455297) / Math.Sqrt((0.04871995731219538 / (144)) + (0.0014131651761271624 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.45099647913463886 = (15.557316732673407 - 8.541021661455297) / 15.557316732673407 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLowerInPlace_Chars(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 22.627537024968692 < 39.87054215975513. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 196.61078642278127 (T) = (0 -22.64577647763986) / Math.Sqrt((0.022140969733334924 / (144)) + (0.2864502221843848 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.4609661181185182 = (42.01178671477096 - 22.64577647763986) / 42.01178671477096 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpper_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 124.36490026225263 < 289.642195906788. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 582.7870260935697 (T) = (0 -124.195912342384) / Math.Sqrt((7.759718304445372 / (143)) + (1.2900373853631029 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5940669011904977 = (305.9516770290936 - 124.195912342384) / 305.9516770290936 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.Equals_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.87791064259562 < 141.61329711978968. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 50.15366227391062 (T) = (0 -40.61392011986922) / Math.Sqrt((728.4975333918017 / (144)) + (0.39699089162557943 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7355317109717985 = (153.56820384442523 - 40.61392011986922) / 153.56820384442523 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.EqualsIgnoreCase_DifferentCase_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 137.77304446806744 < 342.4179856407427. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 787.0361447614855 (T) = (0 -138.88943218109745) / Math.Sqrt((3.8058350334552027 / (141)) + (1.5713619240975818 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.6148585458003832 = (360.6192755067903 - 138.88943218109745) / 360.6192755067903 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.FromUtf16(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.909641679014104 < 28.08159954276893. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 221.7872900686558 (T) = (0 -18.01185385784424) / Math.Sqrt((0.02330070790507213 / (142)) + (0.07663611781682847 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.39099647355958167 = (29.57594344834459 - 18.01185385784424) / 29.57594344834459 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUtf16(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 403.76940938762147 < 513.7130619261106. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 54.144192913927725 (T) = (0 -402.7327241171509) / Math.Sqrt((983.7966544276829 / (142)) + (4.440080183598039 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.2634220549334481 = (546.7618557066121 - 402.7327241171509) / 546.7618557066121 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLower_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 127.3514419861484 < 290.810382557974. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 357.2050727402074 (T) = (0 -129.67387556417663) / Math.Sqrt((6.260574071453557 / (143)) + (6.0776103620476905 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5775737871768174 = (306.97402677152274 - 129.67387556417663) / 306.97402677152274 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.Equals_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 80.04278549365738 < 278.07713106867305. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 225.10383873411982 (T) = (0 -74.48029599058707) / Math.Sqrt((1.880302005911758 / (143)) + (27.739645645112372 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.745341665882079 = (292.4714647512716 - 74.48029599058707) / 292.4714647512716 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUtf16(Size: 6) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 45.319818542458115 < 55.35034281510389. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 295.4023591910916 (T) = (0 -45.36773574204528) / Math.Sqrt((0.10209232324126656 / (140)) + (0.037014524485669005 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.22389837726569964 = (58.4559217673186 - 45.36773574204528) / 58.4559217673186 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
39.08 ns 23.06 ns 0.59 0.05 False
191.07 ns 67.28 ns 0.35 0.01 True
265.15 ns 38.15 ns 0.14 0.01 True
27.42 ns 14.19 ns 0.52 0.02 True
30.19 ns 16.23 ns 0.54 0.03 True
1.26 μs 931.71 ns 0.74 0.06 True
30.24 ns 16.20 ns 0.54 0.01 True
168.35 ns 52.77 ns 0.31 0.02 True
44.95 ns 27.50 ns 0.61 0.03 False
172.40 ns 48.46 ns 0.28 0.01 True
29.91 ns 14.47 ns 0.48 0.02 True
183.06 ns 62.90 ns 0.34 0.02 True
27.04 ns 20.54 ns 0.76 0.02 True
263.18 ns 41.12 ns 0.16 0.02 True
63.65 ns 46.70 ns 0.73 0.02 False
154.97 ns 45.81 ns 0.30 0.12 True
58.85 ns 17.94 ns 0.30 0.02 True
502.99 ns 51.59 ns 0.10 0.05 True
216.90 ns 92.21 ns 0.43 0.01 True
497.85 ns 62.13 ns 0.12 0.02 True
48.30 ns 34.35 ns 0.71 0.02 False
67.81 ns 50.79 ns 0.75 0.02 False
37.56 ns 19.06 ns 0.51 0.05 False
59.48 ns 17.42 ns 0.29 0.01 True
211.68 ns 87.72 ns 0.41 0.01 True
35.23 ns 21.09 ns 0.60 0.05 True
50.66 ns 22.50 ns 0.44 0.01 True
93.79 ns 70.20 ns 0.75 0.02 True
32.17 ns 13.34 ns 0.41 0.09 False
27.24 ns 20.32 ns 0.75 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Byte>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Span<Byte>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Byte>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Span<Byte>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.05515664492993 < 37.219700071165704. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 37.99636191928033 (T) = (0 -23.58117940631509) / Math.Sqrt((23.331123937014738 / (142)) + (0.6272970332399791 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.4094882601734633 = (39.933464173366104 - 23.58117940631509) / 39.933464173366104 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyThreeValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 67.28340938792722 < 181.53012020630396. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 273.58829210580456 (T) = (0 -67.79773354505353) / Math.Sqrt((29.408547218118283 / (140)) + (0.1559919910441797 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.6518413891304554 = (194.73231862835476 - 67.79773354505353) / 194.73231862835476 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.StartsWith(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 38.15212349405252 < 253.4188829787367. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1577.0355769240903 (T) = (0 -38.248711252501955) / Math.Sqrt((2.6323944182581585 / (144)) + (0.07993702394462082 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.8564689440906099 = (266.4838700597871 - 38.248711252501955) / 266.4838700597871 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.Reverse(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.189572699178603 < 25.84329175899091. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 294.55383648784243 (T) = (0 -14.109759876921167) / Math.Sqrt((0.05192601827334431 / (142)) + (0.0494311370403906 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.48366207931092664 = (27.326600103457704 - 14.109759876921167) / 27.326600103457704 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceCompareToDifferent(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.22564682731665 < 28.748874042069517. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 410.3263245354269 (T) = (0 -16.214417780124883) / Math.Sqrt((0.10043737048614226 / (144)) + (0.01428355921363483 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.4643636315128931 = (30.27131601597955 - 16.214417780124883) / 30.27131601597955 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.Reverse(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 931.7085537243836 < 1.2063550996144088. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 126.24043102713404 (T) = (0 -935.3026728093586) / Math.Sqrt((777.8528348270825 / (142)) + (61.83967032842406 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.27039358979034106 = (1281.9277074890158 - 935.3026728093586) / 1281.9277074890158 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceCompareToDifferent(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.20334778895548 < 28.684368026140675. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 583.4454691245229 (T) = (0 -16.2439158453886) / Math.Sqrt((0.030433983854719692 / (143)) + (0.010925373154682758 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.4631673499983517 = (30.258807554530676 - 16.2439158453886) / 30.258807554530676 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 52.76721717911452 < 159.47403780485007. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 266.6507630107079 (T) = (0 -53.79869534615275) / Math.Sqrt((22.95511421077208 / (143)) + (0.7291999186740267 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6806012827430384 = (168.4374183095758 - 53.79869534615275) / 168.4374183095758 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyTwoValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 27.497891433709597 < 42.72832393123211. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 44.06968987502899 (T) = (0 -28.34891553231007) / Math.Sqrt((27.930125407191163 / (143)) + (0.1810500660794359 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.41091831052138766 = (48.123912249591875 - 28.34891553231007) / 48.123912249591875 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfValue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 48.45574855448887 < 163.21794328335767. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 243.7645772094015 (T) = (0 -47.52567922749872) / Math.Sqrt((29.890711884983887 / (143)) + (1.9078712850966912 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7281185180425859 = (174.8029284132814 - 47.52567922749872) / 174.8029284132814 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceEqual(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.473064543611114 < 28.416853670535282. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 351.1418724242768 (T) = (0 -14.52495261557488) / Math.Sqrt((0.01461213551017693 / (141)) + (0.05460583932183707 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.5146465143638431 = (29.926544354650925 - 14.52495261557488) / 29.926544354650925 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyTwoValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 62.89739126621978 < 174.198493552442. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 239.2551227880411 (T) = (0 -63.69170423337855) / Math.Sqrt((31.101311844470356 / (145)) + (1.5836450627159997 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.6601015592825318 = (187.38451432414965 - 63.69170423337855) / 187.38451432414965 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.EndsWith(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.53508056206343 < 25.751480170433855. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 282.19844545467464 (T) = (0 -20.34864532497374) / Math.Sqrt((0.06547459007068648 / (145)) + (0.0037056584941717865 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.2495689646991369 = (27.115943194987338 - 20.34864532497374) / 27.115943194987338 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.EndsWith(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 41.121114013587885 < 252.9629112479959. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 986.5917272683416 (T) = (0 -41.13286331882112) / Math.Sqrt((6.82593269248 / (145)) + (0.14988603432810096 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.8455192060336187 = (266.26522470989266 - 41.13286331882112) / 266.26522470989266 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 46.70223875093339 < 60.51800164576983. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 41.730972093795664 (T) = (0 -47.21390224034281) / Math.Sqrt((30.805752303624853 / (142)) + (0.27429157044783825 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.29590663464353245 = (67.05630895476399 - 47.21390224034281) / 67.05630895476399 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.LastIndexOfValue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 45.806600019817814 < 147.24180423630355. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 117.84822091289071 (T) = (0 -40.908046969085234) / Math.Sqrt((21.517075148705455 / (143)) + (23.948904394204177 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7372597034286582 = (155.69765088537716 - 40.908046969085234) / 155.69765088537716 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceCompareTo(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.939693799124836 < 55.66255789082227. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1105.793949656637 (T) = (0 -17.949071279013637) / Math.Sqrt((0.17356308189299327 / (142)) + (0.0037838951656274118 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.6934639902731868 = (58.55452772093551 - 17.949071279013637) / 58.55452772093551 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceEqual(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 51.58718408145201 < 485.10958694603823. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 713.4794029576038 (T) = (0 -52.02850653727319) / Math.Sqrt((55.22736305314771 / (143)) + (0.7021782843574452 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8977148945717529 = (508.66161128191936 - 52.02850653727319) / 508.66161128191936 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyFiveValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 92.21213131100555 < 206.7338969145944. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 257.562240534928 (T) = (0 -92.2477450760195) / Math.Sqrt((34.96337553452385 / (143)) + (0.2855002541136816 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5845801416896249 = (222.05906441549527 - 92.2477450760195) / 222.05906441549527 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceCompareTo(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 62.133804926109036 < 462.7973425834608. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 858.3460930145775 (T) = (0 -62.15386969418134) / Math.Sqrt((32.80315229402472 / (143)) + (0.590801470337414 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8732954456539251 = (490.54171742254186 - 62.15386969418134) / 490.54171742254186 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyThreeValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 34.353325095229465 < 45.989464229327815. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 42.57959913590797 (T) = (0 -32.188440159799605) / Math.Sqrt((28.363167714591434 / (142)) + (0.29063256077596267 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.3770898271643781 = (51.67428878753233 - 32.188440159799605) / 51.67428878753233 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyFiveValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 50.79064963567287 < 63.92945161106925. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.92004565372363 (T) = (0 -51.15810486770395) / Math.Sqrt((32.35466929244596 / (140)) + (0.6289754579174855 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.27639103619819005 = (70.69855049738673 - 51.15810486770395) / 70.69855049738673 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfValue(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.056105842455324 < 35.5979786112576. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 43.39860498615257 (T) = (0 -19.306861074466887) / Math.Sqrt((31.171778405607647 / (145)) + (0.8844488402433638 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.5263789274455495 = (40.76436246879038 - 19.306861074466887) / 40.76436246879038 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceEqual(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.416939108990743 < 56.495621082376296. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 849.4809038930528 (T) = (0 -17.371447841604684) / Math.Sqrt((0.1961548328441272 / (143)) + (0.03277221980241606 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7082370409079857 = (59.53959301641916 - 17.371447841604684) / 59.53959301641916 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 87.71640352799878 < 201.44279088704693. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 232.17867390383097 (T) = (0 -86.58557038872597) / Math.Sqrt((35.6794283286186 / (142)) + (1.8077293334557933 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.5994634878816897 = (216.17397607724305 - 86.58557038872597) / 216.17397607724305 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.StartsWith(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.091155230756154 < 33.51338467698041. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 122.4722871759286 (T) = (0 -21.2382261304959) / Math.Sqrt((0.2920181989999586 / (144)) + (0.3053600020569734 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.38916859382167324 = (34.769374848246734 - 21.2382261304959) / 34.769374848246734 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.EndsWith(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 22.502373225990706 < 47.57608517977956. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 724.8766164644861 (T) = (0 -22.632159036298198) / Math.Sqrt((0.09866494351410214 / (143)) + (0.022964581403001123 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5499354169275685 = (50.28646973684636 - 22.632159036298198) / 50.28646973684636 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.Reverse(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 70.20294201457611 < 89.10513159182842. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 105.05277744508854 (T) = (0 -70.59029509965907) / Math.Sqrt((1.977244187245721 / (142)) + (1.1979277325376065 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.25670638970698145 = (94.96959764235189 - 70.59029509965907) / 94.96959764235189 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.LastIndexOfValue(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.34253355450924 < 30.503760267406825. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 39.90840082343987 (T) = (0 -14.09943756157589) / Math.Sqrt((21.849173060215996 / (143)) + (2.093544879990105 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5718025136709873 = (32.92741786611599 - 14.09943756157589) / 32.92741786611599 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.StartsWith(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.32091376492907 < 25.931379859201495. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 476.29679934229375 (T) = (0 -20.35399626104093) / Math.Sqrt((0.005606912373914289 / (143)) + (0.005182924920634873 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.25411943749754234 = (27.2885462958741 - 20.35399626104093) / 27.2885462958741 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Boolean

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
26.09 ns 21.89 ns 0.84 0.02 True
22.48 ns 17.45 ns 0.78 0.01 True
21.06 ns 16.32 ns 0.77 0.01 True
22.77 ns 17.50 ns 0.77 0.01 True
22.10 ns 17.36 ns 0.79 0.01 True
71.37 ns 66.75 ns 0.94 0.01 True
25.83 ns 21.86 ns 0.85 0.05 True
21.08 ns 16.37 ns 0.78 0.02 True
22.14 ns 17.18 ns 0.78 0.01 True

graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Boolean*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Boolean* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Boolean*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Boolean* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Boolean.Parse(value: "false") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.891885184448448 < 24.120671573437942. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.2464139127319 (T) = (0 -22.19842926737038) / Math.Sqrt((0.57184003989016 / (143)) + (0.1491676393924633 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.12094770017873988 = (25.252683227020782 - 22.19842926737038) / 25.252683227020782 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Boolean.TryParse(value: "False") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.45274994049213 < 21.60020962627509. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 131.72255174079038 (T) = (0 -17.561802462945252) / Math.Sqrt((0.007351458998764926 / (142)) + (0.04516050439525646 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.22837903971372342 = (22.7596233990711 - 17.561802462945252) / 22.7596233990711 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Boolean.TryParse(value: "TRUE") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.320163104282457 < 20.117950747736337. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 300.215575618623 (T) = (0 -16.303713549024142) / Math.Sqrt((0.010351767886168912 / (142)) + (0.0054935460829906016 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.22758053900680533 = (21.107331407808463 - 16.303713549024142) / 21.107331407808463 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Boolean.TryParse(value: "false") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.50353628876818 < 21.595481974091232. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 113.4298769502238 (T) = (0 -17.572744677187153) / Math.Sqrt((0.010354802565978549 / (143)) + (0.06076612199511315 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.22818962584590013 = (22.768215180376124 - 17.572744677187153) / 22.768215180376124 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Boolean.Parse(value: "true") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.361561159941605 < 20.96344606985449. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 199.83656181185967 (T) = (0 -17.06793592194474) / Math.Sqrt((0.009912744839464964 / (144)) + (0.017087997986745974 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.22829839661694548 = (22.117274147313932 - 17.06793592194474) / 22.117274147313932 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Boolean.TryParse(value: "Bogus") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 66.75132627006018 < 68.28144655120562. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 95.79174219743634 (T) = (0 -66.59719372380984) / Math.Sqrt((0.26136986721802186 / (141)) + (0.02933652496367804 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.07109775627725423 = (71.69451271525558 - 66.59719372380984) / 71.69451271525558 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Boolean.Parse(value: "False") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.864120854991757 < 23.89792504821581. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 21.978424015699662 (T) = (0 -22.21975696555075) / Math.Sqrt((0.2782590745818561 / (143)) + (0.4782032589936078 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.1168308315860692 = (25.159117596297946 - 22.21975696555075) / 25.159117596297946 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Boolean.TryParse(value: "true") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.37010399343108 < 20.026239027773684. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 170.65516989082198 (T) = (0 -16.299073026730746) / Math.Sqrt((0.07323414540775067 / (143)) + (0.008540800752363402 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.22812938622348555 = (21.116327964585448 - 16.299073026730746) / 21.116327964585448 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Boolean.Parse(value: "TRUE") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.17644871188685 < 20.9782793912265. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 238.03652948435283 (T) = (0 -17.082548897487854) / Math.Sqrt((0.007002117817608793 / (143)) + (0.012013822747604633 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2280416607703131 = (22.128848189571986 - 17.082548897487854) / 22.128848189571986 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
169.26 ns 49.83 ns 0.29 0.18 True
1.51 μs 1.07 μs 0.71 0.01 True
1.87 μs 1.42 μs 0.76 0.01 True
1.23 μs 1.12 μs 0.91 0.01 True
297.89 ns 76.99 ns 0.26 0.01 False
601.54 ns 60.30 ns 0.10 0.01 True
67.33 ns 47.57 ns 0.71 0.03 True
597.92 ns 62.49 ns 0.10 0.01 True
1.52 μs 1.07 μs 0.71 0.01 True
602.79 ns 61.94 ns 0.10 0.01 True
397.30 ns 51.62 ns 0.13 0.02 True

graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Tests.Perf_BitArray* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Tests.Perf_BitArray* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 49.825286008223706 < 160.72654524478435. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 232.87323351668985 (T) = (0 -47.1300962035607) / Math.Sqrt((9.886506273866507 / (143)) + (6.3242713875839 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7233224193435667 = (170.34302559586456 - 47.1300962035607) / 170.34302559586456 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayByteArrayCtor(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.0664310539559463 < 1.4502319612690064. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 528.3047062413843 (T) = (0 -1068.475871804388) / Math.Sqrt((35.74014375105704 / (145)) + (15.417126104999753 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.30126557785761926 = (1529.15877326946 - 1068.475871804388) / 1529.15877326946 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArraySetLengthGrow(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.4231532323484095 < 1.7763800375356722. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 210.31930260380489 (T) = (0 -1422.2616840934593) / Math.Sqrt((134.48458357858598 / (146)) + (106.69048216311224 / (30))) is greater than 1.9736914397558152 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (146) + (30) - 2, .975) and 0.2383318503831624 = (1867.2983566516964 - 1422.2616840934593) / 1867.2983566516964 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayCopyToByteArray(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.1175538269083787 < 1.170411414662204. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 131.17434708851255 (T) = (0 -1116.7322222212183) / Math.Sqrt((39.918138822410484 / (144)) + (10.951639807045092 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.08603699911719537 = (1221.8571442635612 - 1116.7322222212183) / 1221.8571442635612 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 76.99073452556965 < 284.04346258989085. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 28.76864900137926 (T) = (0 -76.95123368330843) / Math.Sqrt((9441.117247077766 / (144)) + (1.0197112822868895 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7517339706548384 = (309.9547444581069 - 76.95123368330843) / 309.9547444581069 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 60.300803229424794 < 571.7947327432939. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1939.8131539755402 (T) = (0 -60.49458711227335) / Math.Sqrt((9.695495345364312 / (143)) + (0.23596513010215084 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8981724135840854 = (594.0883923653396 - 60.49458711227335) / 594.0883923653396 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayLengthValueCtor(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 47.56578899568879 < 64.09856487919049. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 117.22377677810597 (T) = (0 -48.55504327308309) / Math.Sqrt((1.103504379013532 / (144)) + (0.49442046199587514 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2727975527224763 = (66.76963678389951 - 48.55504327308309) / 66.76963678389951 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 62.48606132215225 < 571.3084196890401. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1794.0084675258877 (T) = (0 -62.471524749585015) / Math.Sqrt((10.4998136423152 / (143)) + (0.43324120971638846 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8948745744118134 = (594.2570448590428 - 62.471524749585015) / 594.2570448590428 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArraySetLengthShrink(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.0743368761917933 < 1.4530775170226786. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 411.3864049542946 (T) = (0 -1073.5214331696952) / Math.Sqrt((38.12411436234793 / (144)) + (29.159205058481074 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2988178390504661 = (1531.0164647029 - 1073.5214331696952) / 1531.0164647029 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 61.93848865941009 < 572.2776186083188. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1774.0841584357506 (T) = (0 -61.57320701147234) / Math.Sqrt((10.75181085646805 / (144)) + (0.4664356432561512 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.8964161250863127 = (594.4284963540814 - 61.57320701147234) / 594.4284963540814 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 51.623245521375445 < 377.31454556364827. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1183.6587903883376 (T) = (0 -51.61983511996172) / Math.Sqrt((9.014638450855657 / (139)) + (0.5126061362927591 / (30))) is greater than 1.974270957027852 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (139) + (30) - 2, .975) and 0.8677922664729482 = (390.44489866698626 - 51.61983511996172) / 390.44489866698626 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
12.63 ns 9.21 ns 0.73 0.01 True
12.98 ns 9.32 ns 0.72 0.01 True
85.12 ns 79.37 ns 0.93 0.01 True
12.73 ns 8.77 ns 0.69 0.01 True
13.70 ns 10.19 ns 0.74 0.01 True
78.56 ns 66.45 ns 0.85 0.01 True
161.77 ns 153.01 ns 0.95 0.00 False
13.77 ns 10.15 ns 0.74 0.03 True
137.23 ns 129.00 ns 0.94 0.02 True
161.42 ns 150.87 ns 0.93 0.01 False

graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Buffers.Text.Tests.Utf8ParserTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Buffers.Text.Tests.Utf8ParserTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseBool(value: true) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.209498865123484 < 12.012279912858153. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 156.8238377152297 (T) = (0 -9.264999553545346) / Math.Sqrt((0.0071602697233088776 / (144)) + (0.01237557497980599 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.26681738687697426 = (12.636687487828777 - 9.264999553545346) / 12.636687487828777 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseBool(value: True ) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.320091450041353 < 12.481966123180628. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 293.42693142545386 (T) = (0 -9.29683512426099) / Math.Sqrt((0.0072469860324487125 / (144)) + (0.003432111214473804 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2883016603524928 = (13.062887190190107 - 9.29683512426099) / 13.062887190190107 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseSingle(value: 12345) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 79.37061873139459 < 79.47168447383284. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 30.758042351752156 (T) = (0 -78.65721480642682) / Math.Sqrt((1.6401468994351525 / (144)) + (0.7179797895058608 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.06846159747450299 = (84.43797334943892 - 78.65721480642682) / 84.43797334943892 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseBool(value: TRUE) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.766163958227452 < 12.011389952539568. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 172.06260443859196 (T) = (0 -9.214240968934705) / Math.Sqrt((0.01598259728669293 / (142)) + (0.0085822053351638 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.2715774341721274 = (12.649581988804071 - 9.214240968934705) / 12.649581988804071 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseBool(value: false) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 10.186628138746103 < 13.020361285147237. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 237.00750087750887 (T) = (0 -10.168856496564315) / Math.Sqrt((0.014348918282909807 / (144)) + (0.003925919567833149 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.26137238156475817 = (13.767230256169816 - 10.168856496564315) / 13.767230256169816 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseDouble(value: 12345) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 66.45023198791641 < 71.88299335858294. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.725194411801326 (T) = (0 -67.24610324269601) / Math.Sqrt((1.9229051575590206 / (144)) + (2.055055315141453 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.12872330977528978 = (77.18111134747863 - 67.24610324269601) / 77.18111134747863 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseDouble(value: -1.7976931348623157e+308) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 153.00685171092684 < 159.00393406057498. IsChangePoint: Marked as a change because one of 9/13/2023 5:14:37 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 26.614141404232704 (T) = (0 -153.2676224356982) / Math.Sqrt((7.752393991547663 / (144)) + (2.078423635539399 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.05742945115908506 = (162.6059955131978 - 153.2676224356982) / 162.6059955131978 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseBool(value: False) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 10.150117090043548 < 13.030112771907374. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 111.0317294003088 (T) = (0 -10.13132688496257) / Math.Sqrt((0.008242391256006247 / (145)) + (0.030263489701026377 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.26348906537719646 = (13.755840420958888 - 10.13132688496257) / 13.755840420958888 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseSingle(value: -3.4028235E+38) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 129.0020033176139 < 130.34088346802312. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 21.79485739154947 (T) = (0 -129.0791477769943) / Math.Sqrt((13.297811930305302 / (146)) + (1.0236858312852808 / (30))) is greater than 1.9736914397558152 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (146) + (30) - 2, .975) and 0.05637739100804002 = (136.79107150143972 - 129.0791477769943) / 136.79107150143972 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Text.Tests.Utf8ParserTests.TryParseDouble(value: 1.7976931348623157e+308) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 150.87209113106718 < 157.48603785005366. IsChangePoint: Marked as a change because one of 9/13/2023 5:14:37 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 26.629748792492826 (T) = (0 -151.68103692010564) / Math.Sqrt((7.703380424098389 / (142)) + (2.0460456634332522 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.05787925698078237 = (160.99957255373965 - 151.68103692010564) / 160.99957255373965 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Span.IndexerBench

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.04 μs 1.36 μs 0.67 0.07 False
2.03 μs 1.26 μs 0.62 0.10 False
1.59 μs 1.26 μs 0.79 0.01 False
1.36 μs 914.59 ns 0.67 0.01 False
13.53 μs 9.06 μs 0.67 0.01 False
1.37 μs 911.69 ns 0.67 0.01 False
790.05 ns 684.41 ns 0.87 0.02 False
1.59 μs 913.88 ns 0.58 0.01 False

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Span.IndexerBench*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Span.IndexerBench* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Span.IndexerBench*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Span.IndexerBench* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### Span.IndexerBench.SameIndex1(length: 1024) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.3573947011207776 < 1.9293533876576021. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 11.031227847112046 (T) = (0 -1359.743737604312) / Math.Sqrt((99794.03326265945 / (145)) + (4.919371266467875 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.17550011486919007 = (1649.1739563900408 - 1359.743737604312) / 1649.1739563900408 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Span.IndexerBench.CoveredIndex2(length: 1024) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.2603982300661771 < 1.9329235114004417. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 14.795569040347146 (T) = (0 -1253.8785867252266) / Math.Sqrt((96158.94077381628 / (143)) + (6.704525629444136 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.23432541385867164 = (1637.6129094792568 - 1253.8785867252266) / 1637.6129094792568 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Span.IndexerBench.WriteViaIndexer1(length: 1024) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.2633436722680433 < 1.5096252334436573. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.940981309277834 (T) = (0 -1263.2098201632239) / Math.Sqrt((27006.160538402182 / (144)) + (3.7208576476935145 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.1060668260595887 = (1413.092003952667 - 1263.2098201632239) / 1413.092003952667 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Span.IndexerBench.Indexer1(length: 1024) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 914.5949399185253 < 1.2977493268374376. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.886755855808222 (T) = (0 -912.8808550632912) / Math.Sqrt((51251.26935271342 / (142)) + (6.134687736636994 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.18475758845852996 = (1119.7661482517883 - 912.8808550632912) / 1119.7661482517883 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Span.IndexerBench.Indexer4(length: 1024) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.055813213387502 < 12.849066838197714. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.80056287265225 (T) = (0 -9052.212616492387) / Math.Sqrt((5033841.334819268 / (140)) + (256.33332661886357 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.18451960190617966 = (11100.4662253708 - 9052.212616492387) / 11100.4662253708 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Span.IndexerBench.ReadOnlyIndexer1(length: 1024) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 911.6939619187565 < 1.2969112306619606. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.996197512367068 (T) = (0 -912.1046279572969) / Math.Sqrt((50836.43623935103 / (141)) + (3.9184364401076874 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.18630208687185967 = (1120.9376517273427 - 912.1046279572969) / 1120.9376517273427 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Span.IndexerBench.Ref(length: 1024) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 684.4133548190467 < 772.2418680570164. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 11.981629952996 (T) = (0 -683.2124208457824) / Math.Sqrt((3965.7220873181227 / (143)) + (2.587381637578457 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.08466565904752206 = (746.4075040982792 - 683.2124208457824) / 746.4075040982792 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Span.IndexerBench.Indexer6(length: 1024) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 913.87978056694 < 1.5040538140119586. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.955860389362549 (T) = (0 -912.3274134949876) / Math.Sqrt((113287.08428569135 / (141)) + (3.0458943795641735 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.2539602798555767 = (1222.8938873634947 - 912.3274134949876) / 1222.8938873634947 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Document.Tests.Perf_EnumerateArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.05 ms 663.03 μs 0.63 0.01 True
9.75 μs 7.72 μs 0.79 0.03 True
9.91 μs 7.34 μs 0.74 0.01 True
9.76 μs 7.69 μs 0.79 0.01 True
30.33 μs 28.16 μs 0.93 0.01 True
10.19 μs 7.52 μs 0.74 0.01 True
3.17 ms 2.63 ms 0.83 0.02 True
9.93 μs 7.35 μs 0.74 0.01 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Document.Tests.Perf_EnumerateArray* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Document.Tests.Perf_EnumerateArray* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 663.0346875 < 0.995039069813988. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1159.128445722121 (T) = (0 -666580.9862826808) / Math.Sqrt((4135202.882837427 / (141)) + (2341700.2123198644 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.3629903915668272 = (1046422.1849372783 - 666580.9862826808) / 1046422.1849372783 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.716117719882071 < 9.26188916884129. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 81.342645585406 (T) = (0 -7739.604171723427) / Math.Sqrt((4691.230954799888 / (141)) + (17777.426767393965 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.20818866931734845 = (9774.555972886636 - 7739.604171723427) / 9774.555972886636 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.340511100994674 < 9.40736080085889. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 454.8134576228795 (T) = (0 -7341.417744312911) / Math.Sqrt((661.9247053971669 / (142)) + (828.464377475536 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.26033585460916003 = (9925.339480168654 - 7341.417744312911) / 9925.339480168654 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.691760406393819 < 9.239795780209347. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 305.8048215735781 (T) = (0 -7701.994118479456) / Math.Sqrt((1692.7299564177943 / (144)) + (1019.8329189879616 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2117019172473428 = (9770.408284623592 - 7701.994118479456) / 9770.408284623592 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.Parse(TestCase: ArrayOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 28.155623261967346 < 28.837617299791297. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 58.00589629062608 (T) = (0 -28114.73049256639) / Math.Sqrt((62599.805509631995 / (140)) + (14213.864715027139 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.05892216715772051 = (29875.032129545754 - 28114.73049256639) / 29875.032129545754 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.516033161420307 < 9.671898901593643. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 300.91486057739525 (T) = (0 -7533.318485747664) / Math.Sqrt((1854.7981519338766 / (142)) + (1839.2551886476604 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.2562151842747302 = (10128.35745833541 - 7533.318485747664) / 10128.35745833541 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.Parse(TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.6326200291666666 < 3.0160600040649035. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 73.4819744169172 (T) = (0 -2630492.0362310493) / Math.Sqrt((4809024521.826833 / (142)) + (66588600.51275697 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.14369498878925807 = (3071910.1275744713 - 2630492.0362310493) / 3071910.1275744713 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.345476324237561 < 9.41562851441249. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 491.1856749416399 (T) = (0 -7335.97082215496) / Math.Sqrt((656.6682466332177 / (144)) + (697.7776101697356 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2609850319363542 = (9926.687738648303 - 7335.97082215496) / 9926.687738648303 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.Concurrent.IsEmpty<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
400.16 ns 361.08 ns 0.90 0.03 True
36.42 ns 26.95 ns 0.74 0.03 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Concurrent.IsEmpty<Int32>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Concurrent.IsEmpty<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Concurrent.IsEmpty<Int32>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Concurrent.IsEmpty<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.Concurrent.IsEmpty<Int32>.Dictionary(Size: 0) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 361.0756985395861 < 381.3216232669182. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 31.15903159592401 (T) = (0 -354.2958223830147) / Math.Sqrt((16.194076521790436 / (142)) + (63.49738839126539 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.11610044054005422 = (400.83267220936966 - 354.2958223830147) / 400.83267220936966 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Concurrent.IsEmpty<Int32>.Dictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 26.951577576744388 < 34.7550310169021. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 46.684928125123555 (T) = (0 -26.7748107213023) / Math.Sqrt((9.966673501183266 / (143)) + (0.14045566884130697 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.322275991476461 = (39.50695325023644 - 26.7748107213023) / 39.50695325023644 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Document.Tests.Perf_DocumentParse

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
6.11 μs 5.00 μs 0.82 0.01 True
4.15 μs 3.70 μs 0.89 0.02 False
6.51 μs 5.40 μs 0.83 0.01 True
988.51 ns 848.17 ns 0.86 0.01 True
3.04 ms 2.63 ms 0.87 0.02 True
2.97 μs 2.64 μs 0.89 0.03 False
12.54 μs 9.81 μs 0.78 0.01 True
34.47 ms 23.53 ms 0.68 0.01 True
12.98 μs 10.06 μs 0.77 0.02 True
523.28 ns 484.14 ns 0.93 0.02 True
3.37 ms 3.01 ms 0.89 0.02 True
956.70 ns 825.84 ns 0.86 0.01 True
34.05 ms 23.21 ms 0.68 0.01 True
3.79 μs 3.31 μs 0.87 0.01 True
2.55 μs 2.27 μs 0.89 0.02 True
551.96 ns 510.78 ns 0.93 0.01 False

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Document.Tests.Perf_DocumentParse* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Document.Tests.Perf_DocumentParse* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: BasicJson) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 4.998422604166668 < 5.768496536625769. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 123.87484849081822 (T) = (0 -5040.625706962413) / Math.Sqrt((2032.333126760396 / (141)) + (1540.5621762752803 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.16617752288255047 = (6045.202480494426 - 5040.625706962413) / 6045.202480494426 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.696451578342371 < 3.949357844999625. IsChangePoint: Marked as a change because one of 9/18/2023 9:14:49 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 30.474967459573246 (T) = (0 -3683.7050220384463) / Math.Sqrt((13386.050703439252 / (143)) + (888.3952546135565 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.0841096588624884 = (4021.993525407618 - 3683.7050220384463) / 4021.993525407618 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: BasicJson) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.396134720791854 < 6.152307125033752. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 116.80119468637136 (T) = (0 -5409.437941513655) / Math.Sqrt((4569.371570166796 / (144)) + (1173.141145270971 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.15378208873500399 = (6392.488116243229 - 5409.437941513655) / 6392.488116243229 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 848.1738327762798 < 938.8122620634705. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 70.01367633335067 (T) = (0 -847.441809913204) / Math.Sqrt((158.61479160753865 / (143)) + (64.08824248413816 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.12955471205078542 = (973.5727467831928 - 847.441809913204) / 973.5727467831928 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.6325282371794874 < 2.946272421927828. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 87.60682855382045 (T) = (0 -2637049.850402167) / Math.Sqrt((2654355070.390835 / (144)) + (101149271.33506604 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.1342961772777376 = (3046134.002400256 - 2637049.850402167) / 3046134.002400256 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: BasicJson) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.637174397960048 < 2.812996339254331. IsChangePoint: Marked as a change because one of 9/7/2023 11:26:14 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 28.928641245778643 (T) = (0 -2652.9284300078866) / Math.Sqrt((5314.7562622819205 / (144)) + (530.0348262225457 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.07455128621040502 = (2866.640139510791 - 2652.9284300078866) / 2866.640139510791 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.810412243509212 < 11.850850520874767. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 220.55354514682395 (T) = (0 -9715.095379519107) / Math.Sqrt((9526.302778157256 / (143)) + (2382.5294858107877 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.21528261156604656 = (12380.374798253612 - 9715.095379519107) / 12380.374798253612 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.53179425 < 32.67922977445055. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 725.6513207537589 (T) = (0 -23508306.508265566) / Math.Sqrt((13513559921.212769 / (145)) + (3901547587.5323706 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.3156367933915853 = (34350628.849216856 - 23508306.508265566) / 34350628.849216856 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 10.056256066552686 < 12.264714147991045. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 177.07957698587384 (T) = (0 -10123.760802146828) / Math.Sqrt((18804.671059142096 / (143)) + (2612.888450410566 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.20547415488121254 = (12741.889850837075 - 10123.760802146828) / 12741.889850837075 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 484.1351014416205 < 503.25098102242754. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 35.36003010577305 (T) = (0 -490.246384836354) / Math.Sqrt((48.228783951227435 / (144)) + (17.018930991609697 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.06411738280394193 = (523.8331985534168 - 490.246384836354) / 523.8331985534168 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.0053526034722227 < 3.3104383719523804. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.028592113758314 (T) = (0 -2969236.862683276) / Math.Sqrt((3282590957.7875957 / (145)) + (555714407.0978829 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.1097062403258555 = (3335120.3806820386 - 2969236.862683276) / 3335120.3806820386 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 825.8402683760237 < 910.7864266208178. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 73.4321488587992 (T) = (0 -828.2791930587555) / Math.Sqrt((66.44791986528142 / (143)) + (71.88371516948438 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.13039863297807006 = (952.4814753860289 - 828.2791930587555) / 952.4814753860289 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.205903287878787 < 32.42308198225275. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 765.6920042226384 (T) = (0 -23178890.36282717) / Math.Sqrt((11828796203.700022 / (143)) + (3572205227.3028502 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3193854171925585 = (34055823.880848154 - 23178890.36282717) / 34055823.880848154 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.309509223300971 < 3.5442248075360587. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 60.894963673717946 (T) = (0 -3290.0430102175906) / Math.Sqrt((3197.3177597169256 / (144)) + (468.7877212712073 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.10220545776007431 = (3664.5834379981816 - 3290.0430102175906) / 3664.5834379981816 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: BasicJson) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.2732939075579424 < 2.4301844382857567. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.71322172240348 (T) = (0 -2291.577286224515) / Math.Sqrt((1351.5454970731294 / (143)) + (744.9165084378766 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.09001060656773349 = (2518.246149640517 - 2291.577286224515) / 2518.246149640517 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 510.78350175386834 < 533.273186285713. IsChangePoint: Marked as a change because one of 9/19/2023 8:08:41 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 27.415462220253957 (T) = (0 -514.194985156961) / Math.Sqrt((80.23138323085885 / (143)) + (18.92062042528504 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.05500343494217766 = (544.1236552277823 - 514.194985156961) / 544.1236552277823 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.Perf_Directory

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.83 μs 1.62 μs 0.88 0.01 False
1.73 μs 1.59 μs 0.92 0.01 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Directory*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_Directory* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Directory*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_Directory* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_Directory.GetCurrentDirectory #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.6196090272736698 < 1.7397890382346417. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.610688013120765 (T) = (0 -1619.7306399273734) / Math.Sqrt((4906.597138643371 / (142)) + (314.1765701229238 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.12540285671603896 = (1851.9733941109903 - 1619.7306399273734) / 1851.9733941109903 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_Directory.Exists #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.5943475925265091 < 1.6420145533336254. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 25.974956071434192 (T) = (0 -1605.7906996817744) / Math.Sqrt((535.7202014302183 / (142)) + (337.74753696711514 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.0590123250636005 = (1706.49493341165 - 1605.7906996817744) / 1706.49493341165 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
29.77 ns 23.23 ns 0.78 0.04 True
118.15 ns 106.12 ns 0.90 0.02 True
305.82 ns 175.89 ns 0.58 0.01 True
1.84 μs 501.81 ns 0.27 0.10 True
244.64 ns 110.21 ns 0.45 0.03 True
98.15 ns 43.68 ns 0.45 0.01 True
44.79 ns 39.45 ns 0.88 0.06 True
967.66 ns 887.77 ns 0.92 0.03 True
114.03 ns 69.53 ns 0.61 0.03 True
299.24 ns 178.15 ns 0.60 0.02 True
111.65 ns 101.55 ns 0.91 0.01 True
307.76 ns 182.23 ns 0.59 0.01 True
280.93 ns 202.73 ns 0.72 0.01 True
33.44 ns 30.14 ns 0.90 0.11 False
94.23 ns 68.20 ns 0.72 0.06 True
110.46 ns 99.61 ns 0.90 0.01 True
110.60 ns 100.09 ns 0.90 0.01 True
116.51 ns 59.08 ns 0.51 0.02 True
193.79 ns 95.96 ns 0.50 0.04 True
231.61 ns 171.03 ns 0.74 0.02 True
48.05 ns 24.97 ns 0.52 0.08 True
298.82 ns 172.47 ns 0.58 0.01 True
120.73 ns 106.99 ns 0.89 0.02 True
104.86 ns 49.76 ns 0.47 0.03 True
111.09 ns 101.99 ns 0.92 0.01 True
1.73 μs 643.40 ns 0.37 0.04 True
92.77 ns 66.37 ns 0.72 0.10 True
727.85 ns 686.55 ns 0.94 0.02 False
119.38 ns 106.79 ns 0.89 0.02 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_String* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_String* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_String.CtorCharCount(size: 10) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.233730198389516 < 28.00572755125013. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 62.488080537624604 (T) = (0 -22.68251374557637) / Math.Sqrt((0.6343557412237295 / (144)) + (0.1596989985650027 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.21366663944866082 = (28.845925765724196 - 22.68251374557637) / 28.845925765724196 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLower(s: "TeSt") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 106.12176765719207 < 112.37070112886536. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 18.20843321559128 (T) = (0 -108.51538175725533) / Math.Sqrt((0.8724769605853575 / (142)) + (8.742119424042261 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.08385414738001588 = (118.44771380771326 - 108.51538175725533) / 118.44771380771326 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 175.8851283406755 < 289.7394476073765. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 320.86562677612216 (T) = (0 -179.91571693810377) / Math.Sqrt((4.355094775106843 / (141)) + (3.7105600223028934 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.41216755585122955 = (306.06632677214077 - 179.91571693810377) / 306.06632677214077 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58lzfdql1fehvs91yzkt9xam7ahjbhvpd9edll13ab46i74ktwwgkgbi792e5gkuuzevo5qm8qt83edag7zovoe686gmtw730kms2i5xgji4xcp25287q68fvhwszd3mszht2uh7bchlgkj5qnq1x9m4lg7vwn8cq5l756akua6oyx9k71bmxbysnmhvxvlxde4k9maumfgxd8gxhxx4mwpph2ttyox9zilt3ylv1q9s4bopfuoa8qlrzodg2q67sh85wx4slcd6w7ufnendaxai633ove2ktbaxdt2sz6y6mo42473xd274gz833p6hj3mu77c4m4od9e5s8btxleh0efqnu9zj9rwtbk5758lio35b3q426j5fwwq1qyknfedrsmqyfw1m38mkkotdf7n0vr6p3erhy8dkzntr9fwjrslxjgrbegih0n6bpb5bfuy55bu65ce9kejcfifxwpcs05umrsb8kvd64q2iwugbbi7vd35g5ho0rff9rhombgzzaniyq7bbjbqr88jyw4ccgnoyl31of3a5thv0vg08gnrqzxas800hewtw8tnwgw5pav81ntdpdd62689x3iqpc317y82b3e2trbpdzieoxldaz009tz37gqmh4bdp1bv9lnl5s58udb11z0h7i2sdl5nbyhjyfzxwzezmp4qx0i3eyvsd3fg8sryq9jhlvkonnfcvb4snl4mcbimdzg49tzdhqjmfxfcq3p1st6b9x2xyevo17evpqp4yc4f2rm0f26ivr3t2f5m0boc44vituxaovcqy1jrkcs6im2kdu3jvcexx2k76egve63aon5a6nbxss4rcke90npmqp35qluf571ms160y2nhaqef835wah41qru8tauu362v0r8konl8", oldChar: 'b', newChar: '+') #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 501.81066111944347 < 1.7470637521895473. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 342.4405006940196 (T) = (0 -472.210208628862) / Math.Sqrt((140.31171404855206 / (143)) + (452.4714983762523 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7440166678566676 = (1844.6912331169212 - 472.210208628862) / 1844.6912331169212 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+') #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 110.21002769437645 < 231.48550640816873. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 219.27769783330345 (T) = (0 -107.85708390989568) / Math.Sqrt((14.130378912137388 / (144)) + (8.962556505684965 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5615538398258052 = (245.99846847112087 - 107.85708390989568) / 245.99846847112087 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "bad", newValue: "nice") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 43.68125308475258 < 93.34713990539956. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 283.76900026641806 (T) = (0 -43.74872371907608) / Math.Sqrt((0.12098841407854084 / (144)) + (1.0823113364610935 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5548187049247363 = (98.27170234472631 - 43.74872371907608) / 98.27170234472631 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.PadLeft(n: 18) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.45160813748132 < 44.13377216429222. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 54.028816144115346 (T) = (0 -37.29829713360391) / Math.Sqrt((1.865915128369603 / (144)) + (0.6599231065521503 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.21311093478555407 = (47.39968920960827 - 37.29829713360391) / 47.39968920960827 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Join_Enumerable #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 887.7689798197509 < 919.263121468343. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 36.173870915196304 (T) = (0 -882.1940504736265) / Math.Sqrt((110.19217110847057 / (142)) + (150.04161475538393 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.08971668600570143 = (969.1422845076472 - 882.1940504736265) / 969.1422845076472 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'i', newChar: 'I') #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 69.53038126575316 < 108.01775465209775. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 126.96437840488086 (T) = (0 -66.78964570772874) / Math.Sqrt((14.542172683227793 / (143)) + (1.6092206509322262 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.4283148939214137 = (116.82943109339557 - 66.78964570772874) / 116.82943109339557 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 178.14874309254554 < 282.838341582416. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 195.50859636474135 (T) = (0 -179.6562729650167) / Math.Sqrt((4.7000875782795655 / (143)) + (9.974223173920514 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3967795280179868 = (297.8285408230868 - 179.6562729650167) / 297.8285408230868 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpperInvariant(s: "TeSt") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 101.54673173629439 < 106.56746651629147. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.184817008335486 (T) = (0 -100.0393628036914) / Math.Sqrt((0.5384211232239122 / (143)) + (1.015018209397465 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.10291142153005065 = (111.51559077289325 - 100.0393628036914) / 111.51559077289325 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 182.23228693120637 < 290.3685588680685. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 321.2237180988132 (T) = (0 -184.90035559386243) / Math.Sqrt((3.9776482739638146 / (144)) + (3.4028342224451116 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.39484332103011377 = (305.54129536933266 - 184.90035559386243) / 305.54129536933266 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 202.7279245719473 < 266.680101905136. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 128.60908623926295 (T) = (0 -203.2386267930664) / Math.Sqrt((51.71666411949624 / (143)) + (1.863750034410527 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.29175539781296156 = (286.96106707410337 - 203.2386267930664) / 286.96106707410337 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLower(s: "test") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 30.138562860616766 < 31.862849688098844. IsChangePoint: Marked as a change because one of 9/27/2023 2:54:26 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 19.149185556129236 (T) = (0 -31.136935848474256) / Math.Sqrt((0.8166404030178199 / (147)) + (0.2488536981089413 / (25))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (147) + (25) - 2, .975) and 0.07114153398539617 = (33.521722617302075 - 31.136935848474256) / 33.521722617302075 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: RemoveEmptyEntries) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 68.20049991046149 < 89.58466498326858. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 80.72615317864829 (T) = (0 -68.4694799644956) / Math.Sqrt((5.416078068509184 / (144)) + (2.0127441610215664 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.27615026790190944 = (94.59073745324969 - 68.4694799644956) / 94.59073745324969 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLowerInvariant(s: "TeSt") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 99.60518876236463 < 105.59328318348727. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 51.40540174325488 (T) = (0 -100.36684802437934) / Math.Sqrt((0.746943592207526 / (141)) + (1.1537220558729542 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.09676784743452674 = (111.1196581513455 - 100.36684802437934) / 111.1196581513455 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpperInvariant(s: "test") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 100.08687512579986 < 105.220929559892. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 43.143846298417195 (T) = (0 -100.34554931857905) / Math.Sqrt((0.5912152819011625 / (142)) + (1.85140789583635 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.09938629107048035 = (111.41907826147903 - 100.34554931857905) / 111.41907826147903 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.IndexOfAny #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 59.08416491495762 < 110.43246025177508. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 118.1167340017715 (T) = (0 -58.09569618700299) / Math.Sqrt((17.15561689029119 / (144)) + (4.463849856857934 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5127645612938634 = (119.2353666664257 - 58.09569618700299) / 119.2353666664257 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "b") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 95.96220353115292 < 182.91342787737807. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 184.77166072425987 (T) = (0 -94.49928292085241) / Math.Sqrt((13.646735762577276 / (145)) + (6.180826254569411 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.5171883142316633 = (195.72700020810012 - 94.49928292085241) / 195.72700020810012 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 171.03234063178044 < 220.63090148757433. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 106.9277101092922 (T) = (0 -172.0322693115683) / Math.Sqrt((26.467578710554307 / (143)) + (5.166565637106991 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.27089049167344054 = (235.948465006874 - 172.0322693115683) / 235.948465006874 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'z', newChar: 'y') #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.971562359329198 < 45.721760020878676. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 91.86884171221014 (T) = (0 -23.93606316755156) / Math.Sqrt((0.12850524732618562 / (144)) + (2.0533048583158324 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5026448251008366 = (48.12669974209978 - 23.93606316755156) / 48.12669974209978 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 172.46699466310875 < 283.19532809347055. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 434.95705871151785 (T) = (0 -172.5457854243136) / Math.Sqrt((3.529984506554111 / (140)) + (1.7415775889093739 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.4210978513857428 = (298.0569096821351 - 172.5457854243136) / 298.0569096821351 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpper(s: "test") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 106.99271528721518 < 113.60708647881354. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 11.725071642457202 (T) = (0 -108.46558162548696) / Math.Sqrt((1.0860545495036238 / (143)) + (24.069602073018782 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.0886592453482119 = (119.01759146822124 - 108.46558162548696) / 119.01759146822124 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.CtorCharCount(size: 100) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 49.761077761294054 < 98.83960663888308. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 270.51990280187266 (T) = (0 -48.9407423005207) / Math.Sqrt((2.297342871157223 / (143)) + (0.746663219407537 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.527991671386346 = (103.68618376770091 - 48.9407423005207) / 103.68618376770091 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLowerInvariant(s: "TEST") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 101.98945215433999 < 105.43787457117887. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 55.99634117509845 (T) = (0 -100.6840130330157) / Math.Sqrt((0.5814358747320022 / (144)) + (0.9735563048097299 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.096036182266518 = (111.38057857831278 - 100.6840130330157) / 111.38057857831278 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.PadLeft(n: 2142) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 643.4021833134913 < 1.6432468836743908. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 449.5579013349265 (T) = (0 -633.6376390114843) / Math.Sqrt((268.1549380111814 / (141)) + (122.60441947542604 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.6345341083621188 = (1733.780507318365 - 633.6376390114843) / 1733.780507318365 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 66.37040046254414 < 88.80789824180488. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 72.58182610210834 (T) = (0 -67.94104809661115) / Math.Sqrt((5.287486449228983 / (145)) + (3.036445442426165 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.28387181404888673 = (94.87274684821465 - 67.94104809661115) / 94.87274684821465 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Format_MultipleArgs #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 686.5525672492402 < 696.8635450363571. IsChangePoint: Marked as a change because one of 8/14/2023 10:44:26 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 39.795734168050714 (T) = (0 -664.0412962965379) / Math.Sqrt((487.5165387256765 / (144)) + (31.33716894966507 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.11200992263077146 = (747.80260863256 - 664.0412962965379) / 747.80260863256 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLower(s: "TEST") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 106.79221206408283 < 113.46615348117685. IsChangePoint: Marked as a change because one of 9/26/2023 10:37:55 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 43.845381808653 (T) = (0 -108.23559461434103) / Math.Sqrt((1.4410676892866916 / (144)) + (1.3251676073858947 / (28))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (28) - 2, .975) and 0.08842135253180529 = (118.73423638756019 - 108.23559461434103) / 118.73423638756019 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.ContainsKeyTrue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
75.24 μs 58.51 μs 0.78 0.05 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsKeyTrue<String, String>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsKeyTrue<String, String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsKeyTrue<String, String>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsKeyTrue<String, String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsKeyTrue<String, String>.ImmutableDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 58.513885029429986 < 73.39953208018294. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 70.79841276709061 (T) = (0 -56836.175203852385) / Math.Sqrt((2391759.95400965 / (142)) + (1887677.7909767157 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.26025077407560976 = (76831.6791853752 - 56836.175203852385) / 76831.6791853752 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.BinaryReaderTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
29.42 ns 24.79 ns 0.84 0.02 True
3.71 μs 3.12 μs 0.84 0.03 False
28.37 ns 25.08 ns 0.88 0.02 True
30.84 ns 25.08 ns 0.81 0.01 True
29.45 ns 24.84 ns 0.84 0.01 True
186.76 ns 166.00 ns 0.89 0.03 True
32.59 ns 27.18 ns 0.83 0.02 True
28.63 ns 24.87 ns 0.87 0.02 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryReaderTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.BinaryReaderTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryReaderTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.BinaryReaderTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.BinaryReaderTests.ReadUInt64 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.78542705898299 < 27.89917405127887. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 146.24200064442198 (T) = (0 -24.835602470270594) / Math.Sqrt((0.03464608217788591 / (144)) + (0.02041430251517471 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.15161370298711657 = (29.273931648489892 - 24.835602470270594) / 29.273931648489892 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryReaderTests.ReadLargeString #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.1210228858271654 < 3.4772785173167953. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 48.16909500014901 (T) = (0 -3127.7355061574135) / Math.Sqrt((18401.806523817955 / (145)) + (271.8248794461346 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.15224116067429255 = (3689.4165664437774 - 3127.7355061574135) / 3689.4165664437774 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryReaderTests.ReadUInt16 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 25.082310269284097 < 27.094089114887193. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.31956993109247 (T) = (0 -25.07195158426491) / Math.Sqrt((0.01567998587160312 / (142)) + (0.2290012550921444 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.1185513489566362 = (28.444029671595093 - 25.07195158426491) / 28.444029671595093 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryReaderTests.ReadSingle #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 25.07807472540537 < 29.28866835401433. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 116.69110292937872 (T) = (0 -25.224945174270534) / Math.Sqrt((0.011133818103769611 / (142)) + (0.0645644436190424 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.1793061511824712 = (30.736120674737737 - 25.224945174270534) / 30.736120674737737 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryReaderTests.ReadDouble #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.840311629105216 < 28.025857227579603. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 208.1659661835676 (T) = (0 -24.858073701722425) / Math.Sqrt((0.015484083323203742 / (144)) + (0.011077571468272065 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.1545862654349056 = (29.40344198986801 - 24.858073701722425) / 29.40344198986801 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryReaderTests.ReadSmallString #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 166.00217764449488 < 177.41426647407437. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 29.078994909979485 (T) = (0 -167.62778181684817) / Math.Sqrt((11.950501030103704 / (144)) + (9.51114761260709 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.09887029096311042 = (186.01959311274464 - 167.62778181684817) / 186.01959311274464 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryReaderTests.ReadHalf #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 27.182252800278835 < 30.795021168200318. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 41.3929209846837 (T) = (0 -27.706660869976343) / Math.Sqrt((0.024962910218882358 / (144)) + (0.3718135701562374 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.14345347019534338 = (32.34694194172394 - 27.706660869976343) / 32.34694194172394 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryReaderTests.ReadUInt32 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.872878744589777 < 27.225378195587265. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 123.55036080040384 (T) = (0 -24.885512992049698) / Math.Sqrt((0.09694005530294089 / (144)) + (0.00771672518984212 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.13152124331425594 = (28.65414127919128 - 24.885512992049698) / 28.65414127919128 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
47.39 ns 17.55 ns 0.37 0.01 True
77.59 ns 53.90 ns 0.69 0.02 True
40.91 ns 14.92 ns 0.36 0.06 True
511.17 ns 56.34 ns 0.11 0.05 True
23.30 ns 14.32 ns 0.61 0.01 True
46.55 ns 21.26 ns 0.46 0.02 True
1.97 μs 1.30 μs 0.66 0.01 True
58.74 ns 36.05 ns 0.61 0.02 True
33.85 ns 15.34 ns 0.45 0.02 True
353.32 ns 120.76 ns 0.34 0.01 True
50.08 ns 22.31 ns 0.45 0.01 True
32.02 ns 15.96 ns 0.50 0.01 True
320.32 ns 92.78 ns 0.29 0.02 True
48.58 ns 24.94 ns 0.51 0.03 True
37.98 ns 23.83 ns 0.63 0.06 True
1.01 μs 91.78 ns 0.09 0.01 True
54.07 ns 31.84 ns 0.59 0.02 True
295.27 ns 55.53 ns 0.19 0.09 True
138.42 ns 90.83 ns 0.66 0.02 True
90.79 ns 19.52 ns 0.22 0.02 True
47.42 ns 17.46 ns 0.37 0.01 True
319.24 ns 63.67 ns 0.20 0.04 True
35.94 ns 20.49 ns 0.57 0.01 True
77.04 ns 19.87 ns 0.26 0.05 True
387.85 ns 140.67 ns 0.36 0.01 True
512.25 ns 59.17 ns 0.12 0.01 True
378.50 ns 141.67 ns 0.37 0.08 True
27.94 ns 11.47 ns 0.41 0.10 True
66.12 ns 24.06 ns 0.36 0.01 True
996.48 ns 109.64 ns 0.11 0.02 True
325.97 ns 83.21 ns 0.26 0.01 True
73.85 ns 49.47 ns 0.67 0.02 True
341.64 ns 120.82 ns 0.35 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Char>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Span<Char>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Char>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Span<Char>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Memory.Span<Char>.SequenceCompareToDifferent(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.553906153344872 < 45.01853973029631. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 928.9831465317781 (T) = (0 -17.511715365262702) / Math.Sqrt((0.0356174834756492 / (143)) + (0.023915544978211866 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6318014643720647 = (47.56052420305738 - 17.511715365262702) / 47.56052420305738 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfAnyFiveValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 53.89582775729741 < 73.81529852355438. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 75.68794907538498 (T) = (0 -54.12820820158046) / Math.Sqrt((15.42307933678634 / (142)) + (0.4932813633550653 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.33087426736608117 = (80.89392704792928 - 54.12820820158046) / 80.89392704792928 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.LastIndexOfValue(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.924006414221898 < 38.868349215706786. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 96.53977817245813 (T) = (0 -14.994974374606553) / Math.Sqrt((6.721539455225089 / (142)) + (0.819252937537487 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.6375444633759412 = (41.370520958987136 - 14.994974374606553) / 41.370520958987136 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.StartsWith(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 56.34381353864747 < 491.1350072193805. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 657.0550452956326 (T) = (0 -56.95633252772188) / Math.Sqrt((67.91385336144238 / (143)) + (0.330515409514532 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8894019752435646 = (514.9850791020363 - 56.95633252772188) / 514.9850791020363 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.SequenceCompareTo(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.320450831825186 < 21.843021777015867. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 471.71827465990407 (T) = (0 -14.308302288631436) / Math.Sqrt((0.016789540030571162 / (143)) + (0.006647848197809807 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3777270720922514 = (22.99361204213054 - 14.308302288631436) / 22.99361204213054 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfValue(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.262381315681655 < 44.30709033452331. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 88.27003548803528 (T) = (0 -21.147037421291873) / Math.Sqrt((14.68379858413842 / (144)) + (0.06679690897559215 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.573994306798568 = (49.64026950525456 - 21.147037421291873) / 49.64026950525456 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.Reverse(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.303839216558198 < 1.8952378551792448. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 525.3019226239076 (T) = (0 -1302.8570443714934) / Math.Sqrt((89.93460494733955 / (145)) + (33.060323040273836 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.3460322520975284 = (1992.2344007793376 - 1302.8570443714934) / 1992.2344007793376 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfAnyThreeValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 36.052889600144255 < 55.543154954988424. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 79.15790919037828 (T) = (0 -36.0405182143265) / Math.Sqrt((13.95653665082039 / (145)) + (0.17140831254801395 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.41222884047093994 = (61.31726205008637 - 36.0405182143265) / 61.31726205008637 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.SequenceEqual(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.34480010615952 < 32.28267612000601. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 398.43600896599736 (T) = (0 -15.375771593351086) / Math.Sqrt((0.023431292130328837 / (142)) + (0.05984235113188939 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.54633504043654 = (33.89235000240365 - 15.375771593351086) / 33.89235000240365 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfAnyThreeValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 120.75952192472774 < 335.74678654710897. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 309.90171728166257 (T) = (0 -118.15303176072234) / Math.Sqrt((14.847151771563807 / (143)) + (14.251264997473315 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6661747758199217 = (353.93680046474253 - 118.15303176072234) / 353.93680046474253 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.StartsWith(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 22.30640989368657 < 47.57880087896698. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 730.3738545268696 (T) = (0 -22.423813123960993) / Math.Sqrt((0.07321102024015817 / (142)) + (0.027723477958717 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.5527450569725861 = (50.13653504235627 - 22.423813123960993) / 50.13653504235627 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.SequenceCompareToDifferent(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.960046401381915 < 30.42063829495434. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1134.4176652809438 (T) = (0 -15.974176505328852) / Math.Sqrt((0.00725885243516985 / (141)) + (0.0044523195835788395 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.501008573273055 = (32.01292777735467 - 15.974176505328852) / 32.01292777735467 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfValue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 92.78133118161065 < 304.9555630954207. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 181.78372222829933 (T) = (0 -88.89132585089168) / Math.Sqrt((18.539313457367676 / (144)) + (46.838459949052506 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7266665833525309 = (325.212068619985 - 88.89132585089168) / 325.212068619985 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.LastIndexOfAnyValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.942393765535396 < 46.13573759700836. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 69.74624179422499 (T) = (0 -25.63377839540642) / Math.Sqrt((8.17097633633042 / (140)) + (1.6293914575900885 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.47735009744362017 = (49.04579197284214 - 25.63377839540642) / 49.04579197284214 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.EndsWith(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.830079895149883 < 34.308983384478466. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 154.19861213899492 (T) = (0 -21.337036600089593) / Math.Sqrt((0.2643382035590545 / (144)) + (0.22505944601506178 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.4111889437753227 = (36.23749312198352 - 21.337036600089593) / 36.23749312198352 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.SequenceEqual(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 91.78288667064905 < 0.9552466158619047. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1445.4283484324396 (T) = (0 -92.52030332329836) / Math.Sqrt((52.479421534036284 / (142)) + (0.7652871516029698 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.9075770485032055 = (1001.0533295564278 - 92.52030332329836) / 1001.0533295564278 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfAnyTwoValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 31.837506051165693 < 51.58670979387862. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 80.82103056620133 (T) = (0 -31.930199540599048) / Math.Sqrt((13.107084033959442 / (143)) + (0.14054667271835186 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.4399817184771381 = (57.01635213366788 - 31.930199540599048) / 57.01635213366788 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.LastIndexOfValue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 55.52892795478532 < 279.93997348044775. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 100.29797969841493 (T) = (0 -68.30980919352125) / Math.Sqrt((8.364990831561663 / (142)) + (152.18352953769022 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.7688466088129493 = (295.51722707907203 - 68.30980919352125) / 295.51722707907203 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.Reverse(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 90.83490670357251 < 133.07695377551047. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 52.18407070768048 (T) = (0 -92.15362884271093) / Math.Sqrt((0.7627953302019986 / (143)) + (25.023720180696024 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.34159845701032365 = (139.9656939202463 - 92.15362884271093) / 139.9656939202463 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.SequenceEqual(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.52404072593205 < 85.55513350103067. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1362.925367674143 (T) = (0 -19.4921435467615) / Math.Sqrt((0.3739552047704292 / (142)) + (0.0022313275981282786 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.784414464294327 = (90.41489487204302 - 19.4921435467615) / 90.41489487204302 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.SequenceCompareToDifferent(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.45687932443986 < 45.149727596187354. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1368.8824454327485 (T) = (0 -17.546378323498953) / Math.Sqrt((0.056604213949360674 / (143)) + (0.0025768106102882302 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6313089319809392 = (47.59100462556318 - 17.546378323498953) / 47.59100462556318 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.Fill(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 63.669099712930574 < 300.3115451897818. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 648.8742087163587 (T) = (0 -62.14990624756363) / Math.Sqrt((8.917282641356268 / (143)) + (2.730110745553991 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8034840144769162 = (316.25878211451237 - 62.14990624756363) / 316.25878211451237 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.StartsWith(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.486352388267804 < 34.20089358806564. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1038.2869094050868 (T) = (0 -20.490509835052247) / Math.Sqrt((0.0069476836979829515 / (143)) + (0.005235775519314622 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.4308081035537594 = (35.99929999528296 - 20.490509835052247) / 35.99929999528296 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.SequenceCompareTo(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.8729442235515 < 72.72901768038491. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 572.2725132522531 (T) = (0 -19.975795019902304) / Math.Sqrt((1.3542940421050242 / (140)) + (0.003915425902982072 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.7393540252083726 = (76.63956842560829 - 19.975795019902304) / 76.63956842560829 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfAnyFiveValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 140.6696755034827 < 367.95119222287406. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 474.8073495193605 (T) = (0 -140.90255739408528) / Math.Sqrt((35.10107707111764 / (140)) + (0.7486434128345956 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.6388964062526038 = (390.1998203115398 - 140.90255739408528) / 390.1998203115398 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.EndsWith(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 59.16735315301322 < 485.9182623562938. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1469.1636449818193 (T) = (0 -59.95925780823888) / Math.Sqrt((11.284743950907787 / (143)) + (0.5011903192034166 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8834073278787836 = (514.2626609149312 - 59.95925780823888) / 514.2626609149312 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfAnyFourValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 141.6692916292892 < 359.3921999006965. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 227.68514608589126 (T) = (0 -133.4273197013857) / Math.Sqrt((15.715613924343435 / (145)) + (31.586730299753576 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.6477505105616284 = (378.7864104902548 - 133.4273197013857) / 378.7864104902548 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.Fill(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 11.467980424288752 < 26.53517631748031. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 176.05039662796526 (T) = (0 -11.47519575946455) / Math.Sqrt((1.2196957490877667 / (145)) + (0.017105605492736743 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.5924999042778327 = (28.1599829789692 - 11.47519575946455) / 28.1599829789692 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.EndsWith(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.060025011515204 < 62.788082486653394. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1038.9293682291643 (T) = (0 -24.178172538419624) / Math.Sqrt((0.22492223813105594 / (144)) + (0.0022412937788785134 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.6348197148539126 = (66.20886592699642 - 24.178172538419624) / 66.20886592699642 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.SequenceCompareTo(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 109.6421194455263 < 0.9577998121642401. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1011.9148787060767 (T) = (0 -107.6750970617141) / Math.Sqrt((107.54971283612075 / (143)) + (0.8864189292916383 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8925735931948342 = (1002.3149825442761 - 107.6750970617141) / 1002.3149825442761 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.LastIndexOfAnyValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 83.21346274093985 < 309.17535903432986. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 829.7226785666087 (T) = (0 -82.33079249678516) / Math.Sqrt((9.45172567419567 / (143)) + (0.5980723845606706 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7472174837777688 = (325.6981286807228 - 82.33079249678516) / 325.6981286807228 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfAnyFourValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 49.46839880831115 < 70.03035003025632. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 82.08533886366837 (T) = (0 -49.72980994652089) / Math.Sqrt((14.516086529397562 / (143)) + (0.15839489080456182 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.35039889273756325 = (76.55437989644652 - 49.72980994652089) / 76.55437989644652 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Char>.IndexOfAnyTwoValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 120.81723970557856 < 323.56372091054556. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 157.52599948802265 (T) = (0 -116.78097541303866) / Math.Sqrt((16.615923690527296 / (145)) + (58.973553711380596 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.6605093956944967 = (343.9888289454661 - 116.78097541303866) / 343.9888289454661 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
519.60 ns 101.02 ns 0.19 0.01 True
2.53 μs 2.11 μs 0.83 0.01 False
2.54 μs 2.12 μs 0.83 0.01 False
41.55 ns 30.31 ns 0.73 0.02 True
234.70 ns 39.49 ns 0.17 0.05 True
455.09 ns 59.36 ns 0.13 0.01 True
312.10 ns 129.70 ns 0.42 0.01 True
312.04 ns 129.22 ns 0.41 0.01 True
525.31 ns 107.71 ns 0.21 0.01 True
2.92 μs 2.50 μs 0.86 0.01 False
2.54 μs 2.11 μs 0.83 0.01 False
36.28 ns 24.85 ns 0.69 0.01 True
2.53 μs 2.11 μs 0.83 0.01 False
528.72 ns 131.50 ns 0.25 0.05 True
237.67 ns 47.17 ns 0.20 0.03 True
2.54 μs 2.11 μs 0.83 0.01 False
602.31 ns 237.48 ns 0.39 0.01 True
2.52 μs 2.14 μs 0.85 0.01 False

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.StringSearch* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.StringSearch* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, Ordinal, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 101.02023799926944 < 492.76331733990736. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1263.3286256914848 (T) = (0 -102.0899432894834) / Math.Sqrt((1.3085736192313708 / (142)) + (2.985767732909472 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.803174967465316 = (518.6837363864954 - 102.0899432894834) / 518.6837363864954 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, None, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.112116655591533 < 2.407125271819674. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 6.198642503597663 (T) = (0 -2142.9157626824135) / Math.Sqrt((52704.321746476184 / (141)) + (1305.4380802482804 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.055793646249412555 = (2269.5417735437795 - 2142.9157626824135) / 2269.5417735437795 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreNonSpace, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.1158636176781593 < 2.4124262037313104. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 6.29875912983353 (T) = (0 -2141.4686629129446) / Math.Sqrt((53075.94284613019 / (144)) + (1201.2207690206437 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.056120504576972395 = (2268.7945583066003 - 2141.4686629129446) / 2268.7945583066003 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 30.30823733080348 < 39.50819682675831. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 121.35848960221558 (T) = (0 -30.28431962142807) / Math.Sqrt((0.05139323587168321 / (145)) + (0.2513494675082848 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.27245179383548956 = (41.625172551906886 - 30.28431962142807) / 41.625172551906886 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, Ordinal, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.48610171727062 < 223.13393159338798. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 615.9477019280105 (T) = (0 -40.61725352260765) / Math.Sqrt((12.055732513724527 / (143)) + (0.5034009698299097 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8282217567591145 = (236.45167604636555 - 40.61725352260765) / 236.45167604636555 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, Ordinal, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 59.357648835364856 < 430.3934349203419. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1295.9825036085579 (T) = (0 -60.15342536505068) / Math.Sqrt((10.605353856169724 / (144)) + (0.5464436989618631 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.8671968401851601 = (452.95176296196024 - 60.15342536505068) / 452.95176296196024 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 129.69870641305954 < 297.4692467419756. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1085.4927004015613 (T) = (0 -130.28134852421775) / Math.Sqrt((0.7212551370928492 / (143)) + (0.699799735238149 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.583920493508883 = (313.11647531719797 - 130.28134852421775) / 313.11647531719797 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 129.2172776556563 < 296.52543829698465. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1095.201481468557 (T) = (0 -128.89000729746587) / Math.Sqrt((1.5131536753977688 / (144)) + (0.5282259263921196 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5875915621488506 = (312.52999567381823 - 128.89000729746587) / 312.52999567381823 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, Ordinal, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 107.70775032062332 < 499.05812951901237. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1993.4531372223528 (T) = (0 -108.28749473068623) / Math.Sqrt((1.5565917929536965 / (145)) + (0.9920695959639976 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.7939366583849325 = (525.5058657302112 - 108.28749473068623) / 525.5058657302112 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, IgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.5024139380151103 < 2.773430045252624. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 6.480958186956068 (T) = (0 -2524.7044396449533) / Math.Sqrt((52431.395050661355 / (144)) + (1700.2041876087383 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.050022975367235066 = (2657.6478948224403 - 2524.7044396449533) / 2657.6478948224403 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.107330054172541 < 2.4105436282402306. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 6.266428871459148 (T) = (0 -2139.3505176175318) / Math.Sqrt((53847.47713697805 / (143)) + (1266.7117264109713 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.056552107967067264 = (2267.5873629943453 - 2139.3505176175318) / 2267.5873629943453 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, Ordinal, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.851544566835752 < 34.41978648207765. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 132.09646519663002 (T) = (0 -25.118087012156067) / Math.Sqrt((0.03530413972003801 / (141)) + (0.2044260286724059 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.30653129886861163 = (36.22093826466302 - 25.118087012156067) / 36.22093826466302 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, None, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.1124548491795365 < 2.4133213025698956. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 6.471551659872277 (T) = (0 -2141.7310221741277) / Math.Sqrt((53868.6176405231 / (146)) + (1180.1385018483033 / (30))) is greater than 1.9736914397558152 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (146) + (30) - 2, .975) and 0.05754335041975616 = (2272.49818135192 - 2141.7310221741277) / 2272.49818135192 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 131.500881283455 < 513.6208128013765. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 359.1376785802956 (T) = (0 -131.543664728631) / Math.Sqrt((159.95236022458624 / (141)) + (4.1802661943862445 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.7549801118885372 = (536.8693363731766 - 131.543664728631) / 536.8693363731766 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Ordinal, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 47.17060273105444 < 224.77755623881603. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 784.4990583685388 (T) = (0 -44.3586110945846) / Math.Sqrt((3.5599851392956565 / (143)) + (1.0557476742271468 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8125635429285976 = (236.65946202603772 - 44.3586110945846) / 236.65946202603772 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreNonSpace, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.112263536208518 < 2.412522555033174. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 6.409537781922509 (T) = (0 -2138.138941281614) / Math.Sqrt((54584.43015884832 / (144)) + (1354.6878035224383 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.058151982622242616 = (2270.152829152314 - 2138.138941281614) / 2270.152829152314 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 237.48193857803747 < 574.7500290627142. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1705.648726621141 (T) = (0 -237.65158599038276) / Math.Sqrt((3.5182382965632173 / (143)) + (0.647812116068095 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6067026295971315 = (604.2541951067401 - 237.65158599038276) / 604.2541951067401 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, None, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.137664514218009 < 2.4077814046732775. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 6.0617950907972435 (T) = (0 -2139.3661098112516) / Math.Sqrt((53685.31854553801 / (143)) + (1309.6910215299472 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.05482467160194892 = (2263.4595355310407 - 2139.3661098112516) / 2263.4595355310407 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.MathBenchmarks.Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
175.27 μs 142.42 μs 0.81 0.09 True
54.80 μs 18.17 μs 0.33 0.02 True
109.92 μs 30.57 μs 0.28 0.01 True
40.75 μs 19.28 μs 0.47 0.01 True
182.20 μs 124.38 μs 0.68 0.01 True
92.36 μs 77.48 μs 0.84 0.09 False
342.47 μs 117.44 μs 0.34 0.02 True
92.42 μs 78.49 μs 0.85 0.12 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.MathBenchmarks.Double*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.MathBenchmarks.Double* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.MathBenchmarks.Double*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.MathBenchmarks.Double* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.MathBenchmarks.Double.SinCosPi #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 142.4217294128788 < 166.36420752754435. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 42.62839345960362 (T) = (0 -143937.02194396564) / Math.Sqrt((66028361.28290818 / (144)) + (4468607.311664527 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.18754066758793833 = (177162.12516956346 - 143937.02194396564) / 177162.12516956346 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Double.ScaleB #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 18.16790688244381 < 52.075230957946566. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 817.5224583473636 (T) = (0 -18262.023897568513) / Math.Sqrt((12598.84395902573 / (142)) + (57536.33240239336 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.6672555660255899 = (54883.0334423354 - 18262.023897568513) / 54883.0334423354 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Double.CopySign #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 30.572283995098033 < 103.92126931320882. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 3289.7460343242105 (T) = (0 -30622.309919923802) / Math.Sqrt((66650.61854152971 / (144)) + (3365.425607715424 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7203694044054945 = (109509.86910005174 - 30622.309919923802) / 109509.86910005174 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Double.ILogB #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.278600273029443 < 38.722846030456765. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 760.9717028973611 (T) = (0 -19308.600336763193) / Math.Sqrt((12624.93947963593 / (140)) + (20957.13388435701 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.5253568599697264 = (40680.247344418916 - 19308.600336763193) / 40680.247344418916 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Double.TanPi #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 124.37737209467122 < 171.0854696094046. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 562.1005422594899 (T) = (0 -124257.14769654603) / Math.Sqrt((1177105.7380959531 / (143)) + (52951.05961783706 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3114325754786601 = (180457.4879256361 - 124257.14769654603) / 180457.4879256361 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Double.SinPi #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 77.4759881840796 < 86.93447543896073. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.101959147047964 (T) = (0 -78020.04117969499) / Math.Sqrt((14746068.040493533 / (143)) + (2459026.8396187373 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.1543558321270749 = (92261.07640042172 - 78020.04117969499) / 92261.07640042172 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Double.Hypot #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 117.43686960510863 < 325.3001786951669. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 437.0478649529435 (T) = (0 -118299.56519090664) / Math.Sqrt((382508.75384423125 / (142)) + (7805369.466984402 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.6544776119460333 = (342378.8711845485 - 118299.56519090664) / 342378.8711845485 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Double.CosPi #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 78.48893895833334 < 87.94528924224849. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 30.653360662721287 (T) = (0 -79184.08074319416) / Math.Sqrt((12503282.417882869 / (143)) + (3812873.1939323624 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.15204091074724574 = (93381.95880767483 - 79184.08074319416) / 93381.95880767483 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Layout.SearchLoops

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
84.06 ns 70.53 ns 0.84 0.01 True
119.14 ns 100.35 ns 0.84 0.01 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Layout.SearchLoops*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Layout.SearchLoops* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Layout.SearchLoops*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Layout.SearchLoops* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### Layout.SearchLoops.LoopGoto #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 70.5280967493638 < 79.89474498809753. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 139.42683000093476 (T) = (0 -70.53451640064375) / Math.Sqrt((0.6751318593419261 / (144)) + (0.1889085862986699 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.17162447945362622 = (85.14799707519256 - 70.53451640064375) / 85.14799707519256 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Layout.SearchLoops.LoopReturn #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 100.34807191225113 < 111.54028225739764. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 37.420128835231154 (T) = (0 -100.64210252077642) / Math.Sqrt((15.93009220873554 / (143)) + (0.4087282096860545 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.11619274231676899 = (113.87336056120957 - 100.64210252077642) / 113.87336056120957 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
218.13 μs 174.63 μs 0.80 0.12 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf2.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf2.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFooInterAlia #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 174.6253861111111 < 207.97543669043483. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 7.073734455344596 (T) = (0 -176881.33074748283) / Math.Sqrt((391150429.53524625 / (142)) + (59640780.93041687 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.08011415201325138 = (192286.17456677178 - 176881.33074748283) / 192286.17456677178 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
296.08 ns 274.18 ns 0.93 0.01 True
296.20 ns 279.11 ns 0.94 0.03 True
299.46 ns 278.46 ns 0.93 0.02 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Single*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Single* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Single*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Single* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Single.ToString(value: 12345) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 274.18342431937185 < 278.4673189011358. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.97893922475173 (T) = (0 -274.48947792694304) / Math.Sqrt((9.034635230360147 / (144)) + (1.9085967115944735 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.06984052516804368 = (295.09937312258484 - 274.48947792694304) / 295.09937312258484 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Single.ToStringWithFormat(value: 12345, format: "R") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 279.1149464276877 < 282.26763606117726. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 9.704730335007772 (T) = (0 -280.15210361758017) / Math.Sqrt((11.621687743739155 / (143)) + (137.38618710974285 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.06958222141947988 = (301.1035580650561 - 280.15210361758017) / 301.1035580650561 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Single.ToStringWithFormat(value: 12345, format: "G") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 278.45934225643776 < 283.79849211850495. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 40.65991544883621 (T) = (0 -277.16376753949174) / Math.Sqrt((28.62265913148676 / (140)) + (4.040148456808166 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.0787054312612849 = (300.8416384337734 - 277.16376753949174) / 300.8416384337734 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
694.21 ns 566.50 ns 0.82 0.01 True
2.05 μs 491.01 ns 0.24 0.02 False
575.39 ns 453.01 ns 0.79 0.02 True
16.08 μs 11.24 μs 0.70 0.02 True
1.33 μs 268.00 ns 0.20 0.02 False
9.75 μs 3.34 μs 0.34 0.05 True
447.98 ns 416.07 ns 0.93 0.01 False
445.11 μs 189.60 μs 0.43 0.03 False
1.83 μs 283.69 ns 0.16 0.06 False
447.77 ns 419.43 ns 0.94 0.01 False
439.54 ns 325.71 ns 0.74 0.02 True
13.33 μs 9.40 μs 0.71 0.02 True
1.34 μs 270.28 ns 0.20 0.03 False
409.27 ns 256.29 ns 0.63 0.02 True
447.98 ns 411.85 ns 0.92 0.01 False
284.82 ns 236.73 ns 0.83 0.03 True
477.80 μs 222.28 μs 0.47 0.02 False
2.04 μs 488.37 ns 0.24 0.02 False
15.16 μs 11.20 μs 0.74 0.01 True
9.14 μs 2.48 μs 0.27 0.01 True
453.20 ns 320.54 ns 0.71 0.02 True
598.67 ns 539.43 ns 0.90 0.03 True
282.66 ns 237.95 ns 0.84 0.02 True
18.26 μs 14.36 μs 0.79 0.01 True
432.56 ns 324.88 ns 0.75 0.02 True
477.65 μs 221.72 μs 0.46 0.03 False
436.70 ns 274.63 ns 0.63 0.01 True
16.53 μs 12.40 μs 0.75 0.01 True
1.94 μs 320.08 ns 0.17 0.03 False
18.23 μs 14.20 μs 0.78 0.01 True
445.15 μs 190.29 μs 0.43 0.02 False
473.31 ns 351.69 ns 0.74 0.01 True
452.57 ns 411.41 ns 0.91 0.01 False
9.50 μs 2.95 μs 0.31 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.RegularExpressions.Tests.Perf_Regex_Common* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.RegularExpressions.Tests.Perf_Regex_Common* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 566.4963978165359 < 658.9379211451097. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.18451643838764 (T) = (0 -567.3672632640785) / Math.Sqrt((714.7365787288553 / (143)) + (34.59652232071278 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2055421153885078 = (714.156501249319 - 567.3672632640785) / 714.156501249319 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsNotMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 491.0121549825533 < 1.9384116002228415. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.518647123676796 (T) = (0 -557.6908761868503) / Math.Sqrt((594168.0143760401 / (145)) + (18491.62804395276 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.511900041687358 = (1142.575135869267 - 557.6908761868503) / 1142.575135869267 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 453.00873995784565 < 546.52710529235. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 62.89627840297512 (T) = (0 -440.7114123905083) / Math.Sqrt((849.2302828628908 / (144)) + (13.498669589767148 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2644664869691345 = (599.1724436518157 - 440.7114123905083) / 599.1724436518157 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 11.23959244299008 < 14.484512711067733. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 71.64945825340241 (T) = (0 -11097.178465493036) / Math.Sqrt((555201.0866580571 / (143)) + (11539.777659811498 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2966491264834591 = (15777.585389224743 - 11097.178465493036) / 15777.585389224743 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 267.9982024017068 < 1.2630604837779296. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.349436867187238 (T) = (0 -317.62280886763136) / Math.Sqrt((278559.82899214484 / (143)) + (9695.391650655461 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5561004065747911 = (715.5284969215601 - 317.62280886763136) / 715.5284969215601 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.3390720974531116 < 9.279464879573412. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 133.11927835856528 (T) = (0 -3398.3230078053316) / Math.Sqrt((32842.04558627709 / (145)) + (63588.232585900325 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.6548583435216903 = (9846.168794808738 - 3398.3230078053316) / 9846.168794808738 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsNotMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 416.067315003036 < 424.9180252649542. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.24065387070894 (T) = (0 -412.5540569653258) / Math.Sqrt((178.17793405426522 / (143)) + (8.595362555083653 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.10293652654889117 = (459.89394192830275 - 412.5540569653258) / 459.89394192830275 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 189.59993470309809 < 422.4038267308237. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.35210576509826 (T) = (0 -201886.05321486492) / Math.Sqrt((15902773359.311571 / (142)) + (616007955.2429655 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.3226097305730845 = (298035.06534816953 - 201886.05321486492) / 298035.06534816953 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 283.6945450240609 < 1.7280304347824198. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.349741505080187 (T) = (0 -353.0665642689428) / Math.Sqrt((582466.7567191848 / (145)) + (21142.766474497403 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.6190560541984681 = (926.8202531111677 - 353.0665642689428) / 926.8202531111677 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 419.4253740283628 < 424.4499816272062. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 36.417019414581375 (T) = (0 -413.13491172379054) / Math.Sqrt((189.00073693318888 / (142)) + (10.491696062571425 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.10255721530760761 = (460.3467973341573 - 413.13491172379054) / 460.3467973341573 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 325.7136677772216 < 411.0739887369063. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 99.25155990899744 (T) = (0 -326.71233147320453) / Math.Sqrt((180.64916519050553 / (142)) + (2.6600975001857146 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.2616580188165534 = (442.49458895664554 - 326.71233147320453) / 442.49458895664554 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.403164952009599 < 12.722143272064624. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 71.0571158651928 (T) = (0 -9373.275282331979) / Math.Sqrt((562340.9994499186 / (144)) + (10095.748628027404 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.3305325066531762 = (14001.090979746894 - 9373.275282331979) / 14001.090979746894 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 270.28105282848145 < 1.2629206223337033. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.440401141844468 (T) = (0 -317.38560100842454) / Math.Sqrt((278666.4259018076 / (144)) + (10055.468900167998 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5589151761233273 = (719.5568376597912 - 317.38560100842454) / 719.5568376597912 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 256.2890939924738 < 388.60855761546037. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 96.84249546192949 (T) = (0 -254.64420484906523) / Math.Sqrt((429.1421707133252 / (142)) + (2.5989073698278924 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.40139169655680323 = (425.3937063424463 - 254.64420484906523) / 425.3937063424463 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 411.8516754207225 < 425.4262614396711. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.672507295569964 (T) = (0 -411.78659612694395) / Math.Sqrt((193.8925362716824 / (145)) + (7.045681114913528 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.10534595638423186 = (460.2746716068006 - 411.78659612694395) / 460.2746716068006 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 236.72564694448315 < 270.5493356582256. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 77.40181893457996 (T) = (0 -239.42608373510265) / Math.Sqrt((41.96652938741528 / (141)) + (3.8367098875487256 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.17415665221251567 = (289.9170700793906 - 239.42608373510265) / 289.9170700793906 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 222.2838846830986 < 453.76728409853973. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.25932351895028 (T) = (0 -234831.05168458528) / Math.Sqrt((15798735880.655745 / (144)) + (641716542.5339553 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2870963899593736 = (329400.8451874762 - 234831.05168458528) / 329400.8451874762 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsNotMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 488.3717301186879 < 1.9386573167284045. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.265573934035494 (T) = (0 -558.9208076428702) / Math.Sqrt((590849.4433937612 / (143)) + (19632.75642212986 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5057101342424879 = (1130.7551426050572 - 558.9208076428702) / 1130.7551426050572 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 11.195081665183867 < 14.402744440178088. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 71.99063510706415 (T) = (0 -11125.550236100642) / Math.Sqrt((550262.8403314294 / (142)) + (10031.06613817314 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.2956875450433495 = (15796.327550086282 - 11125.550236100642) / 15796.327550086282 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.482513790236765 < 8.681544334690221. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 544.761783626789 (T) = (0 -2530.073949593306) / Math.Sqrt((20698.344625959773 / (145)) + (301.529991719362 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.726890663268701 = (9263.959921233092 - 2530.073949593306) / 9263.959921233092 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 320.53782770854974 < 429.8430629721991. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 89.24811806246737 (T) = (0 -316.67490881185165) / Math.Sqrt((380.5088946250281 / (143)) + (6.27654301859099 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.32316192241329894 = (467.873955822597 - 316.67490881185165) / 467.873955822597 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 539.425009139654 < 568.6088535491044. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 63.76430359095338 (T) = (0 -538.914336776421) / Math.Sqrt((79.28469561014032 / (141)) + (10.350392803299066 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.101287934811919 = (599.6518324961376 - 538.914336776421) / 599.6518324961376 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 237.94630654707524 < 268.6827673650386. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 69.12173960479154 (T) = (0 -238.3102828092575) / Math.Sqrt((64.87307166430011 / (143)) + (2.85764811282905 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.1768826731358017 = (289.5216453736188 - 238.3102828092575) / 289.5216453736188 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.356199833157708 < 17.34258821332201. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 63.37469087235964 (T) = (0 -14230.57111826218) / Math.Sqrt((625142.7184524622 / (145)) + (29630.432426918076 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.24481765292551347 = (18843.88740466488 - 14230.57111826218) / 18843.88740466488 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 324.8820528465907 < 411.6478547952213. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 107.95798367063288 (T) = (0 -328.7535465056215) / Math.Sqrt((124.23302140297675 / (142)) + (7.0611433016457354 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.2570662946594502 = (442.5072441085786 - 328.7535465056215) / 442.5072441085786 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 221.71843474178402 < 453.76884062838207. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.243351482929816 (T) = (0 -235077.06868427552) / Math.Sqrt((15818835785.289314 / (144)) + (624081522.3850973 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2861363686623965 = (329302.4863079232 - 235077.06868427552) / 329302.4863079232 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 274.6325930141357 < 416.70743613577037. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 234.06476814087742 (T) = (0 -277.4234579885823) / Math.Sqrt((20.511876718536634 / (144)) + (10.028249524806458 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.36810344968813846 = (439.03303135879565 - 277.4234579885823) / 439.03303135879565 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 12.396766028810655 < 15.691901293800536. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 71.6085381010283 (T) = (0 -12322.335929884108) / Math.Sqrt((572701.8478521744 / (142)) + (9306.231249124568 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.276926763452551 = (17041.61529850165 - 12322.335929884108) / 17041.61529850165 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 320.0792133043679 < 1.836074776395992. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.222854039031764 (T) = (0 -396.27465839468675) / Math.Sqrt((637413.2299958974 / (144)) + (23908.01826220883 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5999515207889115 = (990.5665912695282 - 396.27465839468675) / 990.5665912695282 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.198466520965793 < 17.461654088345405. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 75.13706976276454 (T) = (0 -14187.830043412852) / Math.Sqrt((532363.450478009 / (145)) + (4508.301309313012 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.24664358978859477 = (18832.825806090234 - 14187.830043412852) / 18832.825806090234 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 190.28944994725742 < 422.58160971188823. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.228860386720182 (T) = (0 -201764.97229753935) / Math.Sqrt((15968778992.812685 / (143)) + (648497767.4380666 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3201231363470612 = (296766.9339613469 - 201764.97229753935) / 296766.9339613469 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 351.6873811222064 < 450.4225730753051. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 96.9993483036286 (T) = (0 -354.4909182428802) / Math.Sqrt((98.52880105409801 / (144)) + (23.325826900008668 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.24858757633973738 = (471.76611283067734 - 354.4909182428802) / 471.76611283067734 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsNotMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 411.4063164769214 < 427.8564375637727. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 37.86508011740947 (T) = (0 -413.6051666150519) / Math.Sqrt((183.5725045325403 / (142)) + (8.1975936433865 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.10278881791216991 = (460.98975901368465 - 413.6051666150519) / 460.98975901368465 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.9493562036662384 < 9.038147234596538. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 555.6169964907195 (T) = (0 -2939.5342288875863) / Math.Sqrt((19557.6539047013 / (143)) + (240.93960043419446 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.694606218897193 = (9625.389941709482 - 2939.5342288875863) / 9625.389941709482 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.StringReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
25.00 μs 9.72 μs 0.39 0.01 True
452.27 μs 391.03 μs 0.86 0.02 False
43.98 μs 29.38 μs 0.67 0.01 True
187.02 μs 169.99 μs 0.91 0.03 False
34.50 μs 20.01 μs 0.58 0.01 True
34.80 μs 19.81 μs 0.57 0.01 True
684.92 μs 555.94 μs 0.81 0.04 False
72.49 μs 55.28 μs 0.76 0.02 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StringReaderReadLineTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.StringReaderReadLineTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StringReaderReadLineTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.StringReaderReadLineTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.717993424279584 < 23.749336292440777. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 199.1792046970088 (T) = (0 -9501.823608377601) / Math.Sqrt((28876.51443010555 / (144)) + (174981.19355611692 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.6195138958581683 = (24972.853160586543 - 9501.823608377601) / 24972.853160586543 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 1]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 391.0312803125 < 429.7189251972134. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.19022084872357 (T) = (0 -390792.1630789619) / Math.Sqrt((938096866.8713224 / (143)) + (5813007.994337248 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.1808111320907459 = (477047.6972866433 - 390792.1630789619) / 477047.6972866433 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 33, 128]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 29.38120856046366 < 41.92452938948913. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 170.84399087959474 (T) = (0 -29013.728289471965) / Math.Sqrt((695785.247413949 / (142)) + (100851.5408671578 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.3486256174141819 = (44542.32322476917 - 29013.728289471965) / 44542.32322476917 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 8]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 169.9947697222222 < 178.39564331349206. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 23.922937900234853 (T) = (0 -169947.0578690198) / Math.Sqrt((194240433.85785592 / (145)) + (680026.8192782141 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.1411128193723946 = (197868.8955921268 - 169947.0578690198) / 197868.8955921268 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 1024]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.01327889797223 < 32.65953465914554. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 207.5606387413288 (T) = (0 -19806.326987110566) / Math.Sqrt((65642.04966773017 / (143)) + (134495.40642335798 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.4242016257266585 = (34398.02519780675 - 19806.326987110566) / 34398.02519780675 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 129, 1024]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.806055821233162 < 32.98139594581592. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 221.27329886712417 (T) = (0 -19705.344070613) / Math.Sqrt((51649.64512678501 / (144)) + (129834.49571819694 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.4346198774654725 = (34853.26647544034 - 19705.344070613) / 34853.26647544034 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 555.9397595238096 < 650.927862237493. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 22.149968135186516 (T) = (0 -563189.2668995374) / Math.Sqrt((3600065990.196121 / (142)) + (1045832717.7546656 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.23382636624461353 = (735067.4078133898 - 563189.2668995374) / 735067.4078133898 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 9, 32]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 55.28271797153025 < 68.45465495891939. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 76.85145311325778 (T) = (0 -55106.19577955617) / Math.Sqrt((7929200.936925811 / (143)) + (172236.6200866439 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2564939030335849 = (74116.6696606731 - 55106.19577955617) / 74116.6696606731 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Guid

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
15.94 ns 6.36 ns 0.40 0.02 True
17.65 ns 8.05 ns 0.46 0.01 True
23.41 ns 16.52 ns 0.71 0.01 True
15.88 ns 6.40 ns 0.40 0.01 True
18.30 ns 8.75 ns 0.48 0.02 True

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Guid* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Guid* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Guid.EqualsNotSame #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.359515621311288 < 15.143332990065687. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 960.7287353316199 (T) = (0 -6.362631857268136) / Math.Sqrt((0.013403778113877138 / (143)) + (0.00015175615159525072 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6001281854188606 = (15.911678756185681 - 6.362631857268136) / 15.911678756185681 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Guid.EqualsOperator #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.049005197887338 < 16.765463178545243. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2154.8207296750097 (T) = (0 -8.045836345447649) / Math.Sqrt((0.0011267516566614065 / (143)) + (0.00035932845612168384 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.54409292103688 = (17.647974152422638 - 8.045836345447649) / 17.647974152422638 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Guid.ctor_bytes #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.524147690181557 < 22.186026657700477. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 205.26964899284403 (T) = (0 -16.496582250569965) / Math.Sqrt((0.029369406658221773 / (143)) + (0.027019615038314734 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.29269809151957815 = (23.32325426070385 - 16.496582250569965) / 23.32325426070385 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Guid.EqualsSame #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.398491294055371 < 15.091804372043379. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1671.4748139246706 (T) = (0 -6.382486404393913) / Math.Sqrt((0.003359476238833689 / (141)) + (0.00026067145478989414 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.5989281954346187 = (15.91357540406075 - 6.382486404393913) / 15.91357540406075 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Guid.NotEqualsOperator #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.746399446001394 < 17.398127559460125. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 487.71760550839065 (T) = (0 -8.715265488800847) / Math.Sqrt((0.008333377994294464 / (142)) + (0.009815543691525146 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.5236463760508427 = (18.295789200778824 - 8.715265488800847) / 18.295789200778824 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.Perf_StreamWriter

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.42 secs 5.30 secs 0.71 0.05 True
7.27 secs 5.20 secs 0.72 0.07 True
466.84 ms 425.95 ms 0.91 0.01 True
273.24 ns 225.67 ns 0.83 0.01 True
342.82 ms 300.28 ms 0.88 0.01 True
7.27 secs 5.23 secs 0.72 0.06 True
335.93 ms 292.10 ms 0.87 0.01 True

graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_StreamWriter*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_StreamWriter* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_StreamWriter*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_StreamWriter* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_StreamWriter.WritePartialCharArray(writeLength: 100) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.2972118545 < 7.0837861338499994. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 111.30899582618713 (T) = (0 -5332967236.99525) / Math.Sqrt((8478770700138349 / (144)) + (9338695113257480 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.28651478965110444 = (7474530879.746504 - 5332967236.99525) / 7474530879.746504 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_StreamWriter.WriteCharArray(writeLength: 100) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.200132251285714 < 6.972565731410595. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 95.22643570089402 (T) = (0 -5212022132.542606) / Math.Sqrt((14254427776530040 / (143)) + (11914464294326812 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2893914359266934 = (7334589527.976661 - 5212022132.542606) / 7334589527.976661 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_StreamWriter.WritePartialCharArray(writeLength: 2) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 425.9523390714286 < 443.5138263033333. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 66.5833354650193 (T) = (0 -424914557.2976373) / Math.Sqrt((6214968437211.075 / (145)) + (9982166583384.625 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.08761993798978365 = (465721002.6723264 - 424914557.2976373) / 465721002.6723264 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_StreamWriter.WriteFormat #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 225.6675251155342 < 261.280227219665. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 80.55200200361395 (T) = (0 -225.92407379520722) / Math.Sqrt((25.53440319102724 / (144)) + (8.349625563349136 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.19398543217478723 = (280.2977549212233 - 225.92407379520722) / 280.2977549212233 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_StreamWriter.WriteString(writeLength: 2) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 300.28263 < 325.5896936870238. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 154.09492626203607 (T) = (0 -300149986.00856525) / Math.Sqrt((6325341268117.214 / (142)) + (759489420794.0712 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.11948274492663304 = (340879164.2402919 - 300149986.00856525) / 340879164.2402919 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_StreamWriter.WriteString(writeLength: 100) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.2252168302 < 6.926171236306666. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 107.0275992787518 (T) = (0 -5200874625.437704) / Math.Sqrt((8523430008661932 / (144)) + (10211867278063544 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2914647443611866 = (7340318754.849552 - 5200874625.437704) / 7340318754.849552 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_StreamWriter.WriteCharArray(writeLength: 2) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 292.10424326666663 < 319.209842426282. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 99.72979695868271 (T) = (0 -293573222.3748901) / Math.Sqrt((9456379263044.635 / (143)) + (2880025970567.8115 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.12032640173767897 = (333729718.56243634 - 293573222.3748901) / 333729718.56243634 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Tests.Utf8JsonReaderCommentsTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
14.54 μs 3.15 μs 0.22 0.08 True
54.33 μs 46.83 μs 0.86 0.03 False
3.18 μs 2.15 μs 0.68 0.01 True
1.48 μs 562.78 ns 0.38 0.02 True
1.49 μs 554.38 ns 0.37 0.01 True
53.96 μs 46.22 μs 0.86 0.02 False
3.26 μs 2.23 μs 0.68 0.01 True
14.50 μs 2.47 μs 0.17 0.10 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Tests.Utf8JsonReaderCommentsTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Tests.Utf8JsonReaderCommentsTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: LongMultiLine) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.147868003681392 < 13.790343969106576. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 49.78044862284408 (T) = (0 -2791.7435498707146) / Math.Sqrt((7501070.2498038905 / (141)) + (205782.5750467464 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.813776451882287 = (14991.35623871819 - 2791.7435498707146) / 14991.35623871819 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 100, TestCase: LongMultiLine) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 46.83308055763474 < 51.58898031560965. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 26.166131081222336 (T) = (0 -46658.725374069116) / Math.Sqrt((25386160.95087942 / (144)) + (29986.638752813305 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.1910237614783693 = (57676.2618631864 - 46658.725374069116) / 57676.2618631864 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 100, TestCase: LongSingleLine) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.1548804736648752 < 3.0183832359352962. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 166.31057850272498 (T) = (0 -2187.3410516958597) / Math.Sqrt((91.36899664456443 / (143)) + (1047.9478460113457 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.31199131934949365 = (3179.2346713238376 - 2187.3410516958597) / 3179.2346713238376 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: LongSingleLine) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 562.7794280288051 < 1.406676471775875. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 388.7337439643831 (T) = (0 -557.2807946609163) / Math.Sqrt((59.12914378367643 / (141)) + (157.74288485561877 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.6243554919118569 = (1483.5323894317473 - 557.2807946609163) / 1483.5323894317473 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: LongSingleLine) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 554.3845308000762 < 1.4159147244226804. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 961.1647301912466 (T) = (0 -552.9185937243172) / Math.Sqrt((66.71109176311616 / (143)) + (14.505553924185513 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6288540834173525 = (1489.7606817700018 - 552.9185937243172) / 1489.7606817700018 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 100, TestCase: LongMultiLine) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 46.22486402945619 < 51.47527322872054. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 26.386204462152268 (T) = (0 -46417.655985449186) / Math.Sqrt((24838600.022471875 / (144)) + (34023.808175743005 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.19150368944059223 = (57412.32876292569 - 46417.655985449186) / 57412.32876292569 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 100, TestCase: LongSingleLine) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.228909561262228 < 3.0869888799327545. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 421.0865439085799 (T) = (0 -2230.0942224980026) / Math.Sqrt((290.7289157290455 / (145)) + (114.7661996662801 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.3131564543935793 = (3246.873668338881 - 2230.0942224980026) / 3246.873668338881 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: LongMultiLine) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.470791801905214 < 13.7984629668459. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 53.030236270745355 (T) = (0 -2733.165723879075) / Math.Sqrt((7306002.547395493 / (144)) + (79395.72592895768 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.8176142927377223 = (14985.635469498047 - 2733.165723879075) / 14985.635469498047 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Benchstone.BenchF.MatInv4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.34 ms 3.89 ms 0.53 0.01 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Benchstone.BenchF.MatInv4*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Benchstone.BenchF.MatInv4* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Benchstone.BenchF.MatInv4*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Benchstone.BenchF.MatInv4* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### Benchstone.BenchF.MatInv4.Test #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.888864844866071 < 6.98003416826637. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 542.8481913240647 (T) = (0 -3894368.8085182966) / Math.Sqrt((5118003021.9757595 / (143)) + (83090287.49839437 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.46397486941151717 = (7265272.813315316 - 3894368.8085182966) / 7265272.813315316 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Tests.Perf_Strings

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
56.80 ms 48.84 ms 0.86 0.02 True
40.83 ms 32.99 ms 0.81 0.02 True
207.05 ms 170.53 ms 0.82 0.05 True
59.48 ms 47.73 ms 0.80 0.09 True
213.60 ms 170.13 ms 0.80 0.02 True
56.09 ms 48.63 ms 0.87 0.06 True
39.73 ms 31.98 ms 0.80 0.06 True
207.53 ms 170.46 ms 0.82 0.07 True
57.86 ms 49.07 ms 0.85 0.03 True
40.49 ms 32.78 ms 0.81 0.01 True
206.26 ms 170.05 ms 0.82 0.01 True
39.54 ms 31.59 ms 0.80 0.03 True

graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Strings*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Tests.Perf_Strings* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Strings*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Tests.Perf_Strings* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: OneEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 48.83673971153846 < 54.08207493214286. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 97.69852063874518 (T) = (0 -48900085.04737027) / Math.Sqrt((458604953428.1599 / (144)) + (97730852348.73785 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.13820065968694054 = (56741845.5317066 - 48900085.04737027) / 56741845.5317066 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: NoneEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 32.99175613333333 < 38.63056417386752. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 72.67091795477974 (T) = (0 -33085318.34424091) / Math.Sqrt((158500033741.1088 / (143)) + (276279014471.52747 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.1824112164893455 = (40466942.56515538 - 33085318.34424091) / 40466942.56515538 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 170.53005891666666 < 196.73498776. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 48.52336126233372 (T) = (0 -169978126.78891027) / Math.Sqrt((6930875577484.651 / (144)) + (17303856189400.412 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.1841191546107795 = (208336949.8738768 - 169978126.78891027) / 208336949.8738768 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: True, Escaped: OneEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 47.72614428333333 < 53.08160231428571. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 64.7027997943808 (T) = (0 -47896579.47108517) / Math.Sqrt((1006728964992.3864 / (142)) + (222225624204.2539 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.13989758803420227 = (55687065.638632104 - 47896579.47108517) / 55687065.638632104 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: False, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 170.12934857142858 < 195.63095900315932. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.518833898299015 (T) = (0 -169558957.5862759) / Math.Sqrt((5009612938698.235 / (142)) + (10977899317189.828 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.1818974887823143 = (207258815.68790177 - 169558957.5862759) / 207258815.68790177 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: False, Escaped: OneEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 48.62779544642857 < 53.16776007184066. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.38078629896175 (T) = (0 -48207549.995125145) / Math.Sqrt((977242187698.7823 / (141)) + (323017919598.8561 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.13670203486801855 = (55841148.64414762 - 48207549.995125145) / 55841148.64414762 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: False, Escaped: NoneEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 31.984743740384616 < 37.6774422342361. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 67.6751407809776 (T) = (0 -32021021.12006028) / Math.Sqrt((280621308187.59406 / (144)) + (305142832929.03516 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.18875578163641812 = (39471493.781061575 - 32021021.12006028) / 39471493.781061575 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 170.46189914285713 < 196.46190182170326. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 47.69007822401068 (T) = (0 -170213223.70663616) / Math.Sqrt((10086801591244.205 / (142)) + (17441164609254.086 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.1845420259691359 = (208733287.4620879 - 170213223.70663616) / 208733287.4620879 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: OneEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 49.06867034615385 < 54.25805953124999. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 84.31317098686048 (T) = (0 -49166109.36168346) / Math.Sqrt((641672265805.7761 / (141)) + (116494160021.75124 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.1360599035323397 = (56909164.83990726 - 49166109.36168346) / 56909164.83990726 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: NoneEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 32.777397009615385 < 38.404382035000005. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 109.19232622015588 (T) = (0 -32883948.483762648) / Math.Sqrt((173499173575.92963 / (143)) + (102357541765.97308 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.18422333794991053 = (40309989.25750531 - 32883948.483762648) / 40309989.25750531 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: True, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 170.04557014285714 < 195.4674504782738. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 80.11256233225853 (T) = (0 -168971692.66109896) / Math.Sqrt((3792097902614.174 / (142)) + (5978748624162.154 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.1839342859436363 = (207056478.1127767 - 168971692.66109896) / 207056478.1127767 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: True, Escaped: NoneEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 31.593738901785713 < 37.568914735625. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 141.45462374541762 (T) = (0 -31761530.72021978) / Math.Sqrt((218613052794.5454 / (141)) + (38818141401.858574 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.19193573207368167 = (39305698.792655796 - 31761530.72021978) / 39305698.792655796 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
949.72 μs 591.36 μs 0.62 0.01 True
1.43 ms 1.07 ms 0.74 0.01 True
865.17 μs 577.98 μs 0.67 0.01 True
767.73 μs 564.24 μs 0.73 0.06 True
595.96 μs 314.82 μs 0.53 0.01 False
708.21 μs 435.09 μs 0.61 0.01 True
915.47 μs 362.60 μs 0.40 0.01 False
501.41 μs 255.25 μs 0.51 0.01 False
875.94 μs 713.47 μs 0.81 0.02 False
669.52 μs 365.88 μs 0.55 0.01 False

graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Perf_Utf8Encoding* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Perf_Utf8Encoding* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Perf_Utf8Encoding.GetString(Input: Greek) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 591.3615509259259 < 904.2275540537128. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 179.20376852706687 (T) = (0 -595227.8389850003) / Math.Sqrt((431973338.312064 / (144)) + (43514434.246857546 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.38842549300832707 = (973271.1749430471 - 595227.8389850003) / 973271.1749430471 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Utf8Encoding.GetString(Input: EnglishMostlyAscii) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.065562576111111 < 1.3618772154931005. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 306.475829093037 (T) = (0 -1070520.6443453906) / Math.Sqrt((126167140.9451104 / (143)) + (17926097.90731131 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.25830446924974676 = (1443342.4497819452 - 1070520.6443453906) / 1443342.4497819452 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Utf8Encoding.GetString(Input: Chinese) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 577.9767658950617 < 816.8343426033401. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 500.28556657762283 (T) = (0 -577047.402686827) / Math.Sqrt((34092404.81957307 / (145)) + (2966061.697053011 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.33379523920810455 = (866171.2384056066 - 577047.402686827) / 866171.2384056066 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishAllAscii) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 564.2445885416666 < 739.3049150989584. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 78.87902346185516 (T) = (0 -568568.3298505814) / Math.Sqrt((382780051.5137451 / (143)) + (129558489.19302252 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2684345078342706 = (777194.0256058134 - 568568.3298505814) / 777194.0256058134 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Utf8Encoding.GetBytes(Input: Chinese) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 314.8228329464286 < 566.2619387905091. IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.946510126501295 (T) = (0 -316727.4509627294) / Math.Sqrt((10482646724.245806 / (143)) + (25170908.981945578 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.48719861245552554 = (617641.5638798561 - 316727.4509627294) / 617641.5638798561 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Utf8Encoding.GetString(Input: Cyrillic) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 435.0868101273149 < 671.8805642244474. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 196.35001676214654 (T) = (0 -435856.6640006725) / Math.Sqrt((169596694.16042504 / (143)) + (28019591.948877916 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3961071830368432 = (721745.0709092702 - 435856.6640006725) / 721745.0709092702 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishMostlyAscii) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 362.5973954457364 < 868.8032829901961. IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.78184595336823 (T) = (0 -366109.7057089768) / Math.Sqrt((46755548816.21579 / (144)) + (9275182.936737554 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.6174775081507934 = (957093.2781994425 - 366109.7057089768) / 957093.2781994425 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Utf8Encoding.GetBytes(Input: Cyrillic) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 255.25026529003785 < 473.4913622102864. IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.61261052895219 (T) = (0 -253923.2524471739) / Math.Sqrt((8895988712.631056 / (144)) + (2132996.440427329 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5100525419230922 = (518266.2921527295 - 253923.2524471739) / 518266.2921527295 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Utf8Encoding.GetString(Input: EnglishAllAscii) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 713.4710008741258 < 831.3441619328703. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 30.29655476928219 (T) = (0 -720103.6510513008) / Math.Sqrt((4188135151.6316524 / (144)) + (33934764.807473 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.18782810722679005 = (886639.463220604 - 720103.6510513008) / 886639.463220604 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Utf8Encoding.GetBytes(Input: Greek) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 365.88213468992245 < 632.6964761938864. IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 31.27078385390232 (T) = (0 -364169.81866629486) / Math.Sqrt((15692832064.0904 / (143)) + (9027218.782516366 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.4738963595633903 = (692201.6703098137 - 364169.81866629486) / 692201.6703098137 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Buffers.Binary.Tests.BinaryReadAndWriteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
196.09 ns 147.10 ns 0.75 0.01 True
216.66 ns 158.20 ns 0.73 0.01 True
51.75 ns 34.90 ns 0.67 0.01 True
82.50 ns 53.31 ns 0.65 0.01 True
39.45 ns 21.22 ns 0.54 0.02 True
105.88 ns 68.19 ns 0.64 0.01 True

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Buffers.Binary.Tests.BinaryReadAndWriteTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Buffers.Binary.Tests.BinaryReadAndWriteTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldLE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 147.1046475430598 < 186.3737044310281. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 621.0591454563369 (T) = (0 -146.97475384885902) / Math.Sqrt((0.5322084850779488 / (143)) + (0.07454187059056838 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.24975558343567508 = (195.90249604511067 - 146.97475384885902) / 195.90249604511067 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldBE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 158.19895791733651 < 205.4864816082275. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 314.89521617849726 (T) = (0 -158.0769503423006) / Math.Sqrt((0.5530599844512503 / (144)) + (0.9231833539618249 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.27039920028865433 = (216.66224928048473 - 158.0769503423006) / 216.66224928048473 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructAndReverseBE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 34.897909569381355 < 49.20114678473694. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 354.51401654747224 (T) = (0 -35.07609587415265) / Math.Sqrt((0.11789122574084618 / (145)) + (0.04400222810244582 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.3254999807500848 = (52.003105816304334 - 35.07609587415265) / 52.003105816304334 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 53.31480938407803 < 78.8510702765013. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 287.90292826044356 (T) = (0 -52.112352246881386) / Math.Sqrt((0.1003354289697772 / (146)) + (0.3246334368835222 / (30))) is greater than 1.9736914397558152 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (146) + (30) - 2, .975) and 0.37212293273979946 = (82.99769965206475 - 52.112352246881386) / 82.99769965206475 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructAndReverseLE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.223462180009644 < 37.439543847154596. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 479.4705846900561 (T) = (0 -21.286205691900047) / Math.Sqrt((0.14527185811559576 / (143)) + (0.01251119532534926 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.4602354683968223 = (39.436095640958435 - 21.286205691900047) / 39.436095640958435 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 68.19367771725044 < 100.60538587245185. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 263.52047589599744 (T) = (0 -67.2439364295553) / Math.Sqrt((0.971434378140827 / (144)) + (0.4564269115898303 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.36738417623765535 = (106.29505918716458 - 67.2439364295553) / 106.29505918716458 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Document.Tests.Perf_ParseThenWrite

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
16.18 μs 15.04 μs 0.93 0.02 True
81.16 μs 75.23 μs 0.93 0.01 True
19.73 μs 17.80 μs 0.90 0.01 True
88.62 μs 80.65 μs 0.91 0.01 True
9.40 μs 8.40 μs 0.89 0.01 True
83.95 μs 69.09 μs 0.82 0.01 True
9.77 μs 9.01 μs 0.92 0.01 True
8.56 ms 7.82 ms 0.91 0.01 True
96.66 μs 83.66 μs 0.87 0.01 True
18.06 μs 16.15 μs 0.89 0.01 True
1.08 μs 1.00 μs 0.93 0.03 True
1.15 μs 1.06 μs 0.92 0.02 True
73.85 μs 59.84 μs 0.81 0.01 True
14.83 μs 13.70 μs 0.92 0.02 True
8.04 ms 7.27 ms 0.90 0.01 True
106.96 μs 93.50 μs 0.87 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_ParseThenWrite*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Document.Tests.Perf_ParseThenWrite* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_ParseThenWrite*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Document.Tests.Perf_ParseThenWrite* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.036021276005032 < 15.520309058757332. IsChangePoint: Marked as a change because one of 9/19/2023 3:50:13 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 22.52472995785213 (T) = (0 -15023.915153854057) / Math.Sqrt((38249.52873650877 / (144)) + (42106.88397951844 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.057717782984294586 = (15944.177744790919 - 15023.915153854057) / 15944.177744790919 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json4KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 75.23348782467532 < 77.25864465498726. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 78.57740523804627 (T) = (0 -75119.07808513958) / Math.Sqrt((326108.141744695 / (142)) + (133793.54847710123 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.07917359098188487 = (81577.89280309595 - 75119.07808513958) / 81577.89280309595 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.801579604261796 < 18.757002791454916. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 67.85387151426636 (T) = (0 -17831.606161701817) / Math.Sqrt((50638.461154545395 / (145)) + (5281.905478102486 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.08021775905911487 = (19386.769354734544 - 17831.606161701817) / 19386.769354734544 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json4KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 80.64909997534518 < 83.52637922126021. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 87.82034158417146 (T) = (0 -80626.49095833614) / Math.Sqrt((504619.69131366763 / (144)) + (48081.348097511815 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.07221821847922173 = (86902.42960599724 - 80626.49095833614) / 86902.42960599724 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.39715828402367 < 8.709821053287863. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 69.58939281230963 (T) = (0 -8431.030299325137) / Math.Sqrt((5177.965441635564 / (140)) + (1749.8007795916678 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.07457253992582541 = (9110.417253719028 - 8431.030299325137) / 9110.417253719028 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: DeepTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 69.08671344948378 < 79.03244668597219. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 245.47970714867927 (T) = (0 -68987.22487135962) / Math.Sqrt((239858.92375881897 / (144)) + (43068.76215159677 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.16538809751609845 = (82657.84931420899 - 68987.22487135962) / 82657.84931420899 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.014551567629315 < 9.33282784580844. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 63.75678746531397 (T) = (0 -9025.608640016582) / Math.Sqrt((7397.23827572372 / (143)) + (1865.0316800531532 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.0701035506275903 = (9706.036243183848 - 9025.608640016582) / 9706.036243183848 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.820858875576039 < 8.117995994073784. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 97.81546503283907 (T) = (0 -7796390.544459526) / Math.Sqrt((4064066717.0305076 / (144)) + (352652864.4088912 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.0734970780424872 = (8414858.021156948 - 7796390.544459526) / 8414858.021156948 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: BroadTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 83.66264417016808 < 91.68281833972391. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 195.00482930606202 (T) = (0 -83793.90172959032) / Math.Sqrt((249702.1056151881 / (143)) + (60056.945587453505 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.12470680796623249 = (95732.3814376905 - 83793.90172959032) / 95732.3814376905 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.1495971749226 < 17.161918139733753. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.91415765904237 (T) = (0 -16210.407481513766) / Math.Sqrt((20395.68420441915 / (143)) + (18884.605752509065 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.09313616053220815 = (17875.238570574293 - 16210.407481513766) / 17875.238570574293 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.0030716975662701 < 1.0315926168084868. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 27.400944813241185 (T) = (0 -1011.5086596672137) / Math.Sqrt((219.9428101491398 / (143)) + (163.44193655702747 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.06681613728704293 = (1083.9328669127972 - 1011.5086596672137) / 1083.9328669127972 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.0552229637396833 < 1.092936950699272. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 29.798107287706443 (T) = (0 -1062.3373096589203) / Math.Sqrt((133.15173900451964 / (143)) + (161.22313680211758 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.06579870831970702 = (1137.1610370481897 - 1062.3373096589203) / 1137.1610370481897 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: DeepTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 59.835496579091405 < 70.6119105560021. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 384.3627338703659 (T) = (0 -59920.730114199214) / Math.Sqrt((71090.87955602678 / (141)) + (24878.33600795409 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.18978285907224673 = (73956.38414362098 - 59920.730114199214) / 73956.38414362098 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.702434360507246 < 14.150166437128403. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.59098516595811 (T) = (0 -13688.872035398761) / Math.Sqrt((27915.840168553237 / (142)) + (4566.821487852995 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.07519074571487173 = (14801.8328882102 - 13688.872035398761) / 14801.8328882102 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.2711268549783545 < 7.552529647586805. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 124.94592769716992 (T) = (0 -7273687.145162279) / Math.Sqrt((2026230068.9154036 / (142)) + (320012648.41482604 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.07900270697884772 = (7897620.547072798 - 7273687.145162279) / 7897620.547072798 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: BroadTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 93.5014744632228 < 100.75724223102631. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 114.57599945576968 (T) = (0 -93612.05316144704) / Math.Sqrt((828529.4441155636 / (143)) + (117495.77056452549 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.10762843253164059 = (104902.55020901505 - 93612.05316144704) / 104902.55020901505 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
450.40 ns 393.55 ns 0.87 0.05 True
349.73 ns 319.16 ns 0.91 0.02 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Encodings.Web.Tests.Perf_Encoders* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Encodings.Web.Tests.Perf_Encoders* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,&lorem ipsum=dolor sit amet,16) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 393.5500892567608 < 429.95904873285343. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 77.22872204806107 (T) = (0 -395.0473079769386) / Math.Sqrt((52.14884430043896 / (144)) + (2.382422721775316 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.11496941330472688 = (446.3657120055651 - 395.0473079769386) / 446.3657120055651 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: JavaScript,&Hello+<World>!,16) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 319.1628632386395 < 332.8783318657033. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 70.65182540607942 (T) = (0 -319.98372649099394) / Math.Sqrt((16.83100080008319 / (145)) + (1.4690102964508092 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.08231648526525395 = (348.6863622950494 - 319.98372649099394) / 348.6863622950494 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
5.04 μs 1.34 μs 0.27 0.05 False
1.02 μs 893.31 ns 0.88 0.06 False
  • [EscapeDataString - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 22.04_LLVM=false_MonoAOT=true_MonoInterpreter=false_RunKind=micro_mono/System.Tests.Perf_Uri.EscapeDataString(input%3a%20%22%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b.html>)
  • 📝 - Benchmark Source
  • 📈 - ADX Test Multi Config Graph
28.71 μs 22.95 μs 0.80 0.01 False
1.49 μs 1.30 μs 0.87 0.03 True
53.08 μs 3.09 μs 0.06 0.13 False
2.50 μs 470.93 ns 0.19 0.03 False
593.66 ns 544.19 ns 0.92 0.01 True
6.41 μs 2.33 μs 0.36 0.04 False
944.06 ns 829.14 ns 0.88 0.06 False
5.90 μs 1.94 μs 0.33 0.07 False
  • [EscapeDataString - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 22.04_LLVM=false_MonoAOT=true_MonoInterpreter=false_RunKind=micro_mono/System.Tests.Perf_Uri.EscapeDataString(input%3a%20%22a%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bc.html>)
  • 📝 - Benchmark Source
  • 📈 - ADX Test Multi Config Graph
106.88 μs 37.91 μs 0.35 0.02 False
1.14 μs 953.33 ns 0.84 0.12 True
302.53 ns 278.76 ns 0.92 0.05 True
1.13 μs 934.67 ns 0.83 0.09 False
654.26 ns 593.58 ns 0.91 0.01 False
593.35 ns 500.61 ns 0.84 0.15 False
625.87 ns 533.36 ns 0.85 0.05 False
1.49 μs 1.31 μs 0.88 0.03 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Uri*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Uri* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Uri*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Uri* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path with escapable values?key=va lue") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.3442582177913232 < 4.784353488428316. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.090361025170537 (T) = (0 -1500.48035993298) / Math.Sqrt((3365731.7130017853 / (143)) + (129930.58321167383 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.47370998508451106 = (2851.052304638395 - 1500.48035993298) / 2851.052304638395 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://contoso.com/path/with?key=value#fragment") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 893.313572058238 < 0.9653333728605825. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 11.278970392094962 (T) = (0 -903.810249377499) / Math.Sqrt((1778.6289582491308 / (144)) + (730.4220757580783 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.07028597113288351 = (972.137906189087 - 903.810249377499) / 972.137906189087 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.EscapeDataString(input: "{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 22.945891956453636 < 27.32441559772421. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/27/2023 6:04:31 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 12.13902926385725 (T) = (0 -22998.165934583612) / Math.Sqrt((7456236.721087675 / (149)) + (4185.9272245359725 / (22))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (149) + (22) - 2, .975) and 0.1057848688453891 = (25718.828873862123 - 22998.165934583612) / 25718.828873862123 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://xn--hst-sna.with.xn--nicode-2ya/path/with?key=value#fragment") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.303043948363021 < 1.42386652598674. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.88739887679206 (T) = (0 -1308.6160526281217) / Math.Sqrt((977.9402042756917 / (145)) + (451.4990816698456 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.10500504701491847 = (1462.1490861635448 - 1308.6160526281217) / 1462.1490861635448 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.EscapeDataString(input: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.0870579787234043 < 50.49504155341451. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.434213001065496 (T) = (0 -5228.815333508047) / Math.Sqrt((619139319.4736214 / (143)) + (18587505.59848535 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.782062713005084 = (23992.293405166707 - 5228.815333508047) / 23992.293405166707 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.BuilderToString #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 470.92922923150957 < 2.3817717259558253. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.454272126346817 (T) = (0 -569.0436599114059) / Math.Sqrt((966367.4159775651 / (142)) + (42708.155168659345 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.5740574978770951 = (1335.9635562905378 - 569.0436599114059) / 1335.9635562905378 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.Ctor(input: "http://xn--hst-sna.with.xn--nicode-2ya") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 544.1907429171564 < 563.5087016248502. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 35.822437612647875 (T) = (0 -534.489326454041) / Math.Sqrt((153.77961194023675 / (141)) + (87.43282671126585 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.11826562997891628 = (606.1795305101473 - 534.489326454041) / 606.1795305101473 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path%20with%20escapable%20values?key=%C3%BCnicode") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.3349408780427225 < 6.096028168355083. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 9.83087700784196 (T) = (0 -2520.4555915420283) / Math.Sqrt((3534443.3456906057 / (143)) + (141488.30024058645 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.40089516668636677 = (4207.035983337764 - 2520.4555915420283) / 4207.035983337764 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://dot.net/path/with?key=value#fragment") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 829.1439049642338 < 907.3376198727043. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 12.12938431926339 (T) = (0 -842.6521270900279) / Math.Sqrt((1996.984934070692 / (143)) + (175.77782453584337 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.060229593334552564 = (896.6574400655786 - 842.6521270900279) / 896.6574400655786 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path with escapable values?key=ünicode") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.9405172287784684 < 5.627694200988108. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 9.351846585033226 (T) = (0 -2142.50279576701) / Math.Sqrt((3491290.8415198936 / (144)) + (145589.08602962532 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.4267941706891564 = (3737.7547230161767 - 2142.50279576701) / 3737.7547230161767 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.EscapeDataString(input: "a{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{ü") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 37.91017894975787 < 101.47795704754574. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 7.898572572480086 (T) = (0 -41627.50971689288) / Math.Sqrt((1146710694.2666633 / (143)) + (60452145.21072021 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3754073076169139 = (66647.44916893962 - 41627.50971689288) / 66647.44916893962 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.Ctor(input: "http://höst.with.ünicode") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 953.3305720706917 < 1.101355373299713. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 39.17986104832363 (T) = (0 -982.7695264872766) / Math.Sqrt((1189.745650143802 / (144)) + (464.7196343777118 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.16268806313379222 = (1173.71971330717 - 982.7695264872766) / 1173.71971330717 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.Ctor(input: "http://dot.net") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 278.7576600651262 < 285.551508583038. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.14370340051424 (T) = (0 -274.6234517481626) / Math.Sqrt((33.848411217096476 / (140)) + (6.283371509719952 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.08533757256585921 = (300.2456901160254 - 274.6234517481626) / 300.2456901160254 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://CONTOSO.com/path/with?key=value#fragment") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 934.6739298334552 < 1.0541785932857688. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 15.439461354550863 (T) = (0 -947.9341201624007) / Math.Sqrt((3842.4976644666895 / (143)) + (435.70442566861954 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.09485126516739414 = (1047.2689003290795 - 947.9341201624007) / 1047.2689003290795 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.Ctor(input: "https://a.much.longer.domain.name") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 593.5750076245664 < 618.5101263768266. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 31.51940548947958 (T) = (0 -593.6545617462424) / Math.Sqrt((408.8356396610915 / (142)) + (75.82020485296177 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.10988693426706019 = (666.9428689459955 - 593.6545617462424) / 666.9428689459955 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 500.6084730021524 < 569.2506066268431. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.861016096093971 (T) = (0 -514.2813197611904) / Math.Sqrt((1359.8578661794943 / (143)) + (335.08388956043586 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.0726585581597505 = (554.5760132757921 - 514.2813197611904) / 554.5760132757921 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CombineAbsoluteRelative(input: "/new/path") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 533.3562003731936 < 594.9900920801216. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 17.58343217988584 (T) = (0 -538.7233523102848) / Math.Sqrt((916.012177290405 / (144)) + (88.39872365115278 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.09055989958835271 = (592.3681527419321 - 538.7233523102848) / 592.3681527419321 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://a.much.longer.domain.name/path/with?key=value#fragment") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.30734629854708 < 1.4190996756193226. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 39.083140629676244 (T) = (0 -1314.8625961022337) / Math.Sqrt((960.9840787448392 / (144)) + (269.25955855633043 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.10521302858195462 = (1469.4699834737855 - 1314.8625961022337) / 1469.4699834737855 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
583.42 μs 140.82 μs 0.24 0.08 True
578.60 μs 139.79 μs 0.24 0.09 True
582.94 μs 137.43 μs 0.24 0.11 True
585.22 μs 102.90 μs 0.18 0.06 True
578.66 μs 97.81 μs 0.17 0.01 True
580.56 μs 101.75 μs 0.18 0.07 True
584.32 μs 130.40 μs 0.22 0.09 True

graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsFalse<Int32>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsFalse<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsFalse<Int32>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsFalse<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsFalse<Int32>.Array(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 140.81603222597596 < 553.9605715492726. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 134.7053090861383 (T) = (0 -118237.18679534433) / Math.Sqrt((1424083.434579817 / (145)) + (355877819.79983014 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.7969768571344967 = (582382.8019137351 - 118237.18679534433) / 582382.8019137351 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<Int32>.ICollection(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 139.79349848533167 < 551.6658751215278. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 135.51616925356166 (T) = (0 -119647.21434862945) / Math.Sqrt((1242297.872237243 / (144)) + (347420845.93136615 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7940618553461246 = (580986.1720844532 - 119647.21434862945) / 580986.1720844532 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<Int32>.Stack(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 137.4309417763158 < 553.4050570910492. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 222.259202077514 (T) = (0 -128437.1599247561) / Math.Sqrt((1318366.519648325 / (144)) + (124412742.53384688 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7791484336970251 = (581554.2179517974 - 128437.1599247561) / 581554.2179517974 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<Int32>.Queue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 102.90014440789473 < 555.5140087114197. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 155.54306430461912 (T) = (0 -112687.99054642238) / Math.Sqrt((1323942.9732224639 / (146)) + (275736121.9562757 / (30))) is greater than 1.9736914397558152 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (146) + (30) - 2, .975) and 0.8071997240251588 = (584480.4421396535 - 112687.99054642238) / 584480.4421396535 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<Int32>.Span(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 97.80790132211538 < 548.4427455787036. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 3218.024691640085 (T) = (0 -97020.26570467472) / Math.Sqrt((1392979.2973814248 / (144)) + (382075.27672269655 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.8323624432173639 = (578750.1772676996 - 97020.26570467472) / 578750.1772676996 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<Int32>.List(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 101.75322530148422 < 551.0065259250991. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 211.70247126388185 (T) = (0 -108998.81244922958) / Math.Sqrt((1159490.2840184392 / (146)) + (148784328.97165293 / (30))) is greater than 1.9736914397558152 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (146) + (30) - 2, .975) and 0.8123412099382105 = (580835.101906711 - 108998.81244922958) / 580835.101906711 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<Int32>.ImmutableArray(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 130.39806432506887 < 555.6384512402066. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 177.4100809807884 (T) = (0 -123098.43196334426) / Math.Sqrt((1249163.4533096529 / (143)) + (203111145.29377973 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7895811064085445 = (585016.0594530518 - 123098.43196334426) / 585016.0594530518 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
51.46 ns 21.65 ns 0.42 0.01 True
72.12 ns 26.44 ns 0.37 0.01 True
129.12 ns 67.07 ns 0.52 0.01 True
50.41 ns 20.86 ns 0.41 0.01 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf<Single>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf<Single>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf<Single>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf<Single>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_VectorOf<Single>.LessThanOrEqualBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.645287856327457 < 48.860510089747365. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 812.076513075415 (T) = (0 -21.582546390024202) / Math.Sqrt((0.008690491277656092 / (142)) + (0.0384427980468455 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.5796035488229272 = (51.33855514145037 - 21.582546390024202) / 51.33855514145037 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorOf<Single>.AbsBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 26.437872686150733 < 68.59516472121128. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1449.3933470570696 (T) = (0 -26.439508905293355) / Math.Sqrt((0.06687713664836467 / (143)) + (0.015882397859579723 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6338348410002409 = (72.20651188528494 - 26.439508905293355) / 72.20651188528494 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorOf<Single>.EqualsBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 67.06868352154554 < 122.50107592553181. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1238.5729759673327 (T) = (0 -66.99550402223433) / Math.Sqrt((0.08088599702375077 / (141)) + (0.05823567789826671 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.48108861624338756 = (129.1077939690326 - 66.99550402223433) / 129.1077939690326 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorOf<Single>.LessThanBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.864353769920637 < 47.94111249993515. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1616.379459710017 (T) = (0 -20.855188928926914) / Math.Sqrt((0.016035848227584765 / (141)) + (0.0066375892600834315 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.5865266272238776 = (50.439013252297336 - 20.855188928926914) / 50.439013252297336 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Constructor

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
23.43 ns 15.30 ns 0.65 0.04 True
23.67 ns 15.32 ns 0.65 0.01 True
23.65 ns 15.41 ns 0.65 0.03 True
23.71 ns 15.43 ns 0.65 0.04 True
23.18 ns 15.44 ns 0.67 0.02 True
23.71 ns 15.46 ns 0.65 0.02 True
23.41 ns 15.34 ns 0.66 0.01 True
23.65 ns 15.36 ns 0.65 0.05 True
23.43 ns 15.45 ns 0.66 0.04 True
23.41 ns 15.62 ns 0.67 0.02 True

graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Constructor*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Constructor* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Constructor*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Constructor* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Constructor.ConstructorBenchmark_Byte #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.301390803205013 < 22.286212827618616. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 68.84186552179727 (T) = (0 -15.425671636437178) / Math.Sqrt((1.941942837229495 / (143)) + (0.018970642363444402 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3472742681668774 = (23.632700358105296 - 15.425671636437178) / 23.632700358105296 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Constructor.ConstructorBenchmark_UInt16 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.315051690980729 < 22.342916113377687. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 267.97066556878315 (T) = (0 -15.465660805478251) / Math.Sqrt((0.0533486327284068 / (144)) + (0.01578111810796717 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.34158407278937997 = (23.48919606334334 - 15.465660805478251) / 23.48919606334334 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Constructor.ConstructorBenchmark_Int16 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.408359456303382 < 22.462279511691342. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 15.95590835203652 (T) = (0 -15.902332065078292) / Math.Sqrt((0.9782103469212389 / (143)) + (6.682589771373677 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3246773249157145 = (23.547753765403417 - 15.902332065078292) / 23.547753765403417 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Constructor.ConstructorBenchmark_Single #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.433664123024279 < 22.29454032391322. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 95.66302403645346 (T) = (0 -15.454609451448537) / Math.Sqrt((0.9858560171186964 / (145)) + (0.010971650478397207 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.34380846704278806 = (23.551979376814483 - 15.454609451448537) / 23.551979376814483 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Constructor.ConstructorBenchmark_Double #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.442573814195008 < 22.16137655541341. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 241.90647648441103 (T) = (0 -15.352805955692675) / Math.Sqrt((0.08050799637582738 / (144)) + (0.017106715131684363 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.3461914660720421 = (23.482113124856177 - 15.352805955692675) / 23.482113124856177 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Constructor.ConstructorBenchmark_SByte #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.455774228979847 < 22.210252603751368. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 95.64056188830622 (T) = (0 -15.418017213310348) / Math.Sqrt((0.9834804717512241 / (143)) + (0.010482128301045487 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.34526620744391223 = (23.548528254694542 - 15.418017213310348) / 23.548528254694542 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Constructor.ConstructorBenchmark_Int32 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.343043531971439 < 22.248027147824825. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 293.15053268209755 (T) = (0 -15.418313946492384) / Math.Sqrt((0.04913988018485052 / (142)) + (0.012373415931056515 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.343676751497507 = (23.491951537099663 - 15.418313946492384) / 23.491951537099663 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Constructor.ConstructorBenchmark_UInt64 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.359996198487018 < 22.30949167925052. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 15.870824548478165 (T) = (0 -15.879829399122785) / Math.Sqrt((1.032323509333403 / (144)) + (6.783849741615463 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.32557048413958534 = (23.545573000113777 - 15.879829399122785) / 23.545573000113777 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Constructor.ConstructorBenchmark_Int64 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.453494235124118 < 22.345763855872406. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 264.7077262157116 (T) = (0 -15.407984838404117) / Math.Sqrt((0.09699748325995688 / (144)) + (0.00792579462847507 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.34473730061590063 = (23.514210183009858 - 15.407984838404117) / 23.514210183009858 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Constructor.ConstructorBenchmark_UInt32 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.616120960022553 < 22.36050907683487. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 288.76108410757905 (T) = (0 -15.483562805518067) / Math.Sqrt((0.06800442084452367 / (145)) + (0.008975639350169009 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.34075758382400956 = (23.48690318704341 - 15.483562805518067) / 23.48690318704341 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.Concurrent.IsEmpty<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.91 ns 26.46 ns 0.72 0.05 False
403.33 ns 358.09 ns 0.89 0.03 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Concurrent.IsEmpty<String>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Concurrent.IsEmpty<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Concurrent.IsEmpty<String>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Concurrent.IsEmpty<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.Concurrent.IsEmpty<String>.Dictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 26.460481314348574 < 35.06223524328505. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 36.7868949150895 (T) = (0 -26.700072862636436) / Math.Sqrt((19.838481574058964 / (144)) + (0.5428413926528514 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.35230629793481244 = (41.223301658642946 - 26.700072862636436) / 41.223301658642946 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Concurrent.IsEmpty<String>.Dictionary(Size: 0) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 358.0939433251934 < 384.45198328587287. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 54.12667983071933 (T) = (0 -358.06760131099537) / Math.Sqrt((12.711710990184507 / (145)) + (20.186344604820093 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.116473810087739 = (405.27106655044764 - 358.06760131099537) / 405.27106655044764 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Half

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
28.79 ns 6.39 ns 0.22 0.02 True
71.07 ns 31.03 ns 0.44 0.01 True
67.66 ns 29.44 ns 0.44 0.03 True
66.66 ns 31.17 ns 0.47 0.01 True
28.95 ns 6.35 ns 0.22 0.02 True
67.70 ns 30.05 ns 0.44 0.01 True
28.85 ns 6.36 ns 0.22 0.02 True

graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Half*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Half* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Half*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Half* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Half.HalfToSingle(value: 12344) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.392143438923023 < 27.433909470324405. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 610.2481938070865 (T) = (0 -6.422435001003387) / Math.Sqrt((0.009794616793889883 / (143)) + (0.03863958578579077 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7777560636756791 = (28.898133767893295 - 6.422435001003387) / 28.898133767893295 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.SingleToHalf(value: 65520) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 31.031555051245217 < 67.54749319986946. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 478.6699124709351 (T) = (0 -31.11449959271475) / Math.Sqrt((0.10265873557497406 / (144)) + (0.18691285190319434 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5617705504469522 = (71.00047617623274 - 31.11449959271475) / 71.00047617623274 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.SingleToHalf(value: 6.097555E-05) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 29.444497569650153 < 64.29829165034559. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 275.1262076512875 (T) = (0 -29.289171062659243) / Math.Sqrt((0.10032093094105447 / (144)) + (0.5634443106047388 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.567284652972019 = (67.68692458870726 - 29.289171062659243) / 67.68692458870726 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.SingleToHalf(value: 12345) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 31.170894603760853 < 63.26093738842572. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 559.7353119724144 (T) = (0 -30.785530881904506) / Math.Sqrt((0.048776253543024596 / (141)) + (0.1122021212949758 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.5375106426538968 = (66.56484174805821 - 30.785530881904506) / 66.56484174805821 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.HalfToSingle(value: 6.1E-05) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.350724790576239 < 27.49917734584906. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 714.1271983351825 (T) = (0 -6.384468840201379) / Math.Sqrt((0.01050014831710248 / (143)) + (0.027649647356987266 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7791724568459683 = (28.911560347107972 - 6.384468840201379) / 28.911560347107972 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.SingleToHalf(value: NaN) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 30.054746676034352 < 64.6546941405071. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 475.62178113843316 (T) = (0 -30.27808412555265) / Math.Sqrt((0.020356202136994083 / (141)) + (0.18544171805865434 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.5554303110071841 = (68.10649685575378 - 30.27808412555265) / 68.10649685575378 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.HalfToSingle(value: NaN) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.360845675401264 < 27.49788987178933. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 180.27126707422164 (T) = (0 -6.4956205670972444) / Math.Sqrt((0.004969625091692257 / (144)) + (0.46338421049053496 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7754338580134694 = (28.92519998623324 - 6.4956205670972444) / 28.92519998623324 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
35.54 ns 19.81 ns 0.56 0.05 True
52.23 ns 26.30 ns 0.50 0.03 True
35.55 ns 20.29 ns 0.57 0.01 True
26.72 ns 16.91 ns 0.63 0.02 False
55.69 ns 33.75 ns 0.61 0.02 True
58.32 ns 35.44 ns 0.61 0.02 True
50.39 ns 24.57 ns 0.49 0.02 True

graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Microsoft.Extensions.Primitives.StringSegmentBenchmark* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Microsoft.Extensions.Primitives.StringSegmentBenchmark* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.Primitives.StringSegmentBenchmark.StartsWith #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.811202422590092 < 33.76403214927441. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 125.32505979935758 (T) = (0 -19.95640465225749) / Math.Sqrt((0.09384664449933933 / (144)) + (0.45067323481106175 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.44016058183239626 = (35.64665867504702 - 19.95640465225749) / 35.64665867504702 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Primitives.StringSegmentBenchmark.Equals_String #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 26.298891308407438 < 49.41444232271965. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 248.05380936533516 (T) = (0 -26.530868995765154) / Math.Sqrt((0.13740367641086762 / (144)) + (0.29051764637740357 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.4909211799027364 = (52.11544450169076 - 26.530868995765154) / 52.11544450169076 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Primitives.StringSegmentBenchmark.EndsWith #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.286407673273008 < 33.898496631116. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 482.2368366171203 (T) = (0 -20.3080425106938) / Math.Sqrt((0.05355386532107525 / (143)) + (0.0193349732321241 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.4311777489449682 = (35.701912984288406 - 20.3080425106938) / 35.701912984288406 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Primitives.StringSegmentBenchmark.LastIndexOf #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.9105020318387 < 25.65573556950926. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 35.91382430894717 (T) = (0 -16.802225432509562) / Math.Sqrt((11.768931307045676 / (144)) + (0.16231225907907748 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.3868639320422737 = (27.403746591642395 - 16.802225432509562) / 27.403746591642395 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Primitives.StringSegmentBenchmark.Equals_Valid #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 33.745007817627304 < 53.09552830525226. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 173.10272060659665 (T) = (0 -32.804598243987506) / Math.Sqrt((0.13048176785656507 / (144)) + (0.5177072734613578 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.4155988919736061 = (56.1337030225239 - 32.804598243987506) / 56.1337030225239 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Primitives.StringSegmentBenchmark.Equals_Object_Valid #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 35.4357305495327 < 55.366347150028666. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 172.06205418038166 (T) = (0 -35.69028052360593) / Math.Sqrt((0.2994722971191239 / (141)) + (0.4607593323102861 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.3892897026179694 = (58.44060707114593 - 35.69028052360593) / 58.44060707114593 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Primitives.StringSegmentBenchmark.GetSegmentHashCode #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.571382917770343 < 47.928085547952485. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 456.0514507899059 (T) = (0 -24.67150344864071) / Math.Sqrt((0.04341605986020369 / (146)) + (0.08719631651550766 / (30))) is greater than 1.9736914397558152 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (146) + (30) - 2, .975) and 0.5113145034883495 = (50.48544232384955 - 24.67150344864071) / 50.48544232384955 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
109.37 μs 83.38 μs 0.76 0.07 False
87.68 μs 77.04 μs 0.88 0.02 False
87.58 μs 76.63 μs 0.87 0.02 False
153.22 μs 116.84 μs 0.76 0.01 False
87.51 μs 76.73 μs 0.88 0.01 False
138.72 μs 117.44 μs 0.85 0.01 False

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### PerfLabTests.CastingPerf.ObjObjrefValueType #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 83.3818017631489 < 103.88934049133158. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 11.875007316114328 (T) = (0 -83659.194261858) / Math.Sqrt((160129113.78165013 / (144)) + (811571.4644059858 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.13156249530278488 = (96333.00474629569 - 83659.194261858) / 96333.00474629569 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.CastingPerf.ObjInt #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 77.03658904888215 < 83.15142552769188. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.14606469257253 (T) = (0 -76906.85446866398) / Math.Sqrt((38694172.96590851 / (143)) + (23398.310990764083 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.06430526520517765 = (82192.24882731438 - 76906.85446866398) / 82192.24882731438 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.CastingPerf.IFooFooIsIFoo #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 76.62987342728759 < 83.06617371487596. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 9.459922325800472 (T) = (0 -76657.58923584738) / Math.Sqrt((45906363.57395459 / (144)) + (11205.679938655836 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.0651737610008423 = (82001.96575346282 - 76657.58923584738) / 82001.96575346282 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.CastingPerf.CheckIsInstAnyIsInterfaceNo #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 116.84448940565031 < 145.62743599026393. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 11.041526043409252 (T) = (0 -116887.36317903012) / Math.Sqrt((328742971.08293754 / (143)) + (40990.75349439239 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.12531484927809633 = (133633.64301149905 - 116887.36317903012) / 133633.64301149905 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.CastingPerf.ObjFooIsObj2 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 76.7270181402439 < 83.14465046505055. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.852971440130412 (T) = (0 -76659.00701024734) / Math.Sqrt((29646777.432613544 / (142)) + (15155.830596042346 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.060827533027844424 = (81623.99314940747 - 76659.00701024734) / 81623.99314940747 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.CastingPerf.CheckObjIsInterfaceNo #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 117.43919831194196 < 131.7065762927232. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.205007202881225 (T) = (0 -117749.51367781585) / Math.Sqrt((113572317.42708474 / (140)) + (396845.6569104739 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.07295258410606913 = (127015.63227407592 - 117749.51367781585) / 127015.63227407592 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.StreamReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
122.08 μs 89.63 μs 0.73 0.02 True
114.07 μs 82.77 μs 0.73 0.03 True
160.23 μs 124.18 μs 0.77 0.01 True
669.05 μs 582.07 μs 0.87 0.03 False
332.07 μs 280.98 μs 0.85 0.02 False
1.07 ms 885.04 μs 0.83 0.02 False
116.20 μs 83.85 μs 0.72 0.03 True
112.07 μs 81.35 μs 0.73 0.01 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.StreamReaderReadLineTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.StreamReaderReadLineTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 33, 128]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 89.6263181122449 < 116.14496816029168. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 79.30129703039731 (T) = (0 -89535.49650947424) / Math.Sqrt((22283503.27099647 / (145)) + (937614.9216108071 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.27582607854733177 = (123638.1121400631 - 89535.49650947424) / 123638.1121400631 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 129, 1024]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 82.77181773560208 < 108.98796984407585. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 90.713976834174 (T) = (0 -82706.917958373) / Math.Sqrt((17838168.938611407 / (142)) + (184682.97604602805 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.28477172419414326 = (115637.092039162 - 82706.917958373) / 115637.092039162 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 9, 32]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 124.17577160714286 < 152.032613752613. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 69.39231917621527 (T) = (0 -123770.34545312959) / Math.Sqrt((42251715.95010577 / (143)) + (589995.4631418481 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2393890814609122 = (162724.91287773833 - 123770.34545312959) / 162724.91287773833 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 1]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 582.0704308641976 < 639.0701574766582. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.97978348317165 (T) = (0 -573387.1329027993) / Math.Sqrt((1335357844.0760708 / (143)) + (23531860.630166523 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.17782774500445456 = (697405.1111781994 - 573387.1329027993) / 697405.1111781994 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 8]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 280.9830933712121 < 315.5554593156345. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 47.01469449415308 (T) = (0 -281376.56047165085) / Math.Sqrt((215154055.7553909 / (144)) + (3323664.2148318943 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.174696863081216 = (340937.2240146234 - 281376.56047165085) / 340937.2240146234 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 885.0401236111111 < 1.016761548560554. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 43.864186767479836 (T) = (0 -889420.1300225348) / Math.Sqrt((3599717471.3886585 / (146)) + (59434369.44121779 / (30))) is greater than 1.9736914397558152 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (146) + (30) - 2, .975) and 0.20289049065259013 = (1115806.6985685558 - 889420.1300225348) / 1115806.6985685558 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 83.85400653966131 < 110.34228325925925. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 87.96147311949186 (T) = (0 -84113.2426286694) / Math.Sqrt((19545878.091861024 / (144)) + (251800.9208655488 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.2841863533206639 = (117507.17944379972 - 84113.2426286694) / 117507.17944379972 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 1024]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 81.34593033854168 < 106.61086041361554. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 92.10718283935256 (T) = (0 -80841.22248641806) / Math.Sqrt((16609161.760896232 / (145)) + (261764.05536343975 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.2857292388501525 = (113180.08083696192 - 80841.22248641806) / 113180.08083696192 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.MathBenchmarks.Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.33 μs 19.20 μs 0.53 0.02 True
59.60 μs 21.50 μs 0.36 0.02 True
305.21 μs 104.83 μs 0.34 0.02 True
116.84 μs 35.80 μs 0.31 0.02 True
117.68 μs 57.96 μs 0.49 0.01 True
50.20 μs 45.50 μs 0.91 0.08 False
438.04 μs 206.25 μs 0.47 0.01 True
75.00 μs 17.98 μs 0.24 0.01 True
235.97 μs 115.16 μs 0.49 0.02 True
220.67 μs 105.10 μs 0.48 0.01 True
228.03 μs 110.03 μs 0.48 0.01 True

graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.MathBenchmarks.Single*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.MathBenchmarks.Single* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.MathBenchmarks.Single*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.MathBenchmarks.Single* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.MathBenchmarks.Single.ILogB #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.20030228587688 < 34.413273018071216. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 201.2008161226888 (T) = (0 -19116.3183915893) / Math.Sqrt((7125.039016014956 / (145)) + (217057.2659028406 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.4732130993980258 = (36288.522682975876 - 19116.3183915893) / 36288.522682975876 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.ScaleB #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.50486240936704 < 56.965190249081644. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 829.4682325616075 (T) = (0 -21620.158777142235) / Math.Sqrt((18003.02382068265 / (140)) + (59966.74121650569 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.6389364789095806 = (59879.10025318786 - 21620.158777142235) / 59879.10025318786 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.Hypot #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 104.82789509722222 < 290.08337624069935. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 919.1866814272075 (T) = (0 -104864.57009703966) / Math.Sqrt((254486.1536036232 / (144)) + (1377812.5996592632 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.6568636058093241 = (305606.085138168 - 104864.57009703966) / 305606.085138168 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.Round #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 35.79512356321839 < 110.90574358579742. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 855.6763784110481 (T) = (0 -36100.29927371259) / Math.Sqrt((48710.89846355814 / (142)) + (255486.05817006936 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.6904969773435564 = (116639.56934528831 - 36100.29927371259) / 116639.56934528831 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.CopySign #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 57.96387084986774 < 111.78099977297822. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2098.775709316682 (T) = (0 -57979.86003821324) / Math.Sqrt((66747.05588400386 / (143)) + (10349.162029546997 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5077107008155102 = (117775.99093512852 - 57979.86003821324) / 117775.99093512852 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.ExpM1 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 45.5048047117248 < 48.30018069445782. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 13.923412821244941 (T) = (0 -46125.428705950704) / Math.Sqrt((4097054.0074297837 / (143)) + (638428.6838776522 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.06318952990270991 = (49236.67078695274 - 46125.428705950704) / 49236.67078695274 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.SinCosPi #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 206.25302232142855 < 416.25750288684117. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2669.222185448689 (T) = (0 -206058.30927348963) / Math.Sqrt((654134.2025236551 / (142)) + (88550.77124428033 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.5296717992194684 = (438115.9984273243 - 206058.30927348963) / 438115.9984273243 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.Abs #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.97553649515042 < 70.28072251711659. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1427.2415852203537 (T) = (0 -18015.930838788798) / Math.Sqrt((215389.3331043864 / (145)) + (1645.9123041408632 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.7566418608794201 = (74030.5251506801 - 18015.930838788798) / 74030.5251506801 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.TanPi #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 115.16065999212185 < 224.43762339829865. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 822.428193833976 (T) = (0 -115091.00283462033) / Math.Sqrt((1618890.245442441 / (143)) + (309801.33759377565 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5125244293031738 = (236095.93947467458 - 115091.00283462033) / 236095.93947467458 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.SinPi #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 105.1027146778475 < 209.8768721022516. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1543.9216179788752 (T) = (0 -105697.42627201648) / Math.Sqrt((155529.1178638178 / (140)) + (133205.70322745084 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.521142871995386 = (220728.52233077344 - 105697.42627201648) / 220728.52233077344 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.CosPi #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 110.03337009976525 < 216.62256477924097. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1387.6281051253436 (T) = (0 -110219.04260591132) / Math.Sqrt((191113.99376028072 / (143)) + (175446.25354278937 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5162389463637311 = (227837.7760620288 - 110219.04260591132) / 227837.7760620288 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Node.Tests.Perf_ParseThenWrite

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
168.60 μs 155.73 μs 0.92 0.01 True
124.44 μs 113.44 μs 0.91 0.01 True
112.19 μs 97.12 μs 0.87 0.01 True
159.12 μs 143.37 μs 0.90 0.02 True
35.48 μs 32.79 μs 0.92 0.01 True
13.56 ms 12.45 ms 0.92 0.01 True
24.41 μs 22.42 μs 0.92 0.03 True
16.10 μs 14.67 μs 0.91 0.01 True
1.86 μs 1.70 μs 0.91 0.09 False
17.05 μs 15.42 μs 0.90 0.01 True
23.14 μs 21.34 μs 0.92 0.01 True
122.35 μs 106.70 μs 0.87 0.01 True
34.02 μs 30.33 μs 0.89 0.01 True
12.75 ms 11.78 ms 0.92 0.01 True
129.47 μs 118.42 μs 0.91 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Node.Tests.Perf_ParseThenWrite*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Node.Tests.Perf_ParseThenWrite* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Node.Tests.Perf_ParseThenWrite*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Node.Tests.Perf_ParseThenWrite* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: BroadTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 155.7320936133382 < 160.41341225913536. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.974008187448675 (T) = (0 -152322.07650265534) / Math.Sqrt((1819036.5942460268 / (141)) + (1573885.5761272018 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.08867709897248541 = (167143.91389804045 - 152322.07650265534) / 167143.91389804045 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json4KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 113.44462649880094 < 117.58587466871846. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 48.705117659572956 (T) = (0 -113853.16635808338) / Math.Sqrt((1006780.5018841792 / (143)) + (779270.141848326 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.07212438059213262 = (122703.04766789741 - 113853.16635808338) / 122703.04766789741 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: DeepTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 97.12247887302914 < 106.86511163426951. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 78.97576156791843 (T) = (0 -96830.81475513379) / Math.Sqrt((327053.29315949377 / (144)) + (1109638.1455119536 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.13912101638715735 = (112479.00877863789 - 96830.81475513379) / 112479.00877863789 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: BroadTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 143.36560932721713 < 149.63855027262363. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 50.667271996828674 (T) = (0 -142164.53579744246) / Math.Sqrt((2187095.237907284 / (143)) + (2128875.969044779 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.0947544437892835 = (157045.27332067944 - 142164.53579744246) / 157045.27332067944 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 32.79091108211553 < 33.91127515804365. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.7067908900205 (T) = (0 -32472.268845964) / Math.Sqrt((180383.34752976178 / (144)) + (135850.14790326107 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.08309947031993659 = (35415.25802945565 - 32472.268845964) / 35415.25802945565 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 12.453711238095241 < 12.790636966071427. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 44.87451762482321 (T) = (0 -12522372.615247328) / Math.Sqrt((10863837498.100384 / (144)) + (7388249053.374787 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.060394395845947434 = (13327264.71605232 - 12522372.615247328) / 13327264.71605232 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 22.419039649375605 < 23.24886430156292. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 55.37441061618474 (T) = (0 -22377.819474237225) / Math.Sqrt((83877.76916274306 / (141)) + (12078.58261174388 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.07248809546442174 = (24126.71941439091 - 22377.819474237225) / 24126.71941439091 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.673212344382195 < 15.249666638950277. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 27.079104740282308 (T) = (0 -14773.729853985573) / Math.Sqrt((47773.64270342424 / (145)) + (37236.210254957456 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.06772254110345428 = (15846.923802568313 - 14773.729853985573) / 15846.923802568313 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.6950095720352827 < 1.7685858355641122. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 17.509796034572616 (T) = (0 -1708.5642653610203) / Math.Sqrt((2294.484447449672 / (142)) + (1265.6706055265822 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.072598581409243 = (1842.313620737488 - 1708.5642653610203) / 1842.313620737488 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.41872251806413 < 16.026993978113822. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 26.680618205515543 (T) = (0 -15542.402942886602) / Math.Sqrt((49323.48832068076 / (143)) + (37036.59271104228 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.06386628540017113 = (16602.759520877364 - 15542.402942886602) / 16602.759520877364 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.339686416485 < 21.959465104522998. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 67.78913385497735 (T) = (0 -21088.83071924661) / Math.Sqrt((27392.90684073919 / (143)) + (15184.732902212101 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.07826260584597032 = (22879.434916060807 - 21088.83071924661) / 22879.434916060807 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: DeepTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 106.70237636388434 < 116.01326146711078. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 120.24979658282092 (T) = (0 -106286.95082071605) / Math.Sqrt((587655.1833775234 / (144)) + (373214.72845383093 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.12695883731562702 = (121743.34425871915 - 106286.95082071605) / 121743.34425871915 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 30.330636040921288 < 32.47840834564448. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 45.22971004533506 (T) = (0 -30675.966151566085) / Math.Sqrt((122956.07155780253 / (144)) + (115517.71523468521 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.0918420308006139 = (33778.22712783042 - 30675.966151566085) / 33778.22712783042 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 11.784132308333332 < 12.182479273440478. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 46.201968704481764 (T) = (0 -11900838.44150697) / Math.Sqrt((11728280981.290672 / (144)) + (6764637227.795147 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.06368369071465875 = (12710275.708633635 - 11900838.44150697) / 12710275.708633635 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json4KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 118.42121863162879 < 122.7242544039709. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 42.735984284177114 (T) = (0 -119678.1803626486) / Math.Sqrt((1287512.2460440705 / (143)) + (923475.530557092 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.06649100890467974 = (128202.49349952786 - 119678.1803626486) / 128202.49349952786 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Vector3

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
51.33 ns 20.01 ns 0.39 0.01 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector3* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector3* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_Vector3.AbsBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.014678919840296 < 48.696049704089724. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1589.1673114685595 (T) = (0 -19.80123627471445) / Math.Sqrt((0.033505353427855954 / (143)) + (0.0047628147533661435 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6140696858636201 = (51.30780233997658 - 19.80123627471445) / 51.30780233997658 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Tests.Perf_Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
112.91 ns 85.53 ns 0.76 0.05 True
2.29 μs 1.76 μs 0.77 0.01 False
60.98 ns 46.82 ns 0.77 0.02 True
2.52 μs 2.02 μs 0.80 0.03 False
1.77 μs 1.12 μs 0.63 0.03 True
127.83 ns 105.95 ns 0.83 0.07 True
2.53 μs 2.00 μs 0.79 0.02 False
2.32 μs 1.67 μs 0.72 0.07 True
2.32 μs 1.73 μs 0.75 0.03 False
113.74 ns 85.86 ns 0.75 0.02 True
77.34 ns 58.82 ns 0.76 0.03 True
150.85 ns 120.37 ns 0.80 0.06 True

graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Tests.Perf_Encoding* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Tests.Perf_Encoding* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Tests.Perf_Encoding.GetChars(size: 16, encName: "ascii") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 85.5292045651577 < 105.81156204435489. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 50.839415233045415 (T) = (0 -86.96456015052611) / Math.Sqrt((16.55031119008459 / (143)) + (4.387538111295589 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.23031071508236972 = (112.98658024040543 - 86.96456015052611) / 112.98658024040543 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetChars(size: 512, encName: "ascii") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.7583051528306892 < 2.187171039357965. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 52.09077720392375 (T) = (0 -1772.4690742448513) / Math.Sqrt((15744.736058637885 / (145)) + (135.20365995521905 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.23811466508035461 = (2326.424978939633 - 1772.4690742448513) / 2326.424978939633 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetBytes(size: 16, encName: "ascii") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 46.82198102634845 < 57.72801112640601. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.99326720177985 (T) = (0 -47.94822534756381) / Math.Sqrt((0.1681895341920511 / (142)) + (1.346831788656246 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.21171973216429935 = (60.82636760553487 - 47.94822534756381) / 60.82636760553487 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetString(size: 512, encName: "utf-8") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.0159839429998363 < 2.3999585989232566. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 27.513330823344706 (T) = (0 -2012.8958855299888) / Math.Sqrt((46749.732107084885 / (144)) + (2450.4271966145843 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.21600958789315036 = (2567.5006408823942 - 2012.8958855299888) / 2567.5006408823942 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetBytes(size: 512, encName: "ascii") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.1169043671607168 < 1.6900458274943602. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 274.3376166673512 (T) = (0 -1119.1513629973524) / Math.Sqrt((595.138994313171 / (140)) + (50.67888714221836 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.37400164612802006 = (1787.7864311864068 - 1119.1513629973524) / 1787.7864311864068 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "utf-8") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 105.94706095336039 < 122.19401207311708. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.95127274206566 (T) = (0 -107.22258707572723) / Math.Sqrt((16.723104821969862 / (144)) + (1.0561607954002215 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.1737293069751838 = (129.76690082423983 - 107.22258707572723) / 129.76690082423983 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetChars(size: 512, encName: "utf-8") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.000063421500237 < 2.402829453651885. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 28.741155367203195 (T) = (0 -2011.048018097852) / Math.Sqrt((44697.25148889986 / (145)) + (1882.957605267161 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.21586122477069364 = (2564.6582998140343 - 2011.048018097852) / 2564.6582998140343 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetBytes(size: 512, encName: "utf-8") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.6684947605419616 < 2.2114658250381827. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 72.56741473974164 (T) = (0 -1679.6327873027103) / Math.Sqrt((3855.39549225401 / (142)) + (1654.862680342747 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.28159746955150866 = (2338.010678016589 - 1679.6327873027103) / 2338.010678016589 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetString(size: 512, encName: "ascii") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.7289137926026337 < 2.194335215743578. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 48.69368380584175 (T) = (0 -1769.7064389768916) / Math.Sqrt((17339.51556296957 / (142)) + (240.79405435819643 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.2388980128969331 = (2325.1896184278935 - 1769.7064389768916) / 2325.1896184278935 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "ascii") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 85.86387608514089 < 108.15662162315107. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 78.17217546897147 (T) = (0 -86.4792846942186) / Math.Sqrt((15.16261653971429 / (144)) + (0.599314310769834 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.24239000315407624 = (114.14749680475246 - 86.4792846942186) / 114.14749680475246 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetBytes(size: 16, encName: "utf-8") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 58.8231210159907 < 72.80090141440041. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 100.72906534590973 (T) = (0 -58.88858627912683) / Math.Sqrt((1.0259376395426505 / (141)) + (0.6042072638701645 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.22071235537455117 = (75.56720125779822 - 58.88858627912683) / 75.56720125779822 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Tests.Perf_Encoding.GetChars(size: 16, encName: "utf-8") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 120.37217425069484 < 144.65830491077602. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 50.73586694266154 (T) = (0 -120.72995647374088) / Math.Sqrt((19.85096232494209 / (143)) + (8.963551939991621 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.21752557691025723 = (154.2925275397714 - 120.72995647374088) / 154.2925275397714 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.ContainsTrueComparer<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
54.16 μs 33.32 μs 0.62 0.03 True
78.60 μs 58.37 μs 0.74 0.01 True
54.75 μs 34.82 μs 0.64 0.03 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsTrueComparer<String>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsTrueComparer<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsTrueComparer<String>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsTrueComparer<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsTrueComparer<String>.FrozenSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 33.32444827374118 < 51.806047179131355. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 216.26935538346987 (T) = (0 -33997.519051336574) / Math.Sqrt((372625.0114887275 / (88)) + (147355.4093736439 / (30))) is greater than 1.9806260024587088 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (88) + (30) - 2, .975) and 0.37825297241483485 = (54680.62981077893 - 33997.519051336574) / 54680.62981077893 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrueComparer<String>.ImmutableHashSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 58.373688299573566 < 75.7853275521636. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 154.76695014368883 (T) = (0 -59444.56529726512) / Math.Sqrt((436794.59407270356 / (141)) + (427834.81957688514 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.2554133524441945 = (79835.65847225302 - 59444.56529726512) / 79835.65847225302 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrueComparer<String>.HashSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 34.82354875739098 < 52.35920640116888. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 58.40802079195811 (T) = (0 -35717.87018442552) / Math.Sqrt((739858.561529407 / (144)) + (3287521.8932759883 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.356446782622271 = (55501.03584279211 - 35717.87018442552) / 55501.03584279211 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
  • [IndexOfString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 22.04_LLVM=false_MonoAOT=true_MonoInterpreter=false_RunKind=micro_mono/System.Memory.ReadOnlySpan.IndexOfString(input%3a%20%22%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%99%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%22%2c%20value%3a%20%22%e3%81%99%22%2c%20comparisonType%3a%20Ordinal).html>)
  • 📝 - Benchmark Source
  • 📈 - ADX Test Multi Config Graph
95.10 ns 44.15 ns 0.46 0.02 True
239.71 ns 71.35 ns 0.30 0.01 True
761.30 ns 101.75 ns 0.13 0.02 False
118.13 ns 58.45 ns 0.49 0.05 True
45.23 ns 38.97 ns 0.86 0.10 False
81.95 ns 36.19 ns 0.44 0.01 True
785.92 ns 684.67 ns 0.87 0.27 False
152.33 ns 64.29 ns 0.42 0.02 False
67.77 ns 57.22 ns 0.84 0.04 True
  • [IndexOfString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 22.04_LLVM=false_MonoAOT=true_MonoInterpreter=false_RunKind=micro_mono/System.Memory.ReadOnlySpan.IndexOfString(input%3a%20%22%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0%e3%81%a0.html>)
  • 📝 - Benchmark Source
  • 📈 - ADX Test Multi Config Graph
594.71 ns 159.75 ns 0.27 0.01 True
131.75 ns 87.75 ns 0.67 0.02 False
635.53 ns 131.16 ns 0.21 0.03 False
592.35 ns 160.17 ns 0.27 0.01 True
46.32 ns 38.01 ns 0.82 0.10 True

graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.ReadOnlySpan*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.ReadOnlySpan* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.ReadOnlySpan*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.ReadOnlySpan* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだすだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "す", comparisonType: Ordinal) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 44.15260463304578 < 90.2806643827229. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 170.39779619156548 (T) = (0 -43.30917965536626) / Math.Sqrt((12.977035000220623 / (144)) + (0.3815722966589543 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5578566446209196 = (97.95279998776476 - 43.30917965536626) / 97.95279998776476 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "~", comparisonType: Ordinal) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 71.35145916675285 < 227.6940497211778. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 241.36172044542826 (T) = (0 -68.0986219481095) / Math.Sqrt((1.0185365076116886 / (143)) + (15.008346924614226 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7162859879904532 = (240.02558585586533 - 68.0986219481095) / 240.02558585586533 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 101.74874678049594 < 722.3964972863562. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.211143819787026 (T) = (0 -131.84688110263878) / Math.Sqrt((106192.51195968494 / (143)) + (3655.7143746315764 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6467776261255589 = (373.26877019830573 - 131.84688110263878) / 373.26877019830573 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: OrdinalIgnoreCase) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 58.44703582324133 < 112.59365766104223. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 148.75282917418374 (T) = (0 -57.98521014836606) / Math.Sqrt((14.98366846762061 / (143)) + (2.375607709409561 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5238820322692793 = (121.78748561986829 - 57.98521014836606) / 121.78748561986829 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "w", comparisonType: OrdinalIgnoreCase) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 38.96668153117342 < 42.96963324222473. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 25.61406386362459 (T) = (0 -38.451084947394676) / Math.Sqrt((14.682936409963908 / (142)) + (1.458285950389845 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.20617328196159285 = (48.43763011959283 - 38.451084947394676) / 48.43763011959283 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello Worldbbbbbbbbbbbbbbcbbbbbbbbbbbbbbbbbbba!", value: "y", comparisonType: Ordinal) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 36.187584352585695 < 77.77853018043963. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 734.6768602423952 (T) = (0 -36.251327740253274) / Math.Sqrt((0.08389798699420815 / (143)) + (0.09797917062760166 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.557114748380904 = (81.85264153124538 - 36.251327740253274) / 81.85264153124538 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "だ", value: "た", comparisonType: InvariantCulture) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 684.6734700750883 < 747.1060731948077. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.088675722884049 (T) = (0 -674.8198371752175) / Math.Sqrt((5254.052491603174 / (141)) + (868.6121330737575 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.057814469728268696 = (716.2281901958268 - 674.8198371752175) / 716.2281901958268 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "ABCDE", value: "c", comparisonType: InvariantCultureIgnoreCase) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 64.29095504911147 < 145.04179671526344. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.311491781024172 (T) = (0 -66.21313203025848) / Math.Sqrt((2043.972149609227 / (143)) + (73.29766994625157 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.33929539727659147 = (100.21593879826104 - 66.21313203025848) / 100.21593879826104 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "More Test's", value: "Tests", comparisonType: OrdinalIgnoreCase) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 57.21915954573484 < 65.0513569456773. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.389430549135 (T) = (0 -57.82408584953521) / Math.Sqrt((2.9091763014149774 / (143)) + (1.6380024404363882 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.15379638361916192 = (68.3335366691593 - 57.82408584953521) / 68.3335366691593 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだxだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "x", comparisonType: Ordinal) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 159.7549209315845 < 566.1855287901726. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 180.87052644163705 (T) = (0 -153.62969257545782) / Math.Sqrt((18.50025253602784 / (144)) + (178.4121404762521 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7437158431492998 = (599.4506038270471 - 153.62969257545782) / 599.4506038270471 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 87.75148231163324 < 125.1274861733532. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 7.129358376030474 (T) = (0 -87.77474472904133) / Math.Sqrt((571.8101650980971 / (144)) + (29.2619313830732 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.15300392368641344 = (103.63063913007332 - 87.77474472904133) / 103.63063913007332 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 131.15512544567954 < 604.0015828948764. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 8.206824758822426 (T) = (0 -151.48797735224582) / Math.Sqrt((62779.376749054696 / (140)) + (1999.2668734554852 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.5514689667182531 = (337.74246620988635 - 151.48797735224582) / 337.74246620988635 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "X", comparisonType: Ordinal) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 160.1660638852315 < 565.8204559017819. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 181.49770985995622 (T) = (0 -153.83616062990555) / Math.Sqrt((20.076685289982315 / (143)) + (176.7034511005941 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7434104909555583 = (599.541895546715 - 153.83616062990555) / 599.541895546715 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "StrIng", value: "string", comparisonType: OrdinalIgnoreCase) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 38.0097168701641 < 43.82032047857828. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 39.754031006982004 (T) = (0 -37.79201479539617) / Math.Sqrt((2.179503437826916 / (143)) + (0.9752557649148411 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.18690056055151102 = (46.47895812230522 - 37.79201479539617) / 46.47895812230522 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
63.32 ns 56.49 ns 0.89 0.12 True
66.34 ns 56.91 ns 0.86 0.16 False
69.92 ns 55.03 ns 0.79 0.17 True
187.83 ns 171.54 ns 0.91 0.04 True
63.09 ns 55.23 ns 0.88 0.11 True

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.TryGetValueMiss #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 56.49088030578943 < 61.55326057211566. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.66644304062437 (T) = (0 -54.49698270650309) / Math.Sqrt((3.3919391785010986 / (142)) + (1.4881101824995067 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.1434479478472195 = (63.623667201001155 - 54.49698270650309) / 63.623667201001155 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.GetHit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 56.908985580830745 < 62.93053892789206. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 25.65915949691618 (T) = (0 -56.427384494225606) / Math.Sqrt((5.393494158366783 / (141)) + (2.470994140915774 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.1363883906003891 = (65.33884431388589 - 56.427384494225606) / 65.33884431388589 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.GetMiss #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 55.02517497819912 < 61.24925401426898. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.92232986068518 (T) = (0 -56.52011848462941) / Math.Sqrt((3.2017698036420734 / (142)) + (1.4840834628394928 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.13518502475083827 = (65.35515700146775 - 56.52011848462941) / 65.35515700146775 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.SetOverride #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 171.54090041860803 < 180.8628818127419. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 23.73708485496401 (T) = (0 -173.57000621707323) / Math.Sqrt((12.04074570103227 / (142)) + (12.338164376254591 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.08785867014542088 = (190.28850084530777 - 173.57000621707323) / 190.28850084530777 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.TryGetValueHit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 55.230307685386414 < 60.89835259817433. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 42.92264997586893 (T) = (0 -54.125007936801346) / Math.Sqrt((3.6476707457167588 / (140)) + (0.6198040377867496 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.14632224315747835 = (63.40215321644583 - 54.125007936801346) / 63.40215321644583 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Environment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
208.71 ns 122.54 ns 0.59 0.01 True
375.70 ns 246.35 ns 0.66 0.01 True
119.40 ns 87.25 ns 0.73 0.03 True
9.08 μs 6.83 μs 0.75 0.07 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Environment*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Environment* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Environment*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Environment* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Environment.SetEnvironmentVariable #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 122.54080773286657 < 197.42740333189514. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 213.36460888133595 (T) = (0 -120.57302316008213) / Math.Sqrt((3.1402535726204275 / (142)) + (4.3720804821472585 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.42028734068126533 = (207.98756284152367 - 120.57302316008213) / 207.98756284152367 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Environment.ExpandEnvironmentVariables #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 246.3504227433977 < 356.8707435357862. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 179.78406253122037 (T) = (0 -246.08076191453665) / Math.Sqrt((27.947401045129737 / (141)) + (10.147421453166299 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.3485795621951797 = (377.7602722195643 - 246.08076191453665) / 377.7602722195643 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Environment.GetEnvironmentVariable #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 87.24637332298556 < 113.53252789882177. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 113.12661114792265 (T) = (0 -86.62059254930766) / Math.Sqrt((3.0152898373306964 / (142)) + (1.9612498816244526 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.27763901858986156 = (119.91316637868991 - 86.62059254930766) / 119.91316637868991 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Environment.GetEnvironmentVariables #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.826952183805961 < 8.596310060110945. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 55.63770467885304 (T) = (0 -7062.3739874437215) / Math.Sqrt((33661.793674665234 / (142)) + (32924.593435808725 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.22347939696957142 = (9094.895821028173 - 7062.3739874437215) / 9094.895821028173 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.64 ms 1.45 ms 0.89 0.01 False
197.23 μs 175.06 μs 0.89 0.01 False
197.04 μs 153.08 μs 0.78 0.02 False
197.09 μs 174.85 μs 0.89 0.01 False
3.02 ms 2.49 ms 0.83 0.03 False
218.41 μs 198.42 μs 0.91 0.35 False

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.LowLevelPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.LowLevelPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.LowLevelPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.LowLevelPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### PerfLabTests.LowLevelPerf.EmptyInstanceFunction #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.4511233645833335 < 1.5537619918973213. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.240246751748579 (T) = (0 -1443289.1774878597) / Math.Sqrt((9472883372.4806 / (140)) + (64372393.140370175 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.05596317308303118 = (1528848.3842323683 - 1443289.1774878597) / 1528848.3842323683 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.LowLevelPerf.SealedClassInterfaceMethod #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 175.06377157738098 < 187.28217560571912. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.968507183259778 (T) = (0 -175069.67417206676) / Math.Sqrt((119596664.26670721 / (143)) + (340981.9910911546 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.054538498209512766 = (185168.48527467798 - 175069.67417206676) / 185168.48527467798 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.LowLevelPerf.ClassVirtualMethod #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 153.07773026960783 < 186.89597769962896. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 9.97572124200903 (T) = (0 -153850.13417423077) / Math.Sqrt((477946462.4285717 / (144)) + (16105865.575040754 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.11294380207035679 = (173438.99353086238 - 153850.13417423077) / 173438.99353086238 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.LowLevelPerf.InstanceDelegate #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 174.85306289682538 < 187.10003486486235. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 11.306245000738956 (T) = (0 -175088.3350832776) / Math.Sqrt((118259982.07937036 / (145)) + (130726.61773824069 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.055242524407728054 = (185326.22350885774 - 175088.3350832776) / 185326.22350885774 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.LowLevelPerf.InterfaceInterfaceMethod #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.494901355867347 < 2.8701320008680558. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 13.499000090411023 (T) = (0 -2512415.3447533646) / Math.Sqrt((50825716691.022995 / (142)) + (919639543.9775996 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.09577021087864172 = (2778514.2393889534 - 2512415.3447533646) / 2778514.2393889534 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 198.4181909612342 < 207.46639633535878. IsChangePoint: Marked as a change because one of 9/1/2023 9:23:50 AM, 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 4.789824346568531 (T) = (0 -187264.22178402086) / Math.Sqrt((163291848.73159054 / (144)) + (151244298.5189597 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.059763511261264064 = (199167.14999566038 - 187264.22178402086) / 199167.14999566038 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Globalization.Tests.Perf_NumberCultureInfo

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
332.02 ns 304.97 ns 0.92 0.01 True
329.25 ns 305.12 ns 0.93 0.01 True
332.02 ns 304.34 ns 0.92 0.02 True
332.59 ns 306.83 ns 0.92 0.01 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.Perf_NumberCultureInfo*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.Perf_NumberCultureInfo* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.Perf_NumberCultureInfo*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.Perf_NumberCultureInfo* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: ja) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 304.96653671636847 < 311.52691140397235. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 53.71530578728271 (T) = (0 -305.8672826260702) / Math.Sqrt((13.555400317832941 / (142)) + (3.5969223103031003 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.07535635721603227 = (330.794771599952 - 305.8672826260702) / 330.794771599952 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: da) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 305.1233608973488 < 311.22555362571325. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 41.30182927102496 (T) = (0 -305.9418604154779) / Math.Sqrt((15.715151466020552 / (141)) + (7.15345175155735 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.07395011071879755 = (330.37297877433923 - 305.9418604154779) / 330.37297877433923 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: ) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 304.34228873234485 < 313.23825103532005. IsChangePoint: Marked as a change because one of 9/26/2023 6:21:18 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 52.36228846156848 (T) = (0 -306.1539510557378) / Math.Sqrt((19.06207151091171 / (144)) + (2.3568983668043257 / (29))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (29) - 2, .975) and 0.0732629721808757 = (330.35687780405743 - 306.1539510557378) / 330.35687780405743 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: fr) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 306.8284865463441 < 311.9998010221784. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 46.72813675732625 (T) = (0 -306.1251261274969) / Math.Sqrt((16.438237163240814 / (141)) + (4.618341317887587 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.07355390199878865 = (330.42950559990015 - 306.1251261274969) / 330.42950559990015 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.TryGetValueTrue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
77.70 μs 56.14 μs 0.72 0.08 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.TryGetValueTrue<String, String>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.TryGetValueTrue<String, String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.TryGetValueTrue<String, String>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.TryGetValueTrue<String, String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.TryGetValueTrue<String, String>.ImmutableDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 56.13602987882653 < 73.90604618563701. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 60.4208728431447 (T) = (0 -57418.02950302053) / Math.Sqrt((2330894.5779264728 / (141)) + (2604108.1059642583 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.252765965043009 = (76840.75780397955 - 57418.02950302053) / 76840.75780397955 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.Perf_Frozen<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
72.53 μs 68.76 μs 0.95 0.03 False
5.74 μs 5.31 μs 0.93 0.01 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Perf_Frozen<Int16>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Perf_Frozen<Int16>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Perf_Frozen<Int16>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Perf_Frozen<Int16>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.Perf_Frozen<Int16>.ToFrozenDictionary(Count: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 68.76478048300818 < 68.90287538495531. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 23.59667476244409 (T) = (0 -68483.22757048844) / Math.Sqrt((2407245.85901074 / (88)) + (220379.74230617716 / (30))) is greater than 1.9806260024587088 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (88) + (30) - 2, .975) and 0.06031433530421798 = (72878.86805494633 - 68483.22757048844) / 72878.86805494633 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Perf_Frozen<Int16>.ToFrozenDictionary(Count: 64) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.31352514228527 < 5.457901478531108. IsChangePoint: Marked as a change because one of 9/9/2023 10:04:41 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 19.419808286494725 (T) = (0 -5314.649158923218) / Math.Sqrt((32492.204150214824 / (89)) + (2721.5984083344765 / (30))) is greater than 1.9804475986831467 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (30) - 2, .975) and 0.07236600213980973 = (5729.252238687594 - 5314.649158923218) / 5729.252238687594 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
28.47 ns 17.32 ns 0.61 0.02 True
1.02 μs 102.08 ns 0.10 0.02 True
605.02 ns 138.40 ns 0.23 0.09 True
2.87 μs 1.56 μs 0.54 0.01 True
63.66 ns 27.47 ns 0.43 0.02 True
618.26 ns 133.70 ns 0.22 0.11 True
28.65 ns 15.24 ns 0.53 0.07 True
2.00 μs 171.55 ns 0.09 0.02 True
39.95 ns 21.20 ns 0.53 0.03 True
40.51 ns 21.88 ns 0.54 0.03 True
81.31 ns 24.30 ns 0.30 0.01 True
23.43 ns 13.15 ns 0.56 0.01 True
586.15 ns 117.55 ns 0.20 0.10 True
151.89 ns 26.02 ns 0.17 0.01 True
190.97 ns 106.94 ns 0.56 0.01 True
15.93 ns 9.58 ns 0.60 0.09 True
1.00 μs 97.69 ns 0.10 0.02 True
96.24 ns 26.57 ns 0.28 0.03 True
58.30 ns 18.01 ns 0.31 0.05 True
45.78 ns 13.97 ns 0.31 0.10 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Int32>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Span<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Int32>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Span<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Memory.Span<Int32>.IndexOfValue(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.32432897700006 < 27.11328724847053. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 56.80210045525074 (T) = (0 -17.342704500865295) / Math.Sqrt((8.661095775154081 / (143)) + (0.028338289381214226 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.44822068651513247 = (31.430508678069383 - 17.342704500865295) / 31.430508678069383 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.EndsWith(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 102.08349391808561 < 0.9526755358085326. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1019.1202084005108 (T) = (0 -102.8438682605728) / Math.Sqrt((111.56926734011545 / (143)) + (0.3072313692241845 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8980647028336868 = (1008.9132137690947 - 102.8438682605728) / 1008.9132137690947 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.IndexOfValue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 138.40060578812847 < 572.2326416015513. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 234.9808082806908 (T) = (0 -125.49506031166673) / Math.Sqrt((10.655290780890033 / (143)) + (122.9045601317021 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7927126912905849 = (605.4160338759161 - 125.49506031166673) / 605.4160338759161 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.Reverse(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.5572791797673464 < 2.719175040891728. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1276.3090887674578 (T) = (0 -1557.7416509769262) / Math.Sqrt((96.44950430174279 / (145)) + (11.902557377668531 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.4577944219201194 = (2872.972381607316 - 1557.7416509769262) / 2872.972381607316 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.IndexOfValue(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 27.46782864464178 < 60.56647438499595. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 150.1150550279675 (T) = (0 -27.567958247562377) / Math.Sqrt((8.641440137357556 / (142)) + (0.19043664183443562 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.5853390175064414 = (66.48312576163497 - 27.567958247562377) / 66.48312576163497 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.Fill(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 133.70445629671977 < 588.4915323546784. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 255.25277970118321 (T) = (0 -127.63322693968081) / Math.Sqrt((9.915703245663764 / (144)) + (109.72524257483303 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.794262105292723 = (620.3681004963953 - 127.63322693968081) / 620.3681004963953 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.SequenceEqual(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.240952169996762 < 27.828659578012918. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 248.54720551419754 (T) = (0 -15.20743634262144) / Math.Sqrt((0.173156025955096 / (143)) + (0.056528413360088806 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.4762405802835022 = (29.035155779829164 - 15.20743634262144) / 29.035155779829164 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.SequenceEqual(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 171.54553402332462 < 1.8801603617161735. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1053.748902534465 (T) = (0 -170.41784946192328) / Math.Sqrt((425.94452436060624 / (145)) + (1.3622048661641428 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.9143789059004575 = (1990.372247098319 - 170.41784946192328) / 1990.372247098319 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.StartsWith(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.202591529041563 < 38.01042317256565. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 280.93315971597855 (T) = (0 -21.36499320681445) / Math.Sqrt((0.021631534320876274 / (143)) + (0.127207733401637 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.46563524263357164 = (39.98204019312579 - 21.36499320681445) / 39.98204019312579 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.EndsWith(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.88310344857861 < 38.47022223584226. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 364.7786366267278 (T) = (0 -21.923099242837964) / Math.Sqrt((0.1316103815853102 / (141)) + (0.04986866652087321 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.45879355404622185 = (40.5078309889722 - 21.923099242837964) / 40.5078309889722 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.StartsWith(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.30344659678741 < 76.64684812079075. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1343.3629524166365 (T) = (0 -24.270106666937863) / Math.Sqrt((0.15093973679337555 / (140)) + (0.02092261973026565 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.6999094598872995 = (80.87594716522253 - 24.270106666937863) / 80.87594716522253 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.LastIndexOfValue(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.151963043104633 < 22.27176037200125. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 102.38313232948029 (T) = (0 -13.156510065809504) / Math.Sqrt((1.4796934817685223 / (143)) + (0.0052964187306970174 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.44392671333158823 = (23.65966929401299 - 13.156510065809504) / 23.65966929401299 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.LastIndexOfValue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 117.54839319177245 < 557.170679375096. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 191.71079604109624 (T) = (0 -133.08937627669857) / Math.Sqrt((4.482998880206946 / (145)) + (166.98729850842756 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.7731346877739793 = (586.6448906217302 - 133.08937627669857) / 586.6448906217302 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.SequenceEqual(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 26.022820769956397 < 145.83883070433382. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1103.5204879557066 (T) = (0 -25.51252273234105) / Math.Sqrt((0.7665674814151323 / (143)) + (0.23646073451818644 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.8327069277060181 = (152.50196784902263 - 25.51252273234105) / 152.50196784902263 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.Reverse(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 106.93638120869923 < 181.27545105742257. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 592.1229365424747 (T) = (0 -106.91278371288841) / Math.Sqrt((0.8730539710529731 / (143)) + (0.42831472382961755 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.44155830807277535 = (191.4484273978977 - 106.91278371288841) / 191.4484273978977 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.Fill(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.57665723206312 < 15.14206016461188. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 179.58025945163226 (T) = (0 -9.599091174662338) / Math.Sqrt((0.15626692568262762 / (143)) + (0.005628108206977212 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.40098907768600034 = (16.024901745665538 - 9.599091174662338) / 16.024901745665538 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.StartsWith(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 97.69409641577587 < 0.957140578295752. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1030.442884449585 (T) = (0 -98.18724506152489) / Math.Sqrt((105.50089472408702 / (143)) + (1.2764452091501757 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.9026339862752123 = (1008.4344763159188 - 98.18724506152489) / 1008.4344763159188 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.EndsWith(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 26.566400432791482 < 92.04200628139667. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 604.3257758694397 (T) = (0 -26.617714457317707) / Math.Sqrt((0.7187821118052924 / (143)) + (0.25542848718186845 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7254203628903001 = (96.93987047802605 - 26.617714457317707) / 96.93987047802605 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.LastIndexOfValue(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 18.00942023344097 < 55.63393426653458. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 215.828564317253 (T) = (0 -18.56667195282372) / Math.Sqrt((1.6146970799436375 / (143)) + (0.700420751090154 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6838945674380303 = (58.73569398141835 - 18.56667195282372) / 58.73569398141835 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.Fill(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.971172611668743 < 42.96772201382592. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 284.6115283728567 (T) = (0 -13.849117071653602) / Math.Sqrt((1.7571828186917098 / (144)) + (0.008198363200093496 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.696550893978725 = (45.63901094730078 - 13.849117071653602) / 45.63901094730078 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Document.Tests.Perf_EnumerateObject

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.62 μs 8.13 μs 0.77 0.01 True
58.02 μs 52.97 μs 0.91 0.04 False
10.75 μs 8.28 μs 0.77 0.01 True
23.20 μs 15.98 μs 0.69 0.01 True
2.74 ms 2.34 ms 0.86 0.01 True
23.47 μs 16.16 μs 0.69 0.01 True
10.62 μs 8.12 μs 0.76 0.01 True
23.21 μs 16.06 μs 0.69 0.01 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Document.Tests.Perf_EnumerateObject* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Document.Tests.Perf_EnumerateObject* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: NumericProperties) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.127353237189038 < 10.07867355961925. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 510.27407071866537 (T) = (0 -8108.512353547837) / Math.Sqrt((2168.8586660527276 / (143)) + (258.6778487786693 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.23485431730049663 = (10597.344449412914 - 8108.512353547837) / 10597.344449412914 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateObject.Parse(TestCase: NumericProperties) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 52.96838315560321 < 55.501257348706154. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 26.689962013968707 (T) = (0 -53188.7210360431) / Math.Sqrt((2071287.990957104 / (144)) + (119354.43621538205 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.0636682870361758 = (56805.42514968526 - 53188.7210360431) / 56805.42514968526 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: ObjectProperties) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.284067375641024 < 10.244400273168102. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 586.0824073348613 (T) = (0 -8267.631485138148) / Math.Sqrt((1127.435439696564 / (143)) + (310.5058034522523 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2323678178025611 = (10770.30337820265 - 8267.631485138148) / 10770.30337820265 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateObject.PropertyIndexer(TestCase: StringProperties) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.983607966400285 < 22.017484909599318. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 915.4400307081627 (T) = (0 -15973.536447203958) / Math.Sqrt((2159.0246511144505 / (143)) + (1398.8237881276007 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.31046742450946835 = (23165.745919749224 - 15973.536447203958) / 23165.745919749224 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateObject.Parse(TestCase: ObjectProperties) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.3425388399234697 < 2.6003815060339397. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 230.35840824962457 (T) = (0 -2345561.1406293605) / Math.Sqrt((213046687.04733366 / (143)) + (32227157.968862206 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.13589115017529013 = (2714427.8653148534 - 2345561.1406293605) / 2714427.8653148534 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateObject.PropertyIndexer(TestCase: ObjectProperties) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.156213089378237 < 22.301331362571133. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 708.9559161434706 (T) = (0 -16169.72183044141) / Math.Sqrt((9198.68542704506 / (144)) + (1164.5101947963506 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.30763226741881416 = (23354.23947352338 - 16169.72183044141) / 23354.23947352338 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: StringProperties) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.117596443591745 < 10.093527148186682. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 430.7106489286013 (T) = (0 -8116.48951077346) / Math.Sqrt((3442.7746804159074 / (144)) + (288.1829277267466 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.23501065353947184 = (10609.937966230558 - 8116.48951077346) / 10609.937966230558 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateObject.PropertyIndexer(TestCase: NumericProperties) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.058378914835167 < 22.046415865057686. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 938.6761770259296 (T) = (0 -15976.3552645876) / Math.Sqrt((2325.580229504868 / (145)) + (1283.1536226526605 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.310617014026591 = (23174.86156411154 - 15976.3552645876) / 23174.86156411154 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Net.Tests.Perf_WebUtility

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
393.00 ns 354.26 ns 0.90 0.02 True
948.71 ns 886.32 ns 0.93 0.01 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Tests.Perf_WebUtility*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Tests.Perf_WebUtility* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Tests.Perf_WebUtility*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Tests.Perf_WebUtility* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 354.25732207120916 < 365.4807589907136. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 52.24796314344765 (T) = (0 -352.37196710399286) / Math.Sqrt((45.23402225077658 / (145)) + (6.328199954725564 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.09683732938860072 = (390.1533783116317 - 352.37196710399286) / 390.1533783116317 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Net.Tests.Perf_WebUtility.HtmlDecode_Entities #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 886.3211918959071 < 901.4812759471431. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 30.734536496547 (T) = (0 -886.9741719732342) / Math.Sqrt((602.4901931103167 / (143)) + (68.07706735518579 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.08107131458181291 = (965.226340245967 - 886.9741719732342) / 965.226340245967 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Tests.Perf_Get

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
31.43 μs 24.67 μs 0.78 0.01 True
10.10 μs 9.23 μs 0.91 0.02 True
10.73 μs 10.07 μs 0.94 0.02 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Tests.Perf_Get* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Tests.Perf_Get* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Tests.Perf_Get.GetString #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 24.67011413571429 < 29.88790182235376. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 61.33891006222529 (T) = (0 -24574.82156382073) / Math.Sqrt((1804070.709765086 / (144)) + (30396.819395608927 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.22507958628236732 = (31712.70381938262 - 24574.82156382073) / 31712.70381938262 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Get.GetDouble #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.230101214275978 < 9.675239512372487. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 27.777510038899944 (T) = (0 -9221.092166089791) / Math.Sqrt((23048.131956163736 / (141)) + (26876.567944710216 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.08929139705517619 = (10125.183989997358 - 9221.092166089791) / 10125.183989997358 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Get.GetSingle #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 10.072682370942601 < 10.191281336660476. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.75279178715155 (T) = (0 -9999.97119520141) / Math.Sqrt((49186.40703300683 / (144)) + (3129.419149792779 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.064687219031484 = (10691.579756716725 - 9999.97119520141) / 10691.579756716725 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
51.38 μs 39.19 μs 0.76 0.06 True
19.37 μs 18.02 μs 0.93 0.01 True
525.51 μs 396.19 μs 0.75 0.04 True
84.09 μs 72.10 μs 0.86 0.02 True
53.68 μs 43.03 μs 0.80 0.01 True
1.75 μs 1.63 μs 0.93 0.02 False

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_File*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_File* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_File*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_File* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_File.AppendAllText(size: 10000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.19359358974359 < 49.17429107115384. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 148.13535707806872 (T) = (0 -38934.55775520859) / Math.Sqrt((798050.8768734081 / (142)) + (66225.74300014746 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.2518433003767602 = (52040.645729451375 - 38934.55775520859) / 52040.645729451375 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_File.AppendAllLines #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 18.021652000000003 < 18.616541159615384. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.21441734317184 (T) = (0 -18005.044453550767) / Math.Sqrt((49306.83415557702 / (141)) + (51350.61700627366 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.0751304048838372 = (19467.6574390894 - 18005.044453550767) / 19467.6574390894 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_File.WriteAllText(size: 100000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 396.18712124999996 < 498.4577431215277. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 116.58667756476052 (T) = (0 -396834.40235727926) / Math.Sqrt((143065320.24809116 / (144)) + (5209230.488362951 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.24092744377103417 = (522788.4990714624 - 396834.40235727926) / 522788.4990714624 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_File.WriteAllText(size: 10000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 72.0997948487443 < 79.89067735064988. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 61.290104725418814 (T) = (0 -71850.36174497688) / Math.Sqrt((1886841.8947360031 / (142)) + (934948.3200065481 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.15243444267630565 = (84772.63041676015 - 71850.36174497688) / 84772.63041676015 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_File.ReadAllLines #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 43.02748709907517 < 51.023096699155055. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 71.4963502887724 (T) = (0 -43271.89542579865) / Math.Sqrt((2769665.956258504 / (142)) + (99677.68003084573 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.19976616740064426 = (54074.06393358916 - 43271.89542579865) / 54074.06393358916 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_File.Exists #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.627172788571577 < 1.6485187823882994. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.357453839385514 (T) = (0 -1625.8045479655277) / Math.Sqrt((676.1130902404618 / (144)) + (139.1757512172071 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.058987968508676374 = (1727.7191933337338 - 1625.8045479655277) / 1727.7191933337338 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Net.Primitives.Tests.CredentialCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
140.83 ns 94.26 ns 0.67 0.08 True
154.58 ns 116.19 ns 0.75 0.05 True
286.86 ns 257.19 ns 0.90 0.03 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Primitives.Tests.CredentialCacheTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Primitives.Tests.CredentialCacheTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_HostPort(host: "notfound", hostPortCount: 10) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 94.25997128070252 < 133.91159605962434. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 89.89799243441044 (T) = (0 -97.9223559467829) / Math.Sqrt((6.368570267899295 / (142)) + (6.02786311805269 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.3127781840969807 = (142.49017374122724 - 97.9223559467829) / 142.49017374122724 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_HostPort(host: "name5", hostPortCount: 10) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 116.18821982187704 < 143.28112412488684. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.89629269827734 (T) = (0 -117.97946990720938) / Math.Sqrt((4.864607171609238 / (143)) + (9.341529042116836 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.22384825461615776 = (152.00567493262955 - 117.97946990720938) / 152.00567493262955 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_Uri(uriString: "http://notfound", uriCount: 0) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 257.1946539214792 < 274.68213929045703. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 41.82467497008939 (T) = (0 -260.18289358553807) / Math.Sqrt((24.76937658069563 / (140)) + (6.745221653986776 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.09246998681513863 = (286.6934314078046 - 260.18289358553807) / 286.6934314078046 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.Perf_Frozen<NotKnownComparable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
94.46 μs 88.21 μs 0.93 0.01 True
82.59 μs 76.03 μs 0.92 0.03 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Perf_Frozen<NotKnownComparable>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Perf_Frozen<NotKnownComparable>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Perf_Frozen<NotKnownComparable>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Perf_Frozen<NotKnownComparable>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.Perf_Frozen<NotKnownComparable>.ToFrozenSet(Count: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 88.21035978464418 < 89.68988384981267. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 29.075807007966976 (T) = (0 -88123.72671656666) / Math.Sqrt((2102708.6900734473 / (87)) + (764935.4819561383 / (30))) is greater than 1.980807541103565 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (87) + (30) - 2, .975) and 0.06849482879204309 = (94603.58293265254 - 88123.72671656666) / 94603.58293265254 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Perf_Frozen<NotKnownComparable>.ToFrozenDictionary(Count: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 76.02790558143549 < 78.47709555335098. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 13.917386936447182 (T) = (0 -76938.53315574283) / Math.Sqrt((2116170.433509182 / (86)) + (5101796.430805421 / (30))) is greater than 1.9809922979755963 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (86) + (30) - 2, .975) and 0.07391148433419757 = (83079.02738695405 - 76938.53315574283) / 83079.02738695405 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
298.39 μs 53.89 μs 0.18 0.01 True
307.78 μs 63.25 μs 0.21 0.11 True
305.38 μs 60.50 μs 0.20 0.10 True
304.37 μs 76.39 μs 0.25 0.10 True
303.99 μs 70.24 μs 0.23 0.08 True
301.33 μs 76.01 μs 0.25 0.08 True
303.37 μs 57.97 μs 0.19 0.09 True

graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsTrue<Int32>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsTrue<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsTrue<Int32>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsTrue<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsTrue<Int32>.Span(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 53.89056013025929 < 283.75607070417783. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2139.2295116485197 (T) = (0 -53185.86283511866) / Math.Sqrt((387283.2141825249 / (144)) + (314567.13826911215 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.8219613522696517 = (298732.12088013784 - 53185.86283511866) / 298732.12088013784 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<Int32>.ImmutableArray(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 63.2469796017403 < 293.1102780428293. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 140.57670204170753 (T) = (0 -70516.07366317231) / Math.Sqrt((2694072.3731266703 / (144)) + (86306308.99019745 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7723285221140912 = (309727.3067226694 - 70516.07366317231) / 309727.3067226694 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<Int32>.Array(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 60.499241071428564 < 290.443729694211. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 138.8286580298168 (T) = (0 -69702.70534458758) / Math.Sqrt((3833328.4000535333 / (145)) + (87656913.48575366 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.7737523166756952 = (308081.4102510623 - 69702.70534458758) / 308081.4102510623 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<Int32>.List(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 76.38691067094314 < 288.9361389855196. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 146.78375927525667 (T) = (0 -68033.60632651445) / Math.Sqrt((2764885.7704246906 / (145)) + (77921025.39935619 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.7772765203736147 = (305462.21009405755 - 68033.60632651445) / 305462.21009405755 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<Int32>.ICollection(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 70.23901855381166 < 288.72399469866065. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 226.1791946147121 (T) = (0 -64103.4890004729) / Math.Sqrt((3550947.3368827743 / (144)) + (33466220.53683169 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7902496094246213 = (305617.9720315506 - 64103.4890004729) / 305617.9720315506 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<Int32>.Stack(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 76.01406947815533 < 286.29242002103365. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 182.04008757784933 (T) = (0 -66172.90082753189) / Math.Sqrt((1156261.666732695 / (144)) + (49894407.28123739 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.7805235258711174 = (301503.38932761125 - 66172.90082753189) / 301503.38932761125 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<Int32>.Queue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 57.96711826032764 < 288.09380926541286. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 220.80509727893775 (T) = (0 -64648.595805024765) / Math.Sqrt((3243646.782039148 / (146)) + (34789384.77291117 / (30))) is greater than 1.9736914397558152 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (146) + (30) - 2, .975) and 0.7878242319567421 = (304693.58683713764 - 64648.595805024765) / 304693.58683713764 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.43 μs 780.40 ns 0.54 0.01 True
91.78 ns 34.64 ns 0.38 0.01 True
91.30 ns 33.94 ns 0.37 0.01 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Array*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Array* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Array*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Array* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Array.Reverse #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 780.3985676755996 < 1.3628234455449695. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1038.3730229816276 (T) = (0 -782.5617262445786) / Math.Sqrt((31.699176767741548 / (144)) + (5.408630518786987 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.4564153958930859 = (1439.6318812787083 - 782.5617262445786) / 1439.6318812787083 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Array.IndexOfChar #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 34.64030211797311 < 87.01826887681669. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 277.5069944931548 (T) = (0 -34.50079321871122) / Math.Sqrt((0.085406034789002 / (144)) + (1.2543563885985678 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.6235442101793597 = (91.64633444779501 - 34.50079321871122) / 91.64633444779501 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Array.IndexOfShort #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 33.93532273534889 < 86.76555847326757. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 299.0133974815353 (T) = (0 -33.72522817109502) / Math.Sqrt((0.03398053560571406 / (142)) + (1.1082563562898573 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.6309433457341249 = (91.382251969365 - 33.72522817109502) / 91.382251969365 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Hashing

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.33 μs 534.19 ns 0.40 0.01 True
147.13 ns 65.55 ns 0.45 0.01 True
13.16 μs 5.20 μs 0.40 0.01 True
22.65 ns 12.56 ns 0.55 0.02 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Hashing*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Hashing* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Hashing*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Hashing* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Hashing.GetStringHashCode(BytesCount: 1000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 534.1851895342771 < 1.263102822090035. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1346.484868101275 (T) = (0 -535.748683090816) / Math.Sqrt((13.241451755350207 / (143)) + (7.643682139820884 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5969877247661366 = (1329.3607068914396 - 535.748683090816) / 1329.3607068914396 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Hashing.GetStringHashCode(BytesCount: 100) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 65.55147757940526 < 140.32832822523267. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 922.783519509609 (T) = (0 -65.46504441934422) / Math.Sqrt((0.4613846692191633 / (143)) + (0.1425861794604708 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5573535677065371 = (147.89466184140082 - 65.46504441934422) / 147.89466184140082 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Hashing.GetStringHashCode(BytesCount: 10000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.202468772420951 < 12.458970059811405. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2293.27445244176 (T) = (0 -5193.974207429723) / Math.Sqrt((1144.391437736264 / (140)) + (116.1573965721369 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.6051220842404291 = (13153.367155108706 - 5193.974207429723) / 13153.367155108706 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Hashing.GetStringHashCode(BytesCount: 10) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 12.564634086215571 < 21.49953108696148. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 229.89607660732034 (T) = (0 -12.694338358193367) / Math.Sqrt((0.004131916530792676 / (145)) + (0.05466808137351506 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.43792029629409357 = (22.584587691918063 - 12.694338358193367) / 22.584587691918063 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Net.Primitives.Tests.IPAddressPerformanceTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
22.73 ns 17.73 ns 0.78 0.02 True
23.71 ns 20.47 ns 0.86 0.03 False
72.14 ns 58.85 ns 0.82 0.06 True
92.96 ns 79.32 ns 0.85 0.01 True
80.77 ns 69.85 ns 0.86 0.03 True
92.32 ns 77.56 ns 0.84 0.01 True

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Primitives.Tests.IPAddressPerformanceTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Primitives.Tests.IPAddressPerformanceTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [143, 24, 20, 36]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.734432152867264 < 21.39229232268777. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 101.17730025889003 (T) = (0 -17.507191401212843) / Math.Sqrt((0.14377906566428003 / (144)) + (0.043640341633346756 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.22253923496101025 = (22.51842432246064 - 17.507191401212843) / 22.51842432246064 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Bytes(address: [143, 24, 20, 36]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.474975356523874 < 22.494153389947193. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 22.217783495117715 (T) = (0 -19.934379526947016) / Math.Sqrt((0.8433626673317949 / (144)) + (0.6788282275503785 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.15832366814328228 = (23.684139344838478 - 19.934379526947016) / 23.684139344838478 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Net.Primitives.Tests.IPAddressPerformanceTests.TryWriteBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 58.84581084776077 < 68.57672962603795. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 88.77156609548483 (T) = (0 -58.98326896260157) / Math.Sqrt((1.2955865149085364 / (145)) + (0.4416125503232895 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.18796810976228762 = (72.63664108725452 - 58.98326896260157) / 72.63664108725452 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Bytes(address: [16, 32, 48, 64, 80, ...]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 79.32479633759371 < 88.32564673047231. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 151.2000557216983 (T) = (0 -79.53936336376248) / Math.Sqrt((0.36026575811758715 / (144)) + (0.16881897285589556 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.14631530629482895 = (93.17182790117147 - 79.53936336376248) / 93.17182790117147 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Net.Primitives.Tests.IPAddressPerformanceTests.GetAddressBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 69.8499788748237 < 77.91736075588173. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.60065504463104 (T) = (0 -69.2451480780118) / Math.Sqrt((0.6953347090337323 / (142)) + (1.2631611272215633 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.1572586650638884 = (82.16655005211211 - 69.2451480780118) / 82.16655005211211 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [16, 32, 48, 64, 80, ...]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 77.55955698440425 < 87.09012197207562. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 132.3167251298191 (T) = (0 -78.32927461311604) / Math.Sqrt((0.524874970876452 / (143)) + (0.19470051430685487 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.1454990390170646 = (91.66668990402727 - 78.32927461311604) / 91.66668990402727 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.BinaryWriterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.27 ns 30.81 ns 0.85 0.02 True
18.39 ns 17.03 ns 0.93 0.01 True
18.03 ns 16.69 ns 0.93 0.01 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.BinaryWriterTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.BinaryWriterTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.BinaryWriterTests.WriteHalf #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 30.807163377408703 < 34.39480931409674. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 44.14953700805671 (T) = (0 -30.802134213944658) / Math.Sqrt((0.09436262695642927 / (141)) + (0.44747121325987715 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.1517776136212036 = (36.313748267649636 - 30.802134213944658) / 36.313748267649636 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryWriterTests.WriteUInt64 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.025097688273064 < 17.48543946615717. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 92.88528158868742 (T) = (0 -17.01371137556317) / Math.Sqrt((0.024399927227499327 / (144)) + (0.002222900441314531 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.07850986578384306 = (18.463259392392157 - 17.01371137556317) / 18.463259392392157 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryWriterTests.WriteDouble #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.687662357333103 < 17.241354589370474. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 82.6137558251766 (T) = (0 -16.73240371124936) / Math.Sqrt((0.017676594818986095 / (142)) + (0.004941838475789766 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.07746150676984925 = (18.13735018542477 - 16.73240371124936) / 18.13735018542477 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Tests.Perf_Base64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.93 μs 1.63 μs 0.84 0.01 True
339.81 ns 312.55 ns 0.92 0.03 True
1.94 μs 1.63 μs 0.84 0.02 True
338.73 ns 309.57 ns 0.91 0.02 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Base64*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Tests.Perf_Base64* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Base64*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.Json.Tests.Perf_Base64* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_HeavyEscaping(NumberOfBytes: 1000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.6289314835983235 < 1.8394396375365316. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 96.23353786149758 (T) = (0 -1628.0436194426866) / Math.Sqrt((132.6839860470975 / (144)) + (274.220485337186 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.15789579171902512 = (1933.304219873316 - 1628.0436194426866) / 1933.304219873316 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_HeavyEscaping(NumberOfBytes: 100) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 312.55430729660804 < 323.480618347511. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 21.73001614631938 (T) = (0 -311.9784458115542) / Math.Sqrt((13.8117902195905 / (144)) + (43.37845901276509 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.07960371363827616 = (338.9610002065392 - 311.9784458115542) / 338.9610002065392 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_NoEscaping(NumberOfBytes: 1000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.6282753959773024 < 1.83529166031896. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 80.73030363359894 (T) = (0 -1622.0598660614155) / Math.Sqrt((249.93776132785615 / (143)) + (391.58485475700695 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.1607035325110045 = (1932.6423128102624 - 1622.0598660614155) / 1932.6423128102624 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_NoEscaping(NumberOfBytes: 100) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 309.56668066666094 < 322.8933014061644. IsChangePoint: Marked as a change because one of 9/26/2023 6:21:18 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 43.15666949390103 (T) = (0 -310.9563634502097) / Math.Sqrt((24.111809768462376 / (142)) + (7.647348934348252 / (29))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (29) - 2, .975) and 0.08372770094964728 = (339.3711277449865 - 310.9563634502097) / 339.3711277449865 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
396.18 μs 348.15 μs 0.88 0.02 True
334.28 μs 292.48 μs 0.87 0.01 True
364.60 μs 337.03 μs 0.92 0.02 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "names.xml") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 348.1524625 < 381.4986115721154. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 40.64185849841341 (T) = (0 -351282.70746699494) / Math.Sqrt((29094851.941851363 / (145)) + (42318891.99908241 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.1280538098835592 = (402872.0022506024 - 351282.70746699494) / 402872.0022506024 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "repeated.xml") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 292.48331080840455 < 318.23815324373095. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 63.34519256590781 (T) = (0 -291607.18427179515) / Math.Sqrt((19308042.41881197 / (143)) + (10809934.548136804 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.1326128405521718 = (336190.3402598558 - 291607.18427179515) / 336190.3402598558 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "deep.xml") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 337.0335928951368 < 347.1433691327519. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 50.377662253423544 (T) = (0 -335230.1006957113) / Math.Sqrt((45123151.95613587 / (143)) + (4083893.0095711146 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.09173234493501087 = (369087.3486755671 - 335230.1006957113) / 369087.3486755671 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.Perf_FileInfo

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
292.90 ns 243.50 ns 0.83 0.06 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_FileInfo*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_FileInfo* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_FileInfo*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_FileInfo* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_FileInfo.ctor_str #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 243.49518781512612 < 275.45655804958847. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 24.34650690630021 (T) = (0 -248.55247519552168) / Math.Sqrt((77.18410080607744 / (143)) + (49.956732952638184 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.12698232644982618 = (284.70497531254955 - 248.55247519552168) / 284.70497531254955 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
24.05 ns 10.87 ns 0.45 0.01 True
70.62 ns 43.76 ns 0.62 0.02 True
24.02 ns 10.86 ns 0.45 0.01 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf<Double>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf<Double>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf<Double>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf<Double>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_VectorOf<Double>.LessThanOrEqualBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 10.871729221558672 < 22.7637493277383. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1000.5140555305196 (T) = (0 -10.86125251045364) / Math.Sqrt((0.015099272857329024 / (143)) + (0.002040951803598043 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.5482864925504127 = (24.04455995079092 - 10.86125251045364) / 24.04455995079092 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorOf<Double>.EqualsBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 43.76071119385178 < 66.93815955284926. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 520.5548341946917 (T) = (0 -43.77970264276233) / Math.Sqrt((0.2883350541864852 / (144)) + (0.018313587567102482 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.3780236348684647 = (70.38804864153289 - 43.77970264276233) / 70.38804864153289 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorOf<Double>.LessThanBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 10.86300229823974 < 22.807883118345924. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1068.8692673753924 (T) = (0 -10.870190011149562) / Math.Sqrt((0.013112521503065883 / (144)) + (0.0018004978106279943 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.5472257834565274 = (24.007970449673063 - 10.870190011149562) / 24.007970449673063 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
216.68 ns 201.88 ns 0.93 0.05 True
942.93 ns 807.47 ns 0.86 0.01 True
941.01 ns 811.18 ns 0.86 0.02 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Enum*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Enum* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Enum*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Enum* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Enum.TryParseGeneric_Flags(text: "Red") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 201.87879035748287 < 208.70515733329054. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.473829599709797 (T) = (0 -204.51445077451743) / Math.Sqrt((37.3889077538529 / (141)) + (44.305263881660274 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.06331417934859346 = (218.3383651866219 - 204.51445077451743) / 218.3383651866219 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Enum.Parse_Flags(text: "Red, Orange, Yellow, Green, Blue") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 807.473585715022 < 895.2946326754004. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 100.96648210113506 (T) = (0 -806.3271482069814) / Math.Sqrt((266.6667839537878 / (142)) + (10.127047377407974 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.1571006147511515 = (956.6113848439101 - 806.3271482069814) / 956.6113848439101 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Enum.TryParseGeneric_Flags(text: "Red, Orange, Yellow, Green, Blue") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 811.1766671875272 < 896.8243219069593. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 94.362882786176 (T) = (0 -808.8928634321503) / Math.Sqrt((265.6927682082636 / (142)) + (17.61016020720547 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.15461837673090723 = (956.8375289542723 - 808.8928634321503) / 956.8375289542723 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
87.41 μs 82.58 μs 0.94 0.01 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.CtorFromCollection<Int32>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.CtorFromCollection<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.CtorFromCollection<Int32>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.CtorFromCollection<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.CtorFromCollection<Int32>.FrozenSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 82.58365876771255 < 83.61681319028807. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 37.13175733216695 (T) = (0 -82239.01866695202) / Math.Sqrt((1018346.7563801942 / (86)) + (372400.5548605536 / (30))) is greater than 1.9809922979755963 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (86) + (30) - 2, .975) and 0.0656979591800392 = (88021.87630327504 - 82239.01866695202) / 88021.87630327504 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
42.66 ns 32.71 ns 0.77 0.22 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.AbsBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 32.713406041097656 < 40.524418391373494. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 73.08291536515934 (T) = (0 -32.667257685609236) / Math.Sqrt((2.226029727710079 / (143)) + (0.006494172769226348 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.21939662988929667 = (41.84872745422101 - 32.667257685609236) / 41.84872745422101 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
82.90 ns 64.39 ns 0.78 0.03 True
2.67 μs 2.23 μs 0.83 0.06 False
483.48 ns 405.71 ns 0.84 0.13 True
3.56 μs 3.29 μs 0.92 0.01 False
5.07 μs 4.25 μs 0.84 0.08 False
999.01 ns 813.46 ns 0.81 0.07 False
102.52 ns 72.95 ns 0.71 0.02 True
529.81 ns 444.58 ns 0.84 0.06 False

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Path*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_Path* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Path*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.Perf_Path* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_Path.GetFileName #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 64.38655090016367 < 78.67898930218263. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 47.095684147720334 (T) = (0 -65.02848613805084) / Math.Sqrt((5.643243543788674 / (144)) + (3.2589431054694162 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.217801931686648 = (83.13557495516613 - 65.02848613805084) / 83.13557495516613 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_Path.GetFullPathForTypicalLongPath #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.2285857961064104 < 2.5531533386158594. IsChangePoint: Marked as a change because one of 9/5/2023 1:44:21 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 25.23194341330346 (T) = (0 -2261.660070113831) / Math.Sqrt((11774.901979190134 / (145)) + (3151.4862454096397 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.1321383099131198 = (2606.014409839222 - 2261.660070113831) / 2606.014409839222 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_Path.GetTempPath #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 405.7117279322276 < 467.11076706506594. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.48408190917442 (T) = (0 -401.113183152033) / Math.Sqrt((156.60484109777056 / (141)) + (121.33928642880875 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.17887548953282353 = (488.4925221923053 - 401.113183152033) / 488.4925221923053 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_Path.GetFullPathWithRedundantSegments #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.294724311552232 < 3.3724792006304907. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 30.86441322813408 (T) = (0 -3262.985114669111) / Math.Sqrt((6097.011772794092 / (144)) + (1007.0109825522271 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.07613647882028705 = (3531.890847365089 - 3262.985114669111) / 3531.890847365089 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_Path.GetFullPathForReallyLongPath #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 4.25182575383931 < 4.900127822057451. IsChangePoint: Marked as a change because one of 9/5/2023 1:44:21 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 24.078666150216694 (T) = (0 -4271.058899001671) / Math.Sqrt((47289.81306933355 / (143)) + (15486.123655384987 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.14094075387122634 = (4971.786193150916 - 4271.058899001671) / 4971.786193150916 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_Path.GetFullPathForLegacyLength #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 813.4643953238525 < 948.7761209460227. IsChangePoint: Marked as a change because one of 9/5/2023 1:44:21 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 19.932199634594436 (T) = (0 -833.6777754383133) / Math.Sqrt((1821.1020169951717 / (143)) + (1072.8789025530255 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.142735502274935 = (972.4860619454741 - 833.6777754383133) / 972.4860619454741 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_Path.GetFileNameWithoutExtension #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 72.95416314994598 < 97.3842828111952. IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 65.19083290543011 (T) = (0 -73.21475250494422) / Math.Sqrt((24.502920094365233 / (144)) + (1.1109258786232812 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.28840616261199303 = (102.88840158268937 - 73.21475250494422) / 102.88840158268937 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_Path.GetFullPathNoRedundantSegments #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 444.5784029472274 < 514.4709116671787. IsChangePoint: Marked as a change because one of 9/5/2023 1:44:21 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 21.908860992167245 (T) = (0 -451.78697371696734) / Math.Sqrt((325.34720210414963 / (144)) + (286.3667609814408 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.14282014500083165 = (527.0620524760299 - 451.78697371696734) / 527.0620524760299 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Matrix3x2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
114.73 ns 59.05 ns 0.51 0.01 True
117.94 ns 63.81 ns 0.54 0.01 True
44.48 ns 36.91 ns 0.83 0.01 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix3x2*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Matrix3x2* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix3x2*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Matrix3x2* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_Matrix3x2.CreateRotationBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 59.04595174003462 < 107.03889517613442. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 390.0038868848448 (T) = (0 -60.11729963103323) / Math.Sqrt((0.8462763273763318 / (144)) + (0.3676064430957346 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.4662459944852119 = (112.631097865115 - 60.11729963103323) / 112.631097865115 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_Matrix3x2.CreateRotationWithCenterBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 63.81486624048565 < 111.89621647424865. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 403.30176937133325 (T) = (0 -63.34589742749837) / Math.Sqrt((0.9276634029521038 / (143)) + (0.34725384204296394 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.46110631530176427 = (117.54804189804186 - 63.34589742749837) / 117.54804189804186 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_Matrix3x2.InvertBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 36.910227698627565 < 42.39617517948172. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 403.26216532802795 (T) = (0 -36.885217851938016) / Math.Sqrt((0.0208075631996397 / (141)) + (0.006838206648251921 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.1748214536849952 = (44.69968107709068 - 36.885217851938016) / 44.69968107709068 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.TryGetValueFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
71.84 μs 51.25 μs 0.71 0.01 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.TryGetValueFalse<String, String>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.TryGetValueFalse<String, String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.TryGetValueFalse<String, String>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.TryGetValueFalse<String, String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.TryGetValueFalse<String, String>.ImmutableDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 51.25026930509868 < 68.81509393810506. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 130.4678501497662 (T) = (0 -51751.44285300181) / Math.Sqrt((569412.7171911789 / (142)) + (655210.4511219334 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.2884257527944197 = (72728.0997819056 - 51751.44285300181) / 72728.0997819056 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.Perf_LengthBucketsFrozenDictionary

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
484.63 ns 397.74 ns 0.82 0.02 True
3.00 μs 2.60 μs 0.87 0.04 True
660.22 μs 481.86 μs 0.73 0.01 True
46.03 μs 41.79 μs 0.91 0.01 True
5.83 μs 4.11 μs 0.71 0.01 True
58.65 μs 44.80 μs 0.76 0.06 True

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Perf_LengthBucketsFrozenDictionary*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Perf_LengthBucketsFrozenDictionary* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Perf_LengthBucketsFrozenDictionary*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Perf_LengthBucketsFrozenDictionary* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.Perf_LengthBucketsFrozenDictionary.TryGetValue_True_FrozenDictionary(Count: 10, ItemsPerBucket: 5) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 397.73556415829245 < 455.04885011062953. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 49.101932443338384 (T) = (0 -387.1963162436081) / Math.Sqrt((9.621991914520168 / (86)) + (100.9648407064702 / (30))) is greater than 1.9809922979755963 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (86) + (30) - 2, .975) and 0.19125220399338777 = (478.7602737905235 - 387.1963162436081) / 478.7602737905235 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Perf_LengthBucketsFrozenDictionary.ToFrozenDictionary(Count: 100, ItemsPerBucket: 1) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.600998591752565 < 2.8771736637274237. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 21.735555958627963 (T) = (0 -2516.711184352275) / Math.Sqrt((25203.440802774916 / (87)) + (3272.005279700925 / (30))) is greater than 1.980807541103565 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (87) + (30) - 2, .975) and 0.14709406677420855 = (2950.7488297493423 - 2516.711184352275) / 2950.7488297493423 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Perf_LengthBucketsFrozenDictionary.TryGetValue_True_FrozenDictionary(Count: 10000, ItemsPerBucket: 5) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 481.8630598484848 < 625.9399525548736. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 77.5020924310117 (T) = (0 -490339.05901813635) / Math.Sqrt((36127707.798188336 / (86)) + (126640021.59463641 / (30))) is greater than 1.9809922979755963 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (86) + (30) - 2, .975) and 0.25402079312190073 = (657309.2848930612 - 490339.05901813635) / 657309.2848930612 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Perf_LengthBucketsFrozenDictionary.ToFrozenDictionary(Count: 1000, ItemsPerBucket: 1) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 41.788023832775295 < 44.017865308626575. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 52.209890903832544 (T) = (0 -41219.928516906926) / Math.Sqrt((347810.1324474286 / (87)) + (105068.11418894563 / (30))) is greater than 1.980807541103565 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (87) + (30) - 2, .975) and 0.09884982501703107 = (45741.46425448561 - 41219.928516906926) / 45741.46425448561 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Perf_LengthBucketsFrozenDictionary.TryGetValue_True_FrozenDictionary(Count: 100, ItemsPerBucket: 5) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 4.1098085803286315 < 5.487775159018668. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 106.56336914759382 (T) = (0 -4165.600501445381) / Math.Sqrt((2876.4692508802505 / (86)) + (6158.815215316532 / (30))) is greater than 1.9809922979755963 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (86) + (30) - 2, .975) and 0.2832928953143799 = (5812.13786526171 - 4165.600501445381) / 5812.13786526171 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Perf_LengthBucketsFrozenDictionary.TryGetValue_True_FrozenDictionary(Count: 1000, ItemsPerBucket: 5) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 44.795336087164735 < 55.15916124571487. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.640412816392256 (T) = (0 -43271.31955378855) / Math.Sqrt((771294.0335108086 / (88)) + (1729002.513526715 / (30))) is greater than 1.9806260024587088 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (88) + (30) - 2, .975) and 0.25553444524151664 = (58124.00490151148 - 43271.31955378855) / 58124.00490151148 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
70.90 μs 50.00 μs 0.71 0.01 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsFalse<String>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsFalse<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsFalse<String>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsFalse<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsFalse<String>.ImmutableHashSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 50.00260327476039 < 67.93366538630866. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 155.395763539635 (T) = (0 -50517.40271305898) / Math.Sqrt((514139.0526175877 / (141)) + (444460.3755939142 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.2947612706766047 = (71631.63424323744 - 50517.40271305898) / 71631.63424323744 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Char

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
42.98 ns 39.43 ns 0.92 0.01 True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Char*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Char* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Char*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Char* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Char.GetUnicodeCategory(c: 'א') #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.42812396765736 < 40.828986858106816. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 104.2603521381537 (T) = (0 -39.37279903178782) / Math.Sqrt((0.03450760124123082 / (144)) + (0.028633794478157698 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.08383337919097371 = (42.97558777793 - 39.37279903178782) / 42.97558777793 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.ContainsTrue<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
75.51 μs 55.61 μs 0.74 0.02 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsTrue<String>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsTrue<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsTrue<String>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsTrue<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsTrue<String>.ImmutableHashSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 55.60815400355872 < 71.75633124094203. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 195.5244945309016 (T) = (0 -56006.038151154244) / Math.Sqrt((664414.8712854239 / (142)) + (174851.40380840818 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.26354702025636756 = (76048.35568816705 - 56006.038151154244) / 76048.35568816705 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
303.11 ns 275.82 ns 0.91 0.02 True
300.53 ns 282.13 ns 0.94 0.02 True
308.15 ns 271.61 ns 0.88 0.02 True
648.35 ns 554.30 ns 0.85 0.10 False
298.98 ns 276.26 ns 0.92 0.01 True

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Double*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Double* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Double*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Double* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Double.ToStringWithFormat(value: 12345, format: "G") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 275.8192350644977 < 287.4286228654535. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 25.352472925976112 (T) = (0 -279.26006614662936) / Math.Sqrt((12.899463722390639 / (145)) + (25.757596241708605 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.08119607310174871 = (303.9386946128657 - 279.26006614662936) / 303.9386946128657 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Double.ToStringWithFormat(value: 12345, format: "R") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 282.1273111662888 < 286.73293696354773. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 46.54245545626521 (T) = (0 -278.7199402222788) / Math.Sqrt((19.868413811452236 / (142)) + (5.077278865399562 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.0849598575129207 = (304.59859330839623 - 278.7199402222788) / 304.59859330839623 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Double.ToStringWithCultureInfo(value: 12345, culture: zh) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 271.61126588833025 < 282.0635848058535. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 50.698596558716844 (T) = (0 -271.87810236282485) / Math.Sqrt((14.00203693706179 / (143)) + (4.949868403423002 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.08727077482734351 = (297.87377774760637 - 271.87810236282485) / 297.87377774760637 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Double.ToStringWithFormat(value: 12345, format: "G17") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 554.3010354109662 < 563.3256959180502. IsChangePoint: Marked as a change because one of 8/21/2023 4:59:56 PM, 9/5/2023 11:57:06 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 14.160173362841627 (T) = (0 -553.4787686461494) / Math.Sqrt((504.61480210903454 / (144)) + (69.73908687872762 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.058174361750363675 = (587.6658546636917 - 553.4787686461494) / 587.6658546636917 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Double.ToString(value: 12345) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 276.25628252833843 < 281.9956972280659. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 53.21494821123747 (T) = (0 -273.8399193708039) / Math.Sqrt((13.61725861311262 / (141)) + (4.426988527517563 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.08760732984809141 = (300.1338440445943 - 273.8399193708039) / 300.1338440445943 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in IfStatements.IfStatements

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
80.65 μs 71.30 μs 0.88 0.04 False
92.79 μs 82.08 μs 0.88 0.04 False
82.02 μs 74.70 μs 0.91 0.03 True
91.43 μs 83.33 μs 0.91 0.01 True
86.65 μs 77.58 μs 0.90 0.05 False

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'IfStatements.IfStatements*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter IfStatements.IfStatements* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'IfStatements.IfStatements*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter IfStatements.IfStatements* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### IfStatements.IfStatements.Single #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 71.29876707572298 < 76.76667467631859. IsChangePoint: Marked as a change because one of 9/6/2023 3:47:50 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 29.432036335263735 (T) = (0 -71776.6310361222) / Math.Sqrt((3979887.0250834967 / (144)) + (1294409.372514944 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.09836464889221354 = (79607.16152925289 - 71776.6310361222) / 79607.16152925289 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### IfStatements.IfStatements.OrArray #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 82.0771620200698 < 88.61611352054625. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 9.417763357946933 (T) = (0 -82137.82817390218) / Math.Sqrt((34122203.624309406 / (142)) + (1434968.4718860767 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.057977525862833126 = (87193.06643839389 - 82137.82817390218) / 87193.06643839389 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### IfStatements.IfStatements.Or #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 74.69761065476193 < 77.85425291541705. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 39.835459388871016 (T) = (0 -74364.80793566076) / Math.Sqrt((1857232.187161126 / (142)) + (1310165.7593341726 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.11316983954660859 = (83854.62205936004 - 74364.80793566076) / 83854.62205936004 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### IfStatements.IfStatements.AndOr #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 83.33391731572951 < 86.72544350877683. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 31.68612141452222 (T) = (0 -83952.97688055925) / Math.Sqrt((2739836.2404898875 / (142)) + (1324352.2029337888 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.08681089718509807 = (91933.83563357744 - 83952.97688055925) / 91933.83563357744 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### IfStatements.IfStatements.And #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 77.58415984660033 < 82.26729694029783. IsChangePoint: Marked as a change because one of 9/9/2023 10:04:41 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 16.69560954256854 (T) = (0 -78526.22046746573) / Math.Sqrt((4195527.948958546 / (145)) + (4779046.855159375 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.08445384806438531 = (85769.81105916771 - 78526.22046746573) / 85769.81105916771 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.BinaryWriterExtendedTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
198.45 ns 145.53 ns 0.73 0.01 True
25.79 μs 15.88 μs 0.62 0.03 True
216.89 ns 180.27 ns 0.83 0.02 True
8.50 ms 5.93 ms 0.70 0.03 True
6.53 ms 3.93 ms 0.60 0.04 True
26.12 μs 15.83 μs 0.61 0.05 True

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.BinaryWriterExtendedTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.BinaryWriterExtendedTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 32) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 145.53019698966227 < 189.0149285966367. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 214.10889481939824 (T) = (0 -145.6696599821603) / Math.Sqrt((6.249565388448628 / (143)) + (0.6570768538350373 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.2735068915902394 = (200.51072514785497 - 145.6696599821603) / 200.51072514785497 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 8000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.879675281461461 < 24.520594376831244. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 273.1859440784092 (T) = (0 -15934.208293791427) / Math.Sqrt((145365.23061468886 / (143)) + (10041.14357434651 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.38658733736617756 = (25976.32762482338 - 15934.208293791427) / 25976.32762482338 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 32) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 180.26791366926173 < 207.3612814847254. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 65.68865095386968 (T) = (0 -164.73308144027672) / Math.Sqrt((11.983287811412525 / (143)) + (17.428881141126695 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.24535089592400136 = (218.29096536459534 - 164.73308144027672) / 218.29096536459534 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 2000000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.933379566666666 < 8.130718773593749. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 80.32140993774327 (T) = (0 -6003106.822403084) / Math.Sqrt((52240320002.41242 / (144)) + (18464175958.896313 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.29502291167121025 = (8515321.876110863 - 6003106.822403084) / 8515321.876110863 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 2000000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.929082759895833 < 6.149799541649306. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 98.86980022022404 (T) = (0 -3968188.9025732595) / Math.Sqrt((7265913008.987638 / (145)) + (17975643222.203777 / (30))) is greater than 1.973771336887317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (145) + (30) - 2, .975) and 0.3883352174016625 = (6487522.276035719 - 3968188.9025732595) / 6487522.276035719 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 8000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.833350916593632 < 24.6728689106893. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 88.48115113056055 (T) = (0 -16014.745519909504) / Math.Sqrt((126715.46023854656 / (142)) + (354211.40877811436 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.38371244302713864 = (25985.832974743516 - 16014.745519909504) / 25985.832974743516 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.02 μs 909.89 ns 0.89 0.01 True
1.02 μs 917.74 ns 0.90 0.01 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Random*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Random* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Random*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Random* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Random.NextBytes_span_unseeded #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 909.8886214477334 < 0.9695800793827489. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 251.7413080157952 (T) = (0 -913.6066222825068) / Math.Sqrt((7.963452624869199 / (141)) + (3.674762315902385 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.104399895477278 = (1020.1055333388787 - 913.6066222825068) / 1020.1055333388787 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Random.NextBytes_unseeded #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 917.7354176437254 < 0.9720901556404373. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 246.66907477201087 (T) = (0 -918.9165045280341) / Math.Sqrt((7.964312160942382 / (142)) + (3.796924694510925 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.10291594225878159 = (1024.3371249309546 - 918.9165045280341) / 1024.3371249309546 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Memory.SequenceReader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.47 μs 362.37 ns 0.25 0.02 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.SequenceReader*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.SequenceReader* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.SequenceReader*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.SequenceReader* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Memory.SequenceReader.TryReadTo #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 362.37068737282647 < 1.3935785489334718. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 595.7428857527373 (T) = (0 -358.36161262854336) / Math.Sqrt((20.550456276349752 / (143)) + (100.51797341472107 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.7565458244401916 = (1471.9879492907116 - 358.36161262854336) / 1471.9879492907116 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Span.Sorting

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
18.48 μs 14.33 μs 0.78 0.02 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Span.Sorting*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Span.Sorting* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Span.Sorting*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Span.Sorting* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### Span.Sorting.QuickSortArray(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.32578405 < 17.5030721085. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 9.659207228014926 (T) = (0 -14334.667113774114) / Math.Sqrt((4311082.518768197 / (144)) + (11190.878767323164 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.10499750578525698 = (16016.34320175952 - 14334.667113774114) / 16016.34320175952 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
209.36 μs 181.95 μs 0.87 0.02 True
191.29 μs 172.85 μs 0.90 0.02 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.CtorFromCollection<String>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.CtorFromCollection<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.CtorFromCollection<String>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.CtorFromCollection<String>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Collections.CtorFromCollection<String>.ImmutableDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 181.94966656976746 < 195.0416731696715. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 44.47184164185917 (T) = (0 -183935.06821827695) / Math.Sqrt((14919070.724239504 / (144)) + (2671372.222892159 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.09594061373068076 = (203454.63031726403 - 183935.06821827695) / 203454.63031726403 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.ImmutableHashSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 172.84776625457874 < 181.496970753938. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 51.48787135567622 (T) = (0 -173294.80652703944) / Math.Sqrt((6161269.080652997 / (143)) + (1718010.2108661772 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.08602387443203427 = (189605.3974269296 - 173294.80652703944) / 189605.3974269296 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
37.76 μs 19.24 μs 0.51 0.01 True
37.84 μs 19.74 μs 0.52 0.01 True
50.70 μs 36.03 μs 0.71 0.02 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorConvert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorConvert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_VectorConvert.Convert_long_double #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.237736770577396 < 35.86626463092112. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1893.8713743395517 (T) = (0 -19268.912046664962) / Math.Sqrt((8106.93349672447 / (140)) + (1093.9484208242584 / (30))) is greater than 1.9741851911431834 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (140) + (30) - 2, .975) and 0.4884396177462674 = (37666.935742314054 - 19268.912046664962) / 37666.935742314054 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorConvert.Convert_ulong_double #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.738975068478723 < 35.9487764231052. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1748.6043080156371 (T) = (0 -19738.302496377568) / Math.Sqrt((6547.06039381397 / (141)) + (1822.716491206304 / (30))) is greater than 1.9741004473987562 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (141) + (30) - 2, .975) and 0.4784033528494422 = (37842.08085731837 - 19738.302496377568) / 37842.08085731837 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorConvert.Convert_uint_float #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 36.025700290808544 < 48.17385921811765. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 63.75097916490544 (T) = (0 -35895.680599092055) / Math.Sqrt((109502.60518128698 / (143)) + (1610659.958585302 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.29300621049558295 = (50772.27145694438 - 35895.680599092055) / 50772.27145694438 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Struct.SpanWrapper

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.96 μs 7.66 μs 0.70 0.01 False
20.81 μs 17.95 μs 0.86 0.08 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Struct.SpanWrapper*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Struct.SpanWrapper* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Struct.SpanWrapper*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Struct.SpanWrapper* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### Struct.SpanWrapper.BaselineSum #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.657998968098504 < 10.434724959977247. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 10.98524483317184 (T) = (0 -7693.4125209660715) / Math.Sqrt((2706183.2213832554 / (143)) + (230.8265226675079 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.16420593581867468 = (9204.914046023887 - 7693.4125209660715) / 9204.914046023887 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Struct.SpanWrapper.WrapperSum #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.953590319332562 < 19.73461480941564. IsChangePoint: Marked as a change because one of 9/18/2023 2:28:59 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 9.09404871093939 (T) = (0 -17968.769100630583) / Math.Sqrt((1547500.338792322 / (144)) + (5827.030932156329 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.05027594265069194 = (18919.98940279732 - 17968.769100630583) / 18919.98940279732 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Vector4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
66.70 ns 19.89 ns 0.30 0.02 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector4* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector4* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_Vector4.AbsBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.88540096817559 < 63.654666056639584. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 681.4768151506737 (T) = (0 -20.06956270469835) / Math.Sqrt((0.1337817553799585 / (142)) + (0.11231235690479599 / (30))) is greater than 1.9740167076307873 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (142) + (30) - 2, .975) and 0.6991925991800861 = (66.71897915408508 - 20.06956270469835) / 66.71897915408508 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Xml.Linq.Perf_XName

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
108.37 ns 99.49 ns 0.92 0.18 False
244.99 ns 195.38 ns 0.80 0.02 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Xml.Linq.Perf_XName*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Xml.Linq.Perf_XName* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Xml.Linq.Perf_XName*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Xml.Linq.Perf_XName* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Xml.Linq.Perf_XName.CreateElement #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 99.48870894268924 < 102.15696845834142. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 22.706855691451995 (T) = (0 -99.06595898393168) / Math.Sqrt((13.489850636457861 / (144)) + (2.2597656589380892 / (30))) is greater than 1.9738521694944169 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (144) + (30) - 2, .975) and 0.0861141163348508 = (108.40079790556189 - 99.06595898393168) / 108.40079790556189 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Xml.Linq.Perf_XName.CreateElementWithNamespace #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 195.38493604261208 < 232.64165896650232. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 84.96020327743803 (T) = (0 -194.29227658944467) / Math.Sqrt((14.26183602469325 / (143)) + (8.068678309516166 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.20980827392144172 = (245.87991771775242 - 194.29227658944467) / 245.87991771775242 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Vector2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.74 ns 13.53 ns 0.37 0.01 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector2*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector2* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector2*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector2* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_Vector2.AbsBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.529405724255799 < 34.91833870220966. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 829.0504446131922 (T) = (0 -14.072671521966345) / Math.Sqrt((0.006182579415911545 / (143)) + (0.021282790023575797 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.6177711490104929 = (36.817397445366225 - 14.072671521966345) / 36.817397445366225 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a644443692370b4a7addebcf9880c35f4c5e631d
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Plane

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
22.61 ns 15.12 ns 0.67 0.04 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order)) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - AOT MONO build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create aot directory mkdir -p $RunDir/artifacts/bin/aot/sgen mkdir -p $RunDir/artifacts/bin/aot/pack cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Plane*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Plane* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200 ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create aot directory mkdir $RunDir\artifacts\bin\aot\sgen mkdir $RunDir\artifacts\bin\aot\pack xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Plane*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Plane* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200 ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_Plane.NormalizeBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.122764229564524 < 21.20634865991815. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 87.03484587545907 (T) = (0 -15.141255919526156) / Math.Sqrt((0.7779129589529263 / (143)) + (0.041453106674728915 / (30))) is greater than 1.9739339541028664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (143) + (30) - 2, .975) and 0.3219256378405824 = (22.329786767496742 - 15.141255919526156) / 22.329786767496742 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)