search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 1073 Improvements on 9/26/2023 6:21:18 AM #22677

Closed performanceautofiler[bot] closed 8 months ago

performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
47.50 μs 41.16 μs 0.87 0.01 True
45.57 μs 39.35 μs 0.86 0.01 True
46.45 μs 39.80 μs 0.86 0.01 True
45.99 μs 38.94 μs 0.85 0.01 True
44.91 μs 38.27 μs 0.85 0.01 True
48.44 μs 41.82 μs 0.86 0.01 True
80.61 μs 67.33 μs 0.84 0.01 False
77.98 μs 65.34 μs 0.84 0.01 False
79.94 μs 66.74 μs 0.83 0.01 False
77.37 μs 64.70 μs 0.84 0.01 False

graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToUtf8Bytes(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 41.16099709645669 < 45.09846537867275. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 121.9804613496455 (T) = (0 -41114.288012322315) / Math.Sqrt((175539.17982830625 / (119)) + (27401.109824827683 / (30))) is greater than 1.976233308895103 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (119) + (30) - 2, .975) and 0.12663530396101957 = (47075.73846159598 - 41114.288012322315) / 47075.73846159598 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToStream(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.35022968134622 < 43.32364547497571. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 128.05268791072217 (T) = (0 -39224.43477927808) / Math.Sqrt((180052.16697192047 / (118)) + (19823.341309263502 / (30))) is greater than 1.9763456545936318 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (30) - 2, .975) and 0.1324403653427429 = (45212.378737254534 - 39224.43477927808) / 45212.378737254534 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToWriter(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.8048294256634 < 44.345308419578664. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 96.16265426984496 (T) = (0 -40204.645107806566) / Math.Sqrt((157715.94637513868 / (118)) + (75360.87783777606 / (30))) is greater than 1.9763456545936318 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (30) - 2, .975) and 0.12920979017681672 = (46170.29986587728 - 40204.645107806566) / 46170.29986587728 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToUtf8Bytes(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 38.93968076196474 < 43.51859724855119. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 105.72273913366625 (T) = (0 -39297.53847163871) / Math.Sqrt((239192.86223419296 / (117)) + (35830.196346386256 / (30))) is greater than 1.9764595626326966 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (117) + (30) - 2, .975) and 0.13277644807566855 = (45314.19653495247 - 39297.53847163871) / 45314.19653495247 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToWriter(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 38.2702980487805 < 42.656416164217035. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 131.40702708976065 (T) = (0 -38401.682383732914) / Math.Sqrt((181185.88240593002 / (118)) + (18312.908160959967 / (30))) is greater than 1.9763456545936318 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (30) - 2, .975) and 0.1368268669254313 = (44488.96856526167 - 38401.682383732914) / 44488.96856526167 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToStream(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 41.823386725713014 < 45.96119456273838. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 109.49986483492236 (T) = (0 -41803.410495145006) / Math.Sqrt((198214.63530056746 / (118)) + (37007.04999585343 / (30))) is greater than 1.9763456545936318 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (30) - 2, .975) and 0.12387021621036189 = (47713.7192098724 - 41803.410495145006) / 47713.7192098724 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeObjectProperty(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 67.3290493355893 < 75.92048712671239. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.8954583339041 (T) = (0 -67214.39718155474) / Math.Sqrt((16552254.13673096 / (118)) + (120014.40414415751 / (30))) is greater than 1.9763456545936318 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (30) - 2, .975) and 0.16075414307090197 = (80089.04259295404 - 67214.39718155474) / 80089.04259295404 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeObjectProperty(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 65.33993726028594 < 74.04823341029888. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.16644375035619 (T) = (0 -65448.95043674344) / Math.Sqrt((16722781.962682558 / (119)) + (50981.11318757853 / (30))) is greater than 1.976233308895103 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (119) + (30) - 2, .975) and 0.1644900603492813 = (78334.1374300156 - 65448.95043674344) / 78334.1374300156 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 66.73676183510639 < 75.53682432987776. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.03886398650107 (T) = (0 -66974.94636114445) / Math.Sqrt((16569190.004243521 / (119)) + (91214.13666562624 / (30))) is greater than 1.976233308895103 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (119) + (30) - 2, .975) and 0.16086526655974306 = (79814.29404854066 - 66974.94636114445) / 79814.29404854066 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToString(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 64.69892719139004 < 73.92704651148118. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 35.05348163793374 (T) = (0 -64888.66803522566) / Math.Sqrt((16409300.808368819 / (120)) + (61086.8238564339 / (30))) is greater than 1.9761224936134032 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (120) + (30) - 2, .975) and 0.16753076189783903 = (77947.22623403712 - 64888.66803522566) / 77947.22623403712 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.04 μs 911.58 ns 0.87 0.01 False
1.51 μs 1.39 μs 0.92 0.02 False
632.77 ns 589.41 ns 0.93 0.01 True
504.98 ns 460.64 ns 0.91 0.01 True
1.18 μs 1.05 μs 0.89 0.01 False
1.62 μs 1.48 μs 0.91 0.01 False

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>.SerializeToString(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 911.5794259325648 < 0.9900867336087579. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 24.490857776272865 (T) = (0 -904.2927308652393) / Math.Sqrt((4210.091614663291 / (125)) + (57.828929680789464 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.13912662295532288 = (1050.4363997985604 - 904.2927308652393) / 1050.4363997985604 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>.SerializeObjectProperty(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.3940375770351998 < 1.4511156410719632. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 21.995873524709747 (T) = (0 -1384.6362643742916) / Math.Sqrt((4517.711285303584 / (124)) + (75.35516894223763 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.09019446937897185 = (1521.903547265916 - 1384.6362643742916) / 1521.903547265916 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>.SerializeToUtf8Bytes(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 589.409359488173 < 600.9070284750724. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 50.10214075923557 (T) = (0 -590.1539658836114) / Math.Sqrt((21.45120359057731 / (124)) + (10.799526666317243 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.058361914269007435 = (626.7311983515148 - 590.1539658836114) / 626.7311983515148 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>.SerializeToWriter(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 460.643618213291 < 479.2316669412132. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 36.68398219845459 (T) = (0 -464.5993283701636) / Math.Sqrt((13.327105201737043 / (125)) + (22.767529697589577 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.0684313272074747 = (498.727943456335 - 464.5993283701636) / 498.727943456335 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>.SerializeToString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.0497182775824692 < 1.1324828348604343. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 25.449402229250552 (T) = (0 -1050.0902861211564) / Math.Sqrt((3987.320816654297 / (124)) + (40.39180363152667 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.12302040163881409 = (1197.394201739086 - 1050.0902861211564) / 1197.394201739086 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>.SerializeObjectProperty(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.4761780793755643 < 1.53576726424564. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 20.448838968382546 (T) = (0 -1477.6491764559416) / Math.Sqrt((4948.982565947325 / (125)) + (159.3753168445284 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.08486467417902407 = (1614.6783265418474 - 1477.6491764559416) / 1614.6783265418474 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Guid

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
23.00 ns 15.92 ns 0.69 0.06 True
17.67 ns 8.07 ns 0.46 0.02 True
16.05 ns 6.34 ns 0.40 0.01 True
18.33 ns 8.94 ns 0.49 0.01 True
16.00 ns 6.35 ns 0.40 0.01 True

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Guid*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Guid*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Guid.ctor_bytes #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.921185701349573 < 21.808184900538556. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 53.06444747039868 (T) = (0 -16.24139226011534) / Math.Sqrt((0.044842381549071025 / (124)) + (0.45683901425733386 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2897432069870681 = (22.86692984831421 - 16.24139226011534) / 22.86692984831421 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Guid.EqualsOperator #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.07071766913262 < 16.780032956962987. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 185.39905197601163 (T) = (0 -8.214088903807893) / Math.Sqrt((0.003421579264107558 / (124)) + (0.07721221822498697 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.5351405334499374 = (17.67004760550205 - 8.214088903807893) / 17.67004760550205 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Guid.EqualsNotSame #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.3440466292302 < 15.208401608234498. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 581.0408261484549 (T) = (0 -6.397771535121655) / Math.Sqrt((0.0022164542627707176 / (125)) + (0.007657672078229434 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.6000872774392033 = (15.997919481416435 - 6.397771535121655) / 15.997919481416435 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Guid.NotEqualsOperator #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.93718950435522 < 17.387822884001345. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 459.9010062557529 (T) = (0 -8.786865306951574) / Math.Sqrt((0.006047941614854098 / (125)) + (0.01143762572644067 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5203560251005648 = (18.319557352500627 - 8.786865306951574) / 18.319557352500627 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Guid.EqualsSame #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.351554142741812 < 15.198160012064552. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 517.9387078862594 (T) = (0 -6.41379464150505) / Math.Sqrt((0.002941516363175094 / (124)) + (0.009550050027690351 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.5989613120878601 = (15.992957375000667 - 6.41379464150505) / 15.992957375000667 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
503.25 ns 68.51 ns 0.14 0.08 True
28.87 ns 15.11 ns 0.52 0.02 True
30.50 ns 14.99 ns 0.49 0.03 True
91.45 ns 68.34 ns 0.75 0.03 True
27.80 ns 20.86 ns 0.75 0.01 True
211.55 ns 81.96 ns 0.39 0.01 True
37.60 ns 18.79 ns 0.50 0.03 False
183.44 ns 58.62 ns 0.32 0.02 True
27.77 ns 20.67 ns 0.74 0.01 True
167.32 ns 49.97 ns 0.30 0.03 True
218.57 ns 87.89 ns 0.40 0.03 True
191.94 ns 64.08 ns 0.33 0.01 True
39.17 ns 23.31 ns 0.60 0.03 False
28.73 ns 15.04 ns 0.52 0.01 True
57.15 ns 16.09 ns 0.28 0.02 True
67.62 ns 50.91 ns 0.75 0.02 False
264.60 ns 40.53 ns 0.15 0.06 True
29.61 ns 15.70 ns 0.53 0.03 True
63.81 ns 47.51 ns 0.74 0.02 False
59.17 ns 17.47 ns 0.30 0.02 True
154.51 ns 33.97 ns 0.22 0.08 True
482.23 ns 43.64 ns 0.09 0.01 True
45.04 ns 28.01 ns 0.62 0.02 False
36.21 ns 23.11 ns 0.64 0.04 True
48.17 ns 32.17 ns 0.67 0.03 False
171.60 ns 39.56 ns 0.23 0.02 True
50.18 ns 23.37 ns 0.47 0.02 True
31.86 ns 13.53 ns 0.42 0.08 False
1.25 μs 923.63 ns 0.74 0.01 True
264.76 ns 41.86 ns 0.16 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Byte>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span<Byte>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Byte>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span<Byte>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Memory.Span<Byte>.SequenceEqual(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 68.51128801828443 < 478.09706505616003. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 833.027227946325 (T) = (0 -64.89741063377973) / Math.Sqrt((1.0655866535621252 / (124)) + (8.067080530705809 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.8711633801859152 = (503.7186688647115 - 64.89741063377973) / 503.7186688647115 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceCompareToDifferent(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.111642866897123 < 27.449712427856376. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 812.1418753249045 (T) = (0 -15.071873951083576) / Math.Sqrt((0.028972173016295365 / (125)) + (0.001724182268457407 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.4781976628955335 = (28.88425919040324 - 15.071873951083576) / 28.88425919040324 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceEqual(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.988615748190153 < 29.07333658944251. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 282.5403091165718 (T) = (0 -15.075773217116367) / Math.Sqrt((0.00716447464901668 / (125)) + (0.08867885646256181 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5070917152853951 = (30.585351645783913 - 15.075773217116367) / 30.585351645783913 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.Reverse(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 68.33545193378129 < 87.33189922534555. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 126.1446249875086 (T) = (0 -67.93927560771836) / Math.Sqrt((0.7807672725308067 / (124)) + (0.9254751143960281 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2635420650970858 = (92.25140009751495 - 67.93927560771836) / 92.25140009751495 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.EndsWith(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.864942411361255 < 26.479028577933395. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 323.0777481588175 (T) = (0 -20.841692883923148) / Math.Sqrt((0.044153668308797356 / (125)) + (0.0038404109548907613 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.25376507848610086 = (27.929131005610486 - 20.841692883923148) / 27.929131005610486 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 81.96479546732688 < 201.45288554353755. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 238.0567891425864 (T) = (0 -81.22103046262133) / Math.Sqrt((37.08382153094854 / (125)) + (0.7389449449874812 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.6242550103557792 = (216.15998270403165 - 81.22103046262133) / 216.15998270403165 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfValue(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 18.791293142492123 < 35.708382214058076. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 42.9820594724623 (T) = (0 -18.888655797166905) / Math.Sqrt((33.50239544637937 / (124)) + (0.0856667522223498 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.5431786264302128 = (41.34801235231902 - 18.888655797166905) / 41.34801235231902 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyTwoValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 58.6228121743661 < 174.303555520248. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 227.13441341262157 (T) = (0 -58.70821673510438) / Math.Sqrt((35.202623249111824 / (124)) + (1.1416122157483382 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.6870312298175405 = (187.58490408125297 - 58.70821673510438) / 187.58490408125297 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.StartsWith(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.66631840492913 < 26.36673596845211. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 583.2536939198679 (T) = (0 -20.537445799864106) / Math.Sqrt((0.0055563670812140455 / (124)) + (0.0031984723143495004 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.25896841623964567 = (27.714669995099435 - 20.537445799864106) / 27.714669995099435 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 49.97464709575156 < 158.70775299654147. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 231.96586532813552 (T) = (0 -49.169769154096805) / Math.Sqrt((24.741474307071105 / (126)) + (2.035122551300518 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.7080192960193448 = (168.40074869247 - 49.169769154096805) / 168.40074869247 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyFiveValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 87.89416433898627 < 207.60321712921055. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 229.02154240799584 (T) = (0 -88.40840002075447) / Math.Sqrt((35.73452155349674 / (125)) + (1.6842230136920084 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.6023823656170438 = (222.34526936399905 - 88.40840002075447) / 222.34526936399905 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyThreeValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 64.07588036464094 < 181.33129319365702. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 245.34252876247828 (T) = (0 -64.74735385358503) / Math.Sqrt((33.87857322176363 / (125)) + (0.2838199752107967 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.6674229835336901 = (194.68378946187323 - 64.74735385358503) / 194.68378946187323 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.314659879858002 < 37.243095892071366. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 36.52179686850732 (T) = (0 -23.503971967702316) / Math.Sqrt((25.644925969952126 / (125)) + (0.14550568611345632 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.415916072251844 = (40.24074426823932 - 23.503971967702316) / 40.24074426823932 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceCompareToDifferent(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.044934209673992 < 27.414380945914825. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1129.8267868180571 (T) = (0 -15.057811997780282) / Math.Sqrt((0.003831166326987733 / (124)) + (0.0035542566011106734 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.47836012666480837 = (28.866297933679125 - 15.057811997780282) / 28.866297933679125 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceCompareTo(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 16.085196622673877 < 54.277013439700895. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 581.6347267224091 (T) = (0 -16.080474130981763) / Math.Sqrt((0.01360546820880441 / (125)) + (0.14574626781771516 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.7182449448867892 = (57.07253104836235 - 16.080474130981763) / 57.07253104836235 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyFiveValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 50.91078234415115 < 64.2263227416193. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.39846572099167 (T) = (0 -51.0599498063834) / Math.Sqrt((33.60826785436725 / (125)) + (0.17735819169798728 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.28274648643929406 = (71.18814873823808 - 51.0599498063834) / 71.18814873823808 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.StartsWith(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 40.529600717596146 < 251.1722302774622. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1153.3573330521021 (T) = (0 -38.16078797328275) / Math.Sqrt((0.20946369484239766 / (122)) + (1.102033415948591 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.8556281137919048 = (264.32284688916826 - 38.16078797328275) / 264.32284688916826 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.Reverse(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.695956486620107 < 28.147828159576616. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 184.29115290668807 (T) = (0 -15.875177106020809) / Math.Sqrt((0.12875535074657926 / (124)) + (0.13553529144723103 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.46389770395661073 = (29.61221621915224 - 15.875177106020809) / 29.61221621915224 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 47.50755721205328 < 60.53619713194682. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.84543855930344 (T) = (0 -47.36599496865448) / Math.Sqrt((33.179655125076664 / (124)) + (0.07240687733737254 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.29880466295603225 = (67.55035646462727 - 47.36599496865448) / 67.55035646462727 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceEqual(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.46954536612844 < 56.32660451887785. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1963.3031171065995 (T) = (0 -17.446745196976742) / Math.Sqrt((0.031491271586288734 / (124)) + (0.006051926037927729 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.7060724589717249 = (59.357299883981966 - 17.446745196976742) / 59.357299883981966 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.LastIndexOfValue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 33.97134540377312 < 147.13030460052633. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 237.39900891472473 (T) = (0 -33.09123411169911) / Math.Sqrt((25.792542893362608 / (124)) + (1.7549127788017012 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.7873936754700844 = (155.6455772652374 - 33.09123411169911) / 155.6455772652374 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.SequenceCompareTo(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 43.643293557453376 < 458.13064666598666. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 3238.717611734459 (T) = (0 -43.640011722833165) / Math.Sqrt((2.0433399510761383 / (126)) + (0.06294798454042848 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.9094508107991104 = (481.948122428957 - 43.640011722833165) / 481.948122428957 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyTwoValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 28.006908802128073 < 42.81793405615691. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 39.163041315852404 (T) = (0 -28.286328187614107) / Math.Sqrt((33.20060050128396 / (124)) + (0.13673416376961597 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.4194425658893877 = (48.722704293585494 - 28.286328187614107) / 48.722704293585494 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.StartsWith(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.114296642415812 < 34.37607120652259. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 133.63914780071238 (T) = (0 -23.31187216669982) / Math.Sqrt((0.0585558836999621 / (125)) + (0.2644251634966379 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.3558034771579233 = (36.18751629371131 - 23.31187216669982) / 36.18751629371131 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfAnyThreeValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 32.174750169972555 < 45.979061223799086. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 37.6987410172719 (T) = (0 -32.259568283424784) / Math.Sqrt((33.168079413284374 / (124)) + (0.332114055964516 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.3814831144699626 = (52.15632594376139 - 32.259568283424784) / 52.15632594376139 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.IndexOfValue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.55833117996638 < 162.74685111380347. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 220.7347020039008 (T) = (0 -40.56185875171449) / Math.Sqrt((33.17733067986393 / (124)) + (3.1528093182676105 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.7686278108748645 = (175.3100011936914 - 40.56185875171449) / 175.3100011936914 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.EndsWith(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.37184785126956 < 47.671999952045624. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 536.3104370542255 (T) = (0 -23.2769969084325) / Math.Sqrt((0.018801307804866874 / (126)) + (0.07055676199437802 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.5353750448351582 = (50.09846468572524 - 23.2769969084325) / 50.09846468572524 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.LastIndexOfValue(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.52771901249122 < 30.397484758619694. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.84351686899946 (T) = (0 -14.172259648628291) / Math.Sqrt((25.724688834637924 / (124)) + (2.578939922447751 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.5711411613600906 = (33.046444125004975 - 14.172259648628291) / 33.046444125004975 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.Reverse(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 923.6296715608077 < 1.1900454034494172. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 425.5803364610732 (T) = (0 -920.298144596551) / Math.Sqrt((18.482649807868206 / (125)) + (14.407199066388365 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.26820065034081597 = (1257.5826215548773 - 920.298144596551) / 1257.5826215548773 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Byte>.EndsWith(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 41.86196094781606 < 251.53729278923905. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1626.3170446920678 (T) = (0 -42.27024031288779) / Math.Sqrt((0.4337738544685965 / (125)) + (0.45580333883491275 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.8401575535764901 = (264.4494078931378 - 42.27024031288779) / 264.4494078931378 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
43.23 ns 39.11 ns 0.90 0.01 True
38.03 ns 34.31 ns 0.90 0.01 True
1.53 μs 1.10 μs 0.72 0.01 True
592.07 ns 56.29 ns 0.10 0.01 True
388.93 ns 46.20 ns 0.12 0.01 True
12.60 μs 10.82 μs 0.86 0.28 False
158.84 ns 34.89 ns 0.22 0.14 True
55.22 ns 39.45 ns 0.71 0.03 True
592.96 ns 54.74 ns 0.09 0.01 True
1.88 μs 1.44 μs 0.77 0.01 True
292.82 ns 73.52 ns 0.25 0.13 False
593.05 ns 54.00 ns 0.09 0.03 True
1.54 μs 1.10 μs 0.71 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Collections.Tests.Perf_BitArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Collections.Tests.Perf_BitArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Collections.Tests.Perf_BitArray.BitArraySetLengthShrink(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.11491639971481 < 40.86534704551849. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 74.62267343686312 (T) = (0 -39.037928661712286) / Math.Sqrt((0.1043268586431077 / (125)) + (0.0549139240493688 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.08981874243653698 = (42.89027964189906 - 39.037928661712286) / 42.89027964189906 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayByteArrayCtor(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 34.310749013358034 < 36.32798245393511. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 50.95825170539406 (T) = (0 -34.51404665401146) / Math.Sqrt((0.10838922668831083 / (125)) + (0.1270369093442183 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.09539669062340675 = (38.15379216089403 - 34.51404665401146) / 38.15379216089403 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayByteArrayCtor(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.0990593553629469 < 1.45255636565399. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 363.23558674895594 (T) = (0 -1093.1396960968116) / Math.Sqrt((34.17043014298613 / (125)) + (34.950719518701504 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.28495863892411577 = (1528.7782715842104 - 1093.1396960968116) / 1528.7782715842104 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 56.293277789374024 < 562.2112501818938. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2604.1477305943436 (T) = (0 -55.683589801160515) / Math.Sqrt((1.6006770882191173 / (125)) + (0.8886964876969863 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.905954410722591 = (592.0914551017277 - 55.683589801160515) / 592.0914551017277 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 46.19530772032489 < 369.3294943004969. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1743.111745058478 (T) = (0 -46.92995426701451) / Math.Sqrt((1.5317619374548779 / (125)) + (0.7840603533960333 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.8791902775155798 = (388.46173388956066 - 46.92995426701451) / 388.46173388956066 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayCopyToBoolArray(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 10.821713351743412 < 11.960930786961907. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/19/2023 8:08:41 AM, 9/27/2023 11:11:59 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 3.653293697920082 (T) = (0 -8988.379498226492) / Math.Sqrt((1950159.6286760415 / (134)) + (1618610.5029422638 / (21))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (134) + (21) - 2, .975) and 0.10955437395846827 = (10094.24858223433 - 8988.379498226492) / 10094.24858223433 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 34.89499629833414 < 151.218930994084. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 62.28472582845581 (T) = (0 -38.129377995977286) / Math.Sqrt((10.649331110470058 / (124)) + (113.50312393843953 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.7626512817382881 = (160.6470777479996 - 38.129377995977286) / 160.6470777479996 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayLengthValueCtor(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.4457460820283 < 54.76975254287878. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 243.1222916191317 (T) = (0 -39.53083823193438) / Math.Sqrt((0.43213228216618244 / (124)) + (0.057170360499993494 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.31108230729396386 = (57.38107563569621 - 39.53083823193438) / 57.38107563569621 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 54.73621359454743 < 563.184051028567. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1952.9795427351899 (T) = (0 -55.51934402565577) / Math.Sqrt((2.3888250978406695 / (124)) + (1.6862454698818299 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.9062249849846412 = (592.0483618857603 - 55.51934402565577) / 592.0483618857603 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArraySetLengthGrow(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.4376144760861627 < 1.788323287557183. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 143.58116348313268 (T) = (0 -1456.5412555665118) / Math.Sqrt((154.9069970865525 / (124)) + (213.71125188701103 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.22193668038617737 = (1872.0086384350302 - 1456.5412555665118) / 1872.0086384350302 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 73.52100271156161 < 278.23818517034704. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 23.985437801144464 (T) = (0 -72.90146968134249) / Math.Sqrt((12723.396155442886 / (125)) + (8.478936249546539 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.7687322390381703 = (315.22538800111766 - 72.90146968134249) / 315.22538800111766 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 54.00331687453577 < 563.2187652023488. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1937.512266942175 (T) = (0 -55.431375328613036) / Math.Sqrt((3.3061943789281196 / (125)) + (1.5071864062310232 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.9063631867029815 = (591.9827189417829 - 55.431375328613036) / 591.9827189417829 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArraySetLengthShrink(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.0955573854203202 < 1.4555246189862823. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 474.73161721337124 (T) = (0 -1098.8578168478132) / Math.Sqrt((32.043680644184946 / (124)) + (17.398701240658923 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2834478776083278 = (1533.5350807141565 - 1098.8578168478132) / 1533.5350807141565 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
170.64 μs 141.12 μs 0.83 0.03 False
161.70 μs 125.02 μs 0.77 0.01 True
116.19 μs 102.02 μs 0.88 0.01 False
160.09 μs 125.62 μs 0.78 0.01 True
119.42 μs 99.43 μs 0.83 0.04 False
112.56 μs 100.21 μs 0.89 0.01 False
123.01 μs 102.59 μs 0.83 0.01 False
170.38 μs 141.51 μs 0.83 0.03 False

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromReader(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 141.1199529904762 < 148.08706602262677. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.1118140653811 (T) = (0 -141914.05400971283) / Math.Sqrt((41100430.239081256 / (125)) + (1328349.4071119174 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.12142798919040725 = (161528.08450947676 - 141914.05400971283) / 161528.08450947676 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromString(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 125.01542449999998 < 152.99228355642754. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 56.29902579573839 (T) = (0 -125826.85578083599) / Math.Sqrt((33132148.590014573 / (122)) + (946616.0435024648 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.1976529995291407 = (156823.4887236996 - 125826.85578083599) / 156823.4887236996 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromStream(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 102.0246503124151 < 110.28352254789219. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.19037134588246 (T) = (0 -102462.49285309015) / Math.Sqrt((29501256.334395416 / (124)) + (348788.901500687 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.13565018672785495 = (118542.85299744642 - 102462.49285309015) / 118542.85299744642 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 125.62345885103541 < 151.7434981021627. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 61.68494018767265 (T) = (0 -126250.53047227989) / Math.Sqrt((30430913.202827774 / (125)) + (300919.63502594264 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.19742482446298865 = (157306.79732001972 - 126250.53047227989) / 157306.79732001972 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromUtf8Bytes(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 99.43053077793493 < 107.61809321188535. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 29.972478038769886 (T) = (0 -99578.62543723677) / Math.Sqrt((35812517.00170677 / (125)) + (221784.2966083422 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.14028343719857247 = (115827.27348273373 - 99578.62543723677) / 115827.27348273373 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromUtf8Bytes(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 100.21050958583834 < 107.13856204553994. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 29.146316798075546 (T) = (0 -100054.86562922568) / Math.Sqrt((34517316.921957195 / (126)) + (370694.8903783684 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.13484979169675243 = (115650.28207697664 - 100054.86562922568) / 115650.28207697664 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromStream(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 102.58971921789679 < 116.90601143560632. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.55796397059121 (T) = (0 -102632.77640751656) / Math.Sqrt((32004365.60870776 / (124)) + (186524.94726597448 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.14391020041148242 = (119885.52656140436 - 102632.77640751656) / 119885.52656140436 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromReader(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 141.5074315295002 < 161.2070894428542. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 27.140633924669572 (T) = (0 -142208.21989672008) / Math.Sqrt((48176068.27556309 / (123)) + (759298.0350296426 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.10971920620002212 = (159734.12083813912 - 142208.21989672008) / 159734.12083813912 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in SIMD.ConsoleMandel

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.31 secs 2.31 secs 0.54 0.01 True
5.58 secs 3.56 secs 0.64 0.01 True
4.36 secs 2.69 secs 0.62 0.01 True
6.46 secs 4.82 secs 0.75 0.01 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'SIMD.ConsoleMandel*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'SIMD.ConsoleMandel*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'SIMD.ConsoleMandel*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'SIMD.ConsoleMandel*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### SIMD.ConsoleMandel.VectorFloatSinglethreadRaw #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.3108575244666665 < 4.0871866160902375. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2264.9942737332067 (T) = (0 -2315336283.9944377) / Math.Sqrt((18903493945446.86 / (125)) + (18553266530220.816 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.4618544379220332 = (4302434967.695581 - 2315336283.9944377) / 4302434967.695581 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### SIMD.ConsoleMandel.VectorFloatSinglethreadADT #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.5614843989333336 < 5.2944760974583325. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2577.396379795311 (T) = (0 -3565359290.96384) / Math.Sqrt((23617469971097.094 / (125)) + (12558862587055.316 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.36039958318865173 = (5574354233.129669 - 3565359290.96384) / 5574354233.129669 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### SIMD.ConsoleMandel.VectorDoubleSinglethreadRaw #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.687649808866667 < 4.142430041128206. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1973.7472741140114 (T) = (0 -2689412693.4192796) / Math.Sqrt((38024623304210.43 / (123)) + (12093582193780.52 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.3824780892077311 = (4355169665.103565 - 2689412693.4192796) / 4355169665.103565 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### SIMD.ConsoleMandel.VectorDoubleSinglethreadADT #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 4.815300892733333 < 6.155066625221667. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1239.2247847106667 (T) = (0 -4810663985.010044) / Math.Sqrt((49593102473738.734 / (124)) + (42316506090803.78 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2573956357090253 = (6478098185.705089 - 4810663985.010044) / 6478098185.705089 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Buffers.Binary.Tests.BinaryReadAndWriteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
80.51 ns 46.04 ns 0.57 0.08 True
196.34 ns 146.98 ns 0.75 0.01 True
52.05 ns 34.97 ns 0.67 0.01 True
97.80 ns 63.25 ns 0.65 0.01 True
39.17 ns 21.31 ns 0.54 0.01 True
216.88 ns 157.86 ns 0.73 0.01 True

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 46.03817413835943 < 76.57728753923111. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 112.61359148277691 (T) = (0 -47.03655712810265) / Math.Sqrt((0.050079842333748 / (125)) + (2.651545893334486 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.4163613201369726 = (80.59191200134565 - 47.03655712810265) / 80.59191200134565 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldLE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 146.98310553876374 < 186.4269110628671. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 746.5290461232017 (T) = (0 -147.2712313815678) / Math.Sqrt((0.12451988158396964 / (124)) + (0.09787630451280592 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.24874917276849728 = (196.03470111878522 - 147.2712313815678) / 196.03470111878522 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructAndReverseBE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 34.965471344917546 < 49.35847083010723. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 447.1838761519609 (T) = (0 -35.02595329489602) / Math.Sqrt((0.05190160753464617 / (125)) + (0.03048239292879328 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.325697870847629 = (51.944005187889985 - 35.02595329489602) / 51.944005187889985 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 63.24759180903355 < 92.91502521319707. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 192.92385050455272 (T) = (0 -62.31686694984043) / Math.Sqrt((0.49535008243526923 / (124)) + (0.9099971972095896 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.3645120409596677 = (98.06144406567016 - 62.31686694984043) / 98.06144406567016 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructAndReverseLE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.305397201151557 < 37.19972669272044. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 720.8668471873531 (T) = (0 -21.28302967694446) / Math.Sqrt((0.04141633903622589 / (124)) + (0.00853550438855756 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.45721824802262373 = (39.211026530294184 - 21.28302967694446) / 39.211026530294184 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldBE #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 157.86145448798132 < 205.79363580199626. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 797.373699254119 (T) = (0 -157.96835505818854) / Math.Sqrt((0.34078642853599467 / (125)) + (0.08048260641489456 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2707318512932748 = (216.6121684298534 - 157.96835505818854) / 216.6121684298534 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Serialization.Tests.WriteJson<Hashtable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
86.01 μs 73.49 μs 0.85 0.01 False
85.18 μs 73.82 μs 0.87 0.02 False
85.40 μs 74.03 μs 0.87 0.01 False
53.23 μs 47.39 μs 0.89 0.02 True
53.78 μs 47.33 μs 0.88 0.01 True
52.14 μs 46.73 μs 0.90 0.01 True
86.20 μs 73.94 μs 0.86 0.01 False
52.70 μs 46.43 μs 0.88 0.01 True
52.71 μs 47.40 μs 0.90 0.01 True
53.29 μs 47.48 μs 0.89 0.01 True

graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<Hashtable>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<Hashtable>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<Hashtable>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<Hashtable>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeObjectProperty(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 73.49195518867926 < 81.85743133050818. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.06067402858116 (T) = (0 -73854.94285174426) / Math.Sqrt((15389872.758199412 / (125)) + (437113.2756183077 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.14612320100829618 = (86493.67559694267 - 73854.94285174426) / 86493.67559694267 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeToString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 73.81893499410378 < 81.27093510239276. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.46696456635453 (T) = (0 -73486.35352002968) / Math.Sqrt((15703047.33563844 / (124)) + (195550.86436469178 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.1461382807783575 = (86063.5298031839 - 73486.35352002968) / 86063.5298031839 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeToString(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 74.0298172071541 < 81.12473732338424. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.792995491961946 (T) = (0 -73651.84123364057) / Math.Sqrt((15580578.314329652 / (125)) + (197872.49378960463 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.14252703533273528 = (85894.06811469626 - 73651.84123364057) / 85894.06811469626 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeToUtf8Bytes(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 47.39304551233635 < 50.54199008892362. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 61.19070310431542 (T) = (0 -47531.16847074706) / Math.Sqrt((804897.7655679174 / (125)) + (61918.35386613007 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.10611525403143088 = (53173.71024073652 - 47531.16847074706) / 53173.71024073652 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeToStream(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 47.32731621797655 < 50.74699774868458. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 92.17843141366114 (T) = (0 -47245.860169038875) / Math.Sqrt((203005.46712702734 / (125)) + (61911.03195482137 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.10592968016773335 = (52843.56176581558 - 47245.860169038875) / 52843.56176581558 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeToWriter(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 46.73471736111111 < 49.92118952655729. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 114.57301679188738 (T) = (0 -46682.11967481542) / Math.Sqrt((202884.5195140816 / (124)) + (19742.00107713528 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.10519148212468389 = (52169.954512346485 - 46682.11967481542) / 52169.954512346485 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeObjectProperty(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 73.93863504815253 < 81.56836089221015. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.29347800952055 (T) = (0 -74048.45789993192) / Math.Sqrt((16186467.23139577 / (122)) + (162077.8466329769 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.14315556803025836 = (86419.95575522028 - 74048.45789993192) / 86419.95575522028 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeToWriter(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 46.42666070772059 < 50.381062699920626. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 94.91895684423125 (T) = (0 -46487.14165089883) / Math.Sqrt((198786.41089323242 / (124)) + (63164.427226165215 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.11059251210443588 = (52267.5402260133 - 46487.14165089883) / 52267.5402260133 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeToStream(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 47.39800637370912 < 50.289102512105245. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 91.39354329980684 (T) = (0 -47449.801824721435) / Math.Sqrt((195655.12290385 / (124)) + (55993.92341159377 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.10155994763725179 = (52813.5424282748 - 47449.801824721435) / 52813.5424282748 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeToUtf8Bytes(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 47.481979076204944 < 50.71364441856489. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 95.03332244448404 (T) = (0 -47489.320668300774) / Math.Sqrt((255473.07495887487 / (126)) + (40552.864051382734 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.10420824367119484 = (53013.79515136949 - 47489.320668300774) / 53013.79515136949 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Globalization.Tests.StringHash

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.70 ms 1.59 ms 0.93 0.01 True
513.22 μs 207.25 μs 0.40 0.01 True
509.63 ns 221.75 ns 0.44 0.01 True
1.88 μs 1.75 μs 0.93 0.01 True
1.89 μs 1.76 μs 0.93 0.01 True
373.80 ns 149.57 ns 0.40 0.01 True
1.70 ms 1.59 ms 0.93 0.01 True
2.32 μs 2.13 μs 0.92 0.01 True
1.91 ms 1.76 ms 0.92 0.01 True
361.04 μs 136.14 μs 0.38 0.01 True
2.31 μs 2.13 μs 0.92 0.01 True
1.89 ms 1.75 ms 0.92 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringHash*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Globalization.Tests.StringHash*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringHash*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Globalization.Tests.StringHash*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (en-US, IgnoreCase)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.5891680196428566 < 1.6134555375706845. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 81.98329543203016 (T) = (0 -1575816.2491452233) / Math.Sqrt((51476783.82140693 / (125)) + (54398761.24914764 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.07201702678889095 = (1698109.0113026644 - 1575816.2491452233) / 1698109.0113026644 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (en-US, OrdinalIgnoreCase)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 207.25385213963963 < 486.8934465401786. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1965.951212764263 (T) = (0 -205670.56983530035) / Math.Sqrt((581297.6246093919 / (125)) + (590617.6713007569 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5985891795326738 = (512369.2719490142 - 205670.56983530035) / 512369.2719490142 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (en-US, OrdinalIgnoreCase)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 221.750116565969 < 484.20443609014427. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 513.3016490906131 (T) = (0 -219.23545267867584) / Math.Sqrt((13.634064904472055 / (124)) + (6.55774246348679 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.5730178031604599 = (513.4533811981499 - 219.23545267867584) / 513.4533811981499 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (, IgnoreCase)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.7546855360815108 < 1.7932909507094401. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 53.02260438710226 (T) = (0 -1761.4244286853266) / Math.Sqrt((180.29942635146003 / (124)) + (115.58583829067331 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.06484837135500185 = (1883.5709362315592 - 1761.4244286853266) / 1883.5709362315592 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (en-US, IgnoreCase)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.7624404165191645 < 1.7929253704995762. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 67.43535491795498 (T) = (0 -1757.034540681119) / Math.Sqrt((209.66019377329587 / (125)) + (54.86614615828746 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.06704746105738926 = (1883.3053851512168 - 1757.034540681119) / 1883.3053851512168 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (en-US, Ordinal)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 149.56698351078848 < 354.9978120940606. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1542.19387915721 (T) = (0 -149.5266878488038) / Math.Sqrt((1.2379781390985827 / (124)) + (0.33163610105988595 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.5993552597071435 = (373.21515250519775 - 149.5266878488038) / 373.21515250519775 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (, IgnoreCase)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.5857198370535712 < 1.613185532755208. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 87.03374624858688 (T) = (0 -1578349.4026669725) / Math.Sqrt((52940199.2426076 / (125)) + (44038673.168266855 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.0704917587740353 = (1698047.7769462548 - 1578349.4026669725) / 1698047.7769462548 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (, None)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.1262209594184487 < 2.1980342218831095. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 75.31513444373853 (T) = (0 -2128.114567133765) / Math.Sqrt((301.12506916585016 / (126)) + (79.72407719261139 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.07365335379404646 = (2297.3198811156744 - 2128.114567133765) / 2297.3198811156744 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (, None)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.7552134479166663 < 1.7976833743501983. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 101.09363442326449 (T) = (0 -1760003.6111806408) / Math.Sqrt((55367484.80577092 / (123)) + (33822571.051200934 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.0672899223236824 = (1886978.2296824527 - 1760003.6111806408) / 1886978.2296824527 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (en-US, Ordinal)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 136.14359514751553 < 343.07129850941203. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2773.330955400322 (T) = (0 -135800.49761109467) / Math.Sqrt((499917.61178605777 / (124)) + (75926.14969133222 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.6232634202151779 = (360465.38854458695 - 135800.49761109467) / 360465.38854458695 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (en-US, None)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.1287336956275227 < 2.191147945489441. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 89.2663651356319 (T) = (0 -2126.4633981754455) / Math.Sqrt((300.5413509122382 / (125)) + (38.750847598487255 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.07467762473749348 = (2298.0784373361607 - 2126.4633981754455) / 2298.0784373361607 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (en-US, None)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.751404554563492 < 1.7986457857514881. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 99.86057541596544 (T) = (0 -1759795.494382801) / Math.Sqrt((37890467.895511314 / (124)) + (39135161.59233957 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.06716738777055176 = (1886507.2589785757 - 1759795.494382801) / 1886507.2589785757 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Document.Tests.Perf_EnumerateArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
9.96 μs 7.44 μs 0.75 0.01 True
3.05 ms 2.65 ms 0.87 0.01 True
10.10 μs 7.65 μs 0.76 0.01 True
9.68 μs 7.72 μs 0.80 0.01 True
1.03 ms 665.68 μs 0.65 0.01 True
9.74 μs 7.70 μs 0.79 0.01 True
29.96 μs 28.12 μs 0.94 0.01 True
9.95 μs 7.46 μs 0.75 0.01 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.44467237327463 < 9.471090830670898. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 559.7344755986244 (T) = (0 -7447.307295576359) / Math.Sqrt((538.6936137621622 / (125)) + (469.2734875929355 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2513395683120964 = (9947.510220068558 - 7447.307295576359) / 9947.510220068558 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.Parse(TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.6491802701388885 < 2.8906059934895834. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 141.63650712129473 (T) = (0 -2649587.6064057415) / Math.Sqrt((663179501.5776244 / (125)) + (54043123.77650563 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.12473184440048518 = (3027172.403628585 - 2649587.6064057415) / 3027172.403628585 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.648431501909639 < 9.601220359622099. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 592.817121943387 (T) = (0 -7642.304855727428) / Math.Sqrt((450.45488757743385 / (125)) + (412.99252907195404 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2443097579874137 = (10113.012489580542 - 7642.304855727428) / 10113.012489580542 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.723028259524988 < 9.186453812634285. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 224.48984127379194 (T) = (0 -7698.5464027763865) / Math.Sqrt((914.1912074053182 / (124)) + (2080.3670844844296 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.20344707796730224 = (9664.82726989528 - 7698.5464027763865) / 9664.82726989528 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 665.6785630580356 < 0.9817097457520604. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1056.2408186966081 (T) = (0 -665951.3419525748) / Math.Sqrt((3144212.851783995 / (125)) + (2871751.5032141157 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.3554360352800799 = (1033181.1556389878 - 665951.3419525748) / 1033181.1556389878 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.700684491427629 < 9.197511258196132. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 382.87078726083723 (T) = (0 -7695.089630331453) / Math.Sqrt((1113.646108732229 / (125)) + (526.8290792495268 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2038129452180342 = (9664.926833605374 - 7695.089630331453) / 9664.926833605374 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.Parse(TestCase: ArrayOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 28.122235427907675 < 28.462867483161894. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.829485997043214 (T) = (0 -28139.956747280816) / Math.Sqrt((234332.57607960378 / (124)) + (19135.779128142098 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.06487331861324316 = (30092.133298507048 - 28139.956747280816) / 30092.133298507048 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.456629732358098 < 9.462847918562858. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 428.5270216498123 (T) = (0 -7445.906537510201) / Math.Sqrt((1935.4532456550025 / (125)) + (559.6189598232864 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2516434511578196 = (9949.677796005304 - 7445.906537510201) / 9949.677796005304 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Node.Tests.Perf_ParseThenWrite

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
12.46 ms 11.46 ms 0.92 0.01 True
154.80 μs 139.95 μs 0.90 0.01 True
125.87 μs 117.47 μs 0.93 0.01 True
22.58 μs 20.77 μs 0.92 0.01 True
165.79 μs 149.77 μs 0.90 0.01 True
15.00 μs 13.72 μs 0.91 0.01 True
13.01 ms 12.10 ms 0.93 0.01 True
120.70 μs 104.37 μs 0.86 0.01 True
34.91 μs 31.69 μs 0.91 0.01 True
24.20 μs 22.25 μs 0.92 0.01 True
121.85 μs 110.35 μs 0.91 0.01 True
33.43 μs 30.14 μs 0.90 0.01 True
15.70 μs 14.40 μs 0.92 0.01 True
110.74 μs 94.21 μs 0.85 0.01 True
1.73 μs 1.63 μs 0.94 0.01 True
1.68 μs 1.57 μs 0.94 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Node.Tests.Perf_ParseThenWrite*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Node.Tests.Perf_ParseThenWrite*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Node.Tests.Perf_ParseThenWrite*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Node.Tests.Perf_ParseThenWrite*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 11.456822842424241 < 11.819032567666666. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 77.75547988064801 (T) = (0 -11547614.220858363) / Math.Sqrt((4495158417.7344 / (125)) + (2960064256.8553867 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.07246682541719646 = (12449812.618349075 - 11547614.220858363) / 12449812.618349075 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: BroadTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 139.9462005589715 < 147.69773500747024. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 143.35750104443025 (T) = (0 -139984.83354768637) / Math.Sqrt((384416.6911221801 / (124)) + (219107.31129283024 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.09457691689358466 = (154607.09601903733 - 139984.83354768637) / 154607.09601903733 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json4KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 117.46902504672896 < 119.5594015329736. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 68.40202790252758 (T) = (0 -117018.97686979269) / Math.Sqrt((362017.6427985144 / (125)) + (376175.1670741897 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.06770543878552937 = (125517.17208063058 - 117018.97686979269) / 125517.17208063058 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.77310912928323 < 21.4473808696965. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 69.53544700990861 (T) = (0 -20810.165192089982) / Math.Sqrt((8335.72173504201 / (124)) + (15726.097643680305 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.07515376822019067 = (22501.216393607516 - 20810.165192089982) / 22501.216393607516 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: BroadTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 149.77414217238726 < 157.38617999696484. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 105.61270110323659 (T) = (0 -150072.36225321968) / Math.Sqrt((613764.6120979077 / (125)) + (394794.04562637047 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.0864246992344936 = (164269.28587875623 - 150072.36225321968) / 164269.28587875623 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.716019170841824 < 14.254154049171614. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.78488216236299 (T) = (0 -13732.800406496273) / Math.Sqrt((10462.365903228489 / (125)) + (6705.949234185667 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.06868824507925007 = (14745.653465594738 - 13732.800406496273) / 14745.653465594738 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 12.095365734126984 < 12.352786644642856. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 80.44651745797582 (T) = (0 -12111217.798642525) / Math.Sqrt((7118604744.482641 / (125)) + (2146938878.8058174 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.07002678600514564 = (13023189.93319902 - 12111217.798642525) / 13023189.93319902 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: DeepTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 104.37108320864508 < 114.64214178965973. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 166.69273693732487 (T) = (0 -104349.4442026375) / Math.Sqrt((307918.54949156876 / (125)) + (183376.51123656123 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1288693542442311 = (119786.2165807596 - 104349.4442026375) / 119786.2165807596 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 31.68779941613588 < 33.1871313280777. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 93.35229503234129 (T) = (0 -31856.41036969051) / Math.Sqrt((35093.83973902496 / (125)) + (20067.254705223328 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.08282549287720598 = (34733.20521045128 - 31856.41036969051) / 34733.20521045128 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfStrings) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 22.254738008039304 < 22.99361690475507. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 66.05607230747219 (T) = (0 -22187.768768907838) / Math.Sqrt((31310.64300631215 / (124)) + (9809.325213035663 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.06687433624453906 = (23777.900052186822 - 22187.768768907838) / 23777.900052186822 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json4KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 110.34504116148752 < 115.49889191857909. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 101.7781043192097 (T) = (0 -110838.58466557287) / Math.Sqrt((407552.7496072003 / (122)) + (139790.34907500187 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.07589887397077594 = (119942.05130106905 - 110838.58466557287) / 119942.05130106905 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfNumbers) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 30.14398223776786 < 31.779498395277. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 101.21056205720906 (T) = (0 -30133.369058791744) / Math.Sqrt((30540.454743442708 / (125)) + (18506.818657631833 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.08972382290630478 = (33103.5457337802 - 30133.369058791744) / 33103.5457337802 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.396982318890421 < 14.81052549102108. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 60.486736992116086 (T) = (0 -14455.399626155786) / Math.Sqrt((10632.852253760182 / (124)) + (5426.684052043777 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.06395686448255955 = (15443.091325235675 - 14455.399626155786) / 15443.091325235675 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: DeepTree) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 94.21478636045447 < 105.25263938339657. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 292.7977628310387 (T) = (0 -94519.54195448037) / Math.Sqrt((115637.33582196225 / (124)) + (59050.90256246062 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.14298843065475966 = (110289.6919194377 - 94519.54195448037) / 110289.6919194377 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.6254213585593802 < 1.6401796056896827. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 46.13841548089193 (T) = (0 -1615.9142355063875) / Math.Sqrt((100.57584233934757 / (125)) + (121.07598347157573 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.05910566613777618 = (1717.4237077965097 - 1615.9142355063875) / 1717.4237077965097 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.5713634649314185 < 1.5995897042866603. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 66.41698808763142 (T) = (0 -1558.8240025651428) / Math.Sqrt((80.49568380935546 / (126)) + (70.96853558183224 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.06877356012141321 = (1673.9473191594325 - 1558.8240025651428) / 1673.9473191594325 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
72.35 ns 26.58 ns 0.37 0.01 True
51.47 ns 21.35 ns 0.41 0.01 True
50.59 ns 20.80 ns 0.41 0.01 True
129.13 ns 66.92 ns 0.52 0.01 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf<Single>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf<Single>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf<Single>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf<Single>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_VectorOf<Single>.AbsBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 26.580638756947163 < 68.49740221903986. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2263.8037065841963 (T) = (0 -26.425874567442694) / Math.Sqrt((0.01862518772526298 / (124)) + (0.007756094208772915 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.6339597036606492 = (72.19389458406414 - 26.425874567442694) / 72.19389458406414 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorOf<Single>.LessThanOrEqualBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.352340737316382 < 48.83393340204397. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1029.420642209991 (T) = (0 -21.42285071119723) / Math.Sqrt((0.041736280817089084 / (125)) + (0.01526678690715288 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5824630869810128 = (51.307680933644875 - 21.42285071119723) / 51.307680933644875 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorOf<Single>.LessThanBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.801884143708268 < 47.91806733352499. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1056.6639792480173 (T) = (0 -20.85304224239253) / Math.Sqrt((0.03864268542047142 / (125)) + (0.014237067662089395 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5865294751583574 = (50.43416879686684 - 20.85304224239253) / 50.43416879686684 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_VectorOf<Single>.EqualsBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 66.92194128459685 < 122.22905161140056. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1518.3537200699732 (T) = (0 -66.83469719406393) / Math.Sqrt((0.08972072557813836 / (125)) + (0.028500993351710958 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.48126735087207007 = (128.84227994213094 - 66.83469719406393) / 128.84227994213094 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
693.61 ns 554.84 ns 0.80 0.03 True
18.38 μs 14.09 μs 0.77 0.01 True
15.26 μs 11.09 μs 0.73 0.01 True
282.01 ns 234.64 ns 0.83 0.01 True
432.00 ns 325.48 ns 0.75 0.03 True
453.33 ns 306.96 ns 0.68 0.01 True
441.00 ns 409.49 ns 0.93 0.01 False
282.01 ns 236.61 ns 0.84 0.03 True
601.26 ns 537.37 ns 0.89 0.03 False
16.76 μs 12.35 μs 0.74 0.01 True
446.88 μs 188.30 μs 0.42 0.03 False
1.79 μs 281.97 ns 0.16 0.08 False
15.25 μs 11.01 μs 0.72 0.01 True
479.07 μs 221.06 μs 0.46 0.02 False
478.63 μs 221.36 μs 0.46 0.02 False
429.85 ns 322.49 ns 0.75 0.01 True
9.70 μs 3.31 μs 0.34 0.07 True
13.65 μs 9.44 μs 0.69 0.02 True
445.76 μs 188.44 μs 0.42 0.03 False
9.62 μs 3.01 μs 0.31 0.02 True
468.18 ns 349.73 ns 0.75 0.02 True
444.59 ns 409.23 ns 0.92 0.04 False
1.90 μs 321.57 ns 0.17 0.03 False
443.07 ns 408.43 ns 0.92 0.01 False
439.19 ns 274.81 ns 0.63 0.01 True
444.48 ns 407.68 ns 0.92 0.03 False
1.33 μs 264.05 ns 0.20 0.03 False
2.04 μs 480.57 ns 0.24 0.02 False
572.28 ns 431.91 ns 0.75 0.02 True
2.04 μs 478.69 ns 0.24 0.02 False
407.94 ns 251.11 ns 0.62 0.03 True
1.33 μs 260.04 ns 0.20 0.03 False
18.34 μs 14.15 μs 0.77 0.01 True
9.25 μs 2.57 μs 0.28 0.02 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 554.8370324494401 < 656.9853615245956. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 54.005993441171185 (T) = (0 -557.55738116247) / Math.Sqrt((1002.5165906045032 / (124)) + (15.525733694564307 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2212402257333344 = (715.955548278678 - 557.55738116247) / 715.955548278678 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.086188106705494 < 17.429934679913213. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 61.01097885319303 (T) = (0 -14092.494463687099) / Math.Sqrt((753612.2107969661 / (124)) + (9716.749270511586 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.25726988717019456 = (18973.91020002491 - 14092.494463687099) / 18973.91020002491 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 11.090307528980368 < 14.520857349377925. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 66.41321863855933 (T) = (0 -11107.228795552497) / Math.Sqrt((607443.8726200536 / (125)) + (4300.117549537705 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2972202763354874 = (15804.708675480766 - 11107.228795552497) / 15804.708675480766 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 234.63962765285814 < 268.41713453917106. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 85.0356557578698 (T) = (0 -236.32559012492055) / Math.Sqrt((32.83010909414832 / (125)) + (2.9302779536047905 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.17762445553408815 = (287.3694283776412 - 236.32559012492055) / 287.3694283776412 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 325.47907523606085 < 409.8538537064669. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 61.00398542742869 (T) = (0 -325.59623391912686) / Math.Sqrt((123.04238737282955 / (123)) + (78.34022839971354 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.26257423530672785 = (441.5308625059455 - 325.59623391912686) / 441.5308625059455 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 306.95711940176756 < 430.2423011669487. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 80.59445641134292 (T) = (0 -308.4764014111313) / Math.Sqrt((475.4081180551993 / (124)) + (5.053107652840445 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.34326638181427843 = (469.7131270107988 - 308.4764014111313) / 469.7131270107988 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 409.49125907464884 < 420.41689040231745. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.80524526829564 (T) = (0 -409.9912955758111) / Math.Sqrt((213.87781381942034 / (124)) + (10.468091809039148 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.10332015366595809 = (457.2326424553945 - 409.9912955758111) / 457.2326424553945 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 236.61328513279787 < 267.80676222131035. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 78.06996815730075 (T) = (0 -237.39593157549203) / Math.Sqrt((32.89066260105462 / (126)) + (4.594165412868924 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.17467412567506996 = (287.6390271535695 - 237.39593157549203) / 287.6390271535695 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 537.3708034503368 < 572.6266339809596. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 42.07100972577936 (T) = (0 -537.211801199542) / Math.Sqrt((301.96583688229913 / (125)) + (4.729918567929487 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1116079003301007 = (604.7012365363834 - 537.211801199542) / 604.7012365363834 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 12.349719218086294 < 15.770106742867759. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 62.04158521718958 (T) = (0 -12411.407008337945) / Math.Sqrt((718410.7411113469 / (125)) + (8970.598896233263 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2798985325921484 = (17235.63632360766 - 12411.407008337945) / 17235.63632360766 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 188.29682317073173 < 424.4852940892856. IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.63327650389609 (T) = (0 -201590.84166132958) / Math.Sqrt((14153336986.379154 / (125)) + (727050562.9552752 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.24677701781770878 = (267637.6669724897 - 201590.84166132958) / 267637.6669724897 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 281.9724157014041 < 1.7069839005047214. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.826291839002868 (T) = (0 -344.1140058957783) / Math.Sqrt((495097.73792886606 / (124)) + (18445.288246894524 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.5347278221299847 = (739.5972126919538 - 344.1140058957783) / 739.5972126919538 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 11.011682155797102 < 14.497767038774924. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 62.5320441992453 (T) = (0 -11101.249602681813) / Math.Sqrt((664869.8596748587 / (124)) + (9238.566151893012 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2978245509069572 = (15809.79457060286 - 11101.249602681813) / 15809.79457060286 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 221.05668801020408 < 455.13252806672494. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.671092445878411 (T) = (0 -233973.15201828227) / Math.Sqrt((14191997874.256435 / (124)) + (687544985.7478541 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.22123454935729386 = (300441.1043468697 - 233973.15201828227) / 300441.1043468697 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 221.35994938380279 < 454.9978822702688. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.635358410882115 (T) = (0 -234207.97074627862) / Math.Sqrt((14183293018.781742 / (124)) + (705549894.4417014 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.22030577792140024 = (300384.38674317696 - 234207.97074627862) / 300384.38674317696 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 322.4886702134759 < 409.7245480469623. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 114.4036119121769 (T) = (0 -323.8325659987632) / Math.Sqrt((122.48445712649138 / (124)) + (2.3103761994304497 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2671550472832565 = (441.88414588689915 - 323.8325659987632) / 441.88414588689915 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.306631848118481 < 9.18766578947861. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 319.1470397805935 (T) = (0 -3241.5396558424413) / Math.Sqrt((18286.94790970099 / (124)) + (8141.344346163716 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.6683227556918736 = (9773.174709661627 - 3241.5396558424413) / 9773.174709661627 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.438599119841745 < 12.867674095645436. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 63.447021998378844 (T) = (0 -9427.535664460784) / Math.Sqrt((628954.2148555202 / (125)) + (9366.282034085587 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.32974589157860096 = (14065.614139486255 - 9427.535664460784) / 14065.614139486255 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 188.4412419425087 < 424.8457727289377. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.657676319877761 (T) = (0 -201758.34921184272) / Math.Sqrt((14258281171.671844 / (124)) + (706673673.5548121 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2481560655853793 = (268351.36918266164 - 201758.34921184272) / 268351.36918266164 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.0085449496425136 < 9.164493785611745. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 365.9919112205867 (T) = (0 -3044.8284044291486) / Math.Sqrt((19565.20742947395 / (125)) + (5345.87598376172 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.6874135256111454 = (9740.75545137443 - 3044.8284044291486) / 9740.75545137443 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 349.73288205182854 < 444.9360865039902. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 94.34892061319705 (T) = (0 -351.24785317101157) / Math.Sqrt((143.048435336144 / (124)) + (11.282283438179997 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.24937303543789655 = (467.93929575381105 - 351.24785317101157) / 467.93929575381105 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 409.230899573041 < 421.4601335806504. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 35.2489482904708 (T) = (0 -408.12306188072216) / Math.Sqrt((226.88041553639877 / (124)) + (2.6424205040978452 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.10682851020591166 = (456.9369561659551 - 408.12306188072216) / 456.9369561659551 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 321.5727068868679 < 1.8108618195358654. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.993635098868484 (T) = (0 -386.81150641255846) / Math.Sqrt((544839.5042163817 / (125)) + (19974.628347164522 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.52343403657863 = (811.6641474677609 - 386.81150641255846) / 811.6641474677609 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsNotMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 408.42804986097275 < 420.7293995433233. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 35.635119331676385 (T) = (0 -409.07206212965866) / Math.Sqrt((210.06739871580717 / (125)) + (3.8956701840176007 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.10491321522298003 = (457.0194411166118 - 409.07206212965866) / 457.0194411166118 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 274.8092852949424 < 416.59911589243865. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 283.95564203716526 (T) = (0 -273.19917426476667) / Math.Sqrt((18.365493181488667 / (124)) + (5.689330576896524 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.3765769271725476 = (438.224355453686 - 273.19917426476667) / 438.224355453686 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsNotMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 407.67632369067314 < 421.8020124717387. IsChangePoint: Marked as a change because one of 9/8/2023 12:12:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.13069887316991 (T) = (0 -408.4228571202288) / Math.Sqrt((238.69234910062423 / (124)) + (5.797581855007821 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.10559369172298412 = (456.6412975183667 - 408.4228571202288) / 456.6412975183667 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 264.04770374349954 < 1.2612597224337674. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.763820347830381 (T) = (0 -311.51234526535234) / Math.Sqrt((245339.2596886118 / (124)) + (9841.999660193236 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.47051635341896364 = (588.3323257986137 - 311.51234526535234) / 588.3323257986137 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsNotMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 480.56754141665937 < 1.9376703534087316. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.9436691977735 (T) = (0 -549.0983474286018) / Math.Sqrt((526507.7775948469 / (125)) + (20336.82797524246 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.4308260786598819 = (964.7285774017043 - 549.0983474286018) / 964.7285774017043 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 431.91414599715205 < 542.8766928031703. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 56.018855177101 (T) = (0 -433.1571754365694) / Math.Sqrt((1010.6943225986133 / (125)) + (15.907049189846397 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2751568350248023 = (597.5874456254146 - 433.1571754365694) / 597.5874456254146 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsNotMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 478.69218769263955 < 1.933060746764334. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.968801948440196 (T) = (0 -548.3643317213399) / Math.Sqrt((527122.3028278331 / (125)) + (19037.878757432725 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.43122267639332656 = (964.1107494302116 - 548.3643317213399) / 964.1107494302116 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 251.11373608189888 < 388.4606689276233. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 79.76518146964393 (T) = (0 -252.3168732733154) / Math.Sqrt((468.29599580629247 / (125)) + (30.19882245485601 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.408006842498922 = (426.21586090352054 - 252.3168732733154) / 426.21586090352054 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsMatch(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 260.0403266625719 < 1.2611431126464956. IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 5.938121621817074 (T) = (0 -309.8795326976418) / Math.Sqrt((246892.27177940932 / (125)) + (9887.168240736408 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.4791531935371804 = (594.9533122840841 - 309.8795326976418) / 594.9533122840841 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 14.15054096722172 < 17.39005181309334. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 63.06942103162435 (T) = (0 -14045.90807940864) / Math.Sqrt((691580.7507211647 / (125)) + (12261.843350527037 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2571179237251215 = (18907.318574491255 - 14045.90807940864) / 18907.318574491255 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.5685149726674377 < 8.824271606473879. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 346.8117926348611 (T) = (0 -2607.1233672187223) / Math.Sqrt((19143.06162075699 / (124)) + (6681.884193828843 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.7209234117046068 = (9341.963735271018 - 2607.1233672187223) / 9341.963735271018 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
984.09 μs 837.31 μs 0.85 0.01 True
772.91 μs 694.89 μs 0.90 0.01 False
736.25 μs 685.50 μs 0.93 0.01 True
1.02 ms 841.15 μs 0.82 0.01 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.DeserializeFromString(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 837.309223363593 < 936.7231898978661. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 116.04077728645258 (T) = (0 -843099.3714371803) / Math.Sqrt((116048901.0317605 / (124)) + (15607031.309651855 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.14242928307687194 = (983125.1869958091 - 843099.3714371803) / 983125.1869958091 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.DeserializeFromUtf8Bytes(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 694.8879939058173 < 733.6666352730294. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 22.554580614043914 (T) = (0 -697953.2822513968) / Math.Sqrt((141470126.48637855 / (124)) + (64010235.11186838 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.055246354829022626 = (738767.4933237059 - 697953.2822513968) / 738767.4933237059 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.DeserializeFromUtf8Bytes(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 685.4985155677655 < 702.2919513034386. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 31.68187955319438 (T) = (0 -692503.245816001) / Math.Sqrt((105138454.86659381 / (125)) + (34549831.34477487 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.06066498330546334 = (737227.116532798 - 692503.245816001) / 737227.116532798 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.DeserializeFromString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 841.1524725225227 < 943.8741901398886. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 94.46010367891172 (T) = (0 -848179.4737202313) / Math.Sqrt((127195978.9302781 / (124)) + (33107719.010071915 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.1397938937318116 = (986018.8942390424 - 848179.4737202313) / 986018.8942390424 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Serialization.Tests.WriteJson<BinaryData>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
9.20 μs 7.46 μs 0.81 0.03 False
8.77 μs 7.11 μs 0.81 0.03 False
2.64 μs 2.33 μs 0.88 0.01 True
2.71 μs 2.46 μs 0.91 0.01 True
9.15 μs 7.42 μs 0.81 0.03 False
2.80 μs 2.52 μs 0.90 0.01 True
1.92 μs 1.64 μs 0.86 0.01 True
2.01 μs 1.73 μs 0.86 0.01 True
8.67 μs 7.00 μs 0.81 0.03 False
2.63 μs 2.34 μs 0.89 0.01 True

graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<BinaryData>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<BinaryData>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<BinaryData>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<BinaryData>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeObjectProperty(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.455978238148902 < 8.723190875713346. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 22.351029266335757 (T) = (0 -7421.110736123493) / Math.Sqrt((837346.812472396 / (122)) + (16293.345993794796 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.20584624433163493 = (9344.677505022735 - 7421.110736123493) / 9344.677505022735 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.110975969061745 < 8.348563525675427. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 23.163242685168207 (T) = (0 -7097.918055677936) / Math.Sqrt((827229.9513849368 / (124)) + (652.8950609623226 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.21072123906534315 = (8992.916580287356 - 7097.918055677936) / 8992.916580287356 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToUtf8Bytes(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.3299402450974758 < 2.4775997958841582. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 62.73453858232696 (T) = (0 -2324.189803516809) / Math.Sqrt((427.7704356494315 / (126)) + (515.2610248996824 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.10906875778851592 = (2608.719610895749 - 2324.189803516809) / 2608.719610895749 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToUtf8Bytes(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.4599284667133303 < 2.575731595163559. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 69.01183835993712 (T) = (0 -2431.8552379794096) / Math.Sqrt((352.1105735540241 / (125)) + (380.2610937419662 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.10047466916913002 = (2703.4872222366025 - 2431.8552379794096) / 2703.4872222366025 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeObjectProperty(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.420879041212264 < 8.696067287317117. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 23.317780716536973 (T) = (0 -7460.251721029422) / Math.Sqrt((827673.1569857149 / (125)) + (1800.0477680254462 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.20349567358617096 = (9366.241304197762 - 7460.251721029422) / 9366.241304197762 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToStream(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.5216809139019896 < 2.6650111009001836. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 61.92014862712943 (T) = (0 -2538.967820972723) / Math.Sqrt((289.0687760607798 / (125)) + (414.53868484511787 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.08921065635697167 = (2787.6564857656454 - 2538.967820972723) / 2787.6564857656454 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToWriter(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.638764971736627 < 1.8293183416552434. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 198.79166515082673 (T) = (0 -1630.7247483009853) / Math.Sqrt((187.36170648535764 / (124)) + (17.43963759475386 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.1498994590298046 = (1918.2728038731582 - 1630.7247483009853) / 1918.2728038731582 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToWriter(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.7323625391655675 < 1.9111804033902116. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 147.79542194186251 (T) = (0 -1731.9700646476024) / Math.Sqrt((138.52816994919186 / (124)) + (65.25687030039691 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.13407735788030786 = (2000.1441011034342 - 1731.9700646476024) / 2000.1441011034342 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToString(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 7.002036002753373 < 8.276224709059404. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 23.729203844469122 (T) = (0 -6969.325208595424) / Math.Sqrt((822486.8725176402 / (125)) + (1782.146215243782 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2171783019585326 = (8902.825798048136 - 6969.325208595424) / 8902.825798048136 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToStream(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.3374556363233614 < 2.5015579804307237. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 81.93519339815794 (T) = (0 -2354.9656834454922) / Math.Sqrt((309.10192310139485 / (124)) + (274.7564679793746 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.10615390143603341 = (2634.6433544084693 - 2354.9656834454922) / 2634.6433544084693 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Document.Tests.Perf_DocumentParse

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.13 μs 3.61 μs 0.87 0.01 True
6.35 μs 5.34 μs 0.84 0.01 True
960.07 ns 827.66 ns 0.86 0.01 True
12.30 μs 9.62 μs 0.78 0.01 True
551.60 ns 502.85 ns 0.91 0.01 True
3.49 ms 2.95 ms 0.85 0.01 True
3.05 ms 2.65 ms 0.87 0.01 True
6.06 μs 5.05 μs 0.83 0.01 True
990.99 ns 843.25 ns 0.85 0.01 True
534.90 ns 489.36 ns 0.91 0.01 True
2.88 μs 2.61 μs 0.91 0.01 True
12.71 μs 10.02 μs 0.79 0.01 True
33.80 ms 23.15 ms 0.68 0.01 True
3.66 μs 3.24 μs 0.89 0.01 True
34.14 ms 23.48 ms 0.69 0.01 True
2.57 μs 2.30 μs 0.89 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.6096278044319314 < 3.855079251345591. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 42.134238603147764 (T) = (0 -3619.5917124762855) / Math.Sqrt((3642.330959991718 / (124)) + (740.9572190372018 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.07884854281518164 = (3929.4208180903483 - 3619.5917124762855) / 3929.4208180903483 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: BasicJson) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.343361804498409 < 6.064919473235424. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 201.3391110533836 (T) = (0 -5355.523773904277) / Math.Sqrt((1428.0820796810515 / (125)) + (322.44345390287083 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.15040084032549758 = (6303.5888311802 - 5355.523773904277) / 6303.5888311802 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 827.6579241254638 < 908.9739014581605. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 159.61688550122716 (T) = (0 -826.632535774662) / Math.Sqrt((29.55660533937118 / (124)) + (9.375188047951578 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.12534981808387854 = (945.1007418345631 - 826.632535774662) / 945.1007418345631 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 9.621540042590311 < 11.673449078553258. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 270.2170588256482 (T) = (0 -9646.713467402218) / Math.Sqrt((8136.1373427527005 / (125)) + (826.6919009601933 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.21236022510165378 = (12247.621025293238 - 9646.713467402218) / 12247.621025293238 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 502.8471099025922 < 523.1567524539446. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.43570360641604 (T) = (0 -509.54954021983673) / Math.Sqrt((22.574963190229067 / (124)) + (15.318594928886679 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.05031324802532584 = (536.5448545642397 - 509.54954021983673) / 536.5448545642397 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.945679656462585 < 3.3092512213879948. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 61.269820075307635 (T) = (0 -2970569.1293124724) / Math.Sqrt((2529768648.7716217 / (125)) + (348419978.1217261 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.10426872118828699 = (3316361.9487010236 - 2970569.1293124724) / 3316361.9487010236 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.6460081808510636 < 2.911068856581514. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 109.68532803773347 (T) = (0 -2650088.980170073) / Math.Sqrt((1146180770.976238 / (124)) + (75932239.27270187 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.12436090448185193 = (3026462.6074078125 - 2650088.980170073) / 3026462.6074078125 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: BasicJson) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.048664465959488 < 5.765542273626972. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 256.4507677934865 (T) = (0 -5029.603777303447) / Math.Sqrt((784.2218263545568 / (124)) + (238.67880080737993 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.16155297932604962 = (5998.713876114213 - 5029.603777303447) / 5998.713876114213 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 843.252055291621 < 925.5086587805935. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 126.82822757560422 (T) = (0 -845.7323938484703) / Math.Sqrt((47.98553719896923 / (122)) + (12.53496756319963 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.11899135446322127 = (959.9592445919582 - 845.7323938484703) / 959.9592445919582 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: HelloWorld) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 489.3613348077968 < 508.04817622402754. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 52.13332425131009 (T) = (0 -490.99669593306567) / Math.Sqrt((21.582361791242786 / (123)) + (4.070776512323902 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.05591630487500954 = (520.0775084544395 - 490.99669593306567) / 520.0775084544395 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: BasicJson) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.610099622662449 < 2.7364479364963494. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 48.8298778664871 (T) = (0 -2621.926968775789) / Math.Sqrt((1001.3200174985266 / (125)) + (143.50912676665962 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.06245452612994988 = (2796.586450311428 - 2621.926968775789) / 2796.586450311428 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 10.016928163635878 < 12.08964594620726. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 198.0623566172979 (T) = (0 -10006.098672120454) / Math.Sqrt((10061.156099869384 / (126)) + (2631.9601308000474 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.20397544628501252 = (12570.087977089066 - 10006.098672120454) / 12570.087977089066 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.151947701298702 < 32.03972785535715. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 642.3128344304703 (T) = (0 -23158931.137054503) / Math.Sqrt((21124281568.77256 / (124)) + (2889778562.8104467 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.311732986574902 = (33648178.229269184 - 23158931.137054503) / 33648178.229269184 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: Json400B) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.239102259986635 < 3.4783005774887603. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 65.82471982079757 (T) = (0 -3270.537076723631) / Math.Sqrt((1085.8355112765535 / (124)) + (492.83637598704007 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.09173790331276901 = (3600.8736780412764 - 3270.537076723631) / 3600.8736780412764 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: Json400KB) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 23.477837466666667 < 32.375307904493724. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 605.2677484139717 (T) = (0 -23505200.16568621) / Math.Sqrt((20942014016.05345 / (125)) + (3911734732.4289517 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.3077027188962828 = (33952466.38584553 - 23505200.16568621) / 33952466.38584553 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: BasicJson) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.29963137476702 < 2.4445122179414494. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 75.2510139049536 (T) = (0 -2296.6326284635584) / Math.Sqrt((522.6348091986202 / (124)) + (97.39864842238592 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.0821492479649073 = (2502.185266364253 - 2296.6326284635584) / 2502.185266364253 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
55.42 μs 48.96 μs 0.88 0.01 True
50.59 μs 43.56 μs 0.86 0.01 True
49.67 μs 42.84 μs 0.86 0.01 True
54.58 μs 48.14 μs 0.88 0.01 True
87.44 μs 74.95 μs 0.86 0.01 False
82.42 μs 70.18 μs 0.85 0.01 False
87.35 μs 74.59 μs 0.85 0.01 False
83.30 μs 71.00 μs 0.85 0.01 False
51.07 μs 44.61 μs 0.87 0.01 True
54.95 μs 48.66 μs 0.89 0.01 True

graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToUtf8Bytes(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 48.9572617014197 < 52.564365006231064. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 76.84743147121746 (T) = (0 -48827.70688604783) / Math.Sqrt((318028.29225301 / (119)) + (84346.98046471433 / (30))) is greater than 1.976233308895103 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (119) + (30) - 2, .975) and 0.10438454605696314 = (54518.606921172424 - 48827.70688604783) / 54518.606921172424 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToStream(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 43.559547606646824 < 48.069431860423315. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 64.76371430131414 (T) = (0 -43839.94056753942) / Math.Sqrt((191568.1329841427 / (118)) + (223643.1314364589 / (30))) is greater than 1.9763456545936318 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (30) - 2, .975) and 0.12338721902398629 = (50010.61074962697 - 43839.94056753942) / 50010.61074962697 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToWriter(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 42.83664137720156 < 47.26007401799931. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 81.56608976134379 (T) = (0 -43039.13804356719) / Math.Sqrt((190495.69107518514 / (118)) + (127402.12522056726 / (30))) is greater than 1.9763456545936318 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (30) - 2, .975) and 0.1267058030357563 = (49283.66430600407 - 43039.13804356719) / 49283.66430600407 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToWriter(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 48.135156032051285 < 51.66428808834772. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 89.94109990327208 (T) = (0 -48161.838698305764) / Math.Sqrt((321534.77036508964 / (118)) + (36088.81613073592 / (30))) is greater than 1.9763456545936318 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (30) - 2, .975) and 0.10477631543271469 = (53798.664544476655 - 48161.838698305764) / 53798.664544476655 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeObjectProperty(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 74.94702471505697 < 83.33812761226422. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 35.0961709612648 (T) = (0 -74718.22674691193) / Math.Sqrt((16198025.038862243 / (119)) + (111784.71760845126 / (30))) is greater than 1.976233308895103 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (119) + (30) - 2, .975) and 0.14940875021791022 = (87842.69385095808 - 74718.22674691193) / 87842.69385095808 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToString(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 70.1812660063244 < 78.84289452738419. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.08337318426563 (T) = (0 -70290.61791106733) / Math.Sqrt((16178962.788804913 / (119)) + (100145.84330542272 / (30))) is greater than 1.976233308895103 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (119) + (30) - 2, .975) and 0.1532407506081207 = (83011.33759276706 - 70290.61791106733) / 83011.33759276706 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 74.5888542942177 < 82.83228511131891. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 31.683028127922903 (T) = (0 -74838.57220017756) / Math.Sqrt((16959347.408424567 / (118)) + (455507.1722605069 / (30))) is greater than 1.9763456545936318 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (30) - 2, .975) and 0.1443931979695929 = (87468.41659344113 - 74838.57220017756) / 87468.41659344113 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeObjectProperty(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 70.99986384469697 < 79.11217761757993. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 32.92316212324095 (T) = (0 -70812.50337347305) / Math.Sqrt((16479041.68051826 / (116)) + (102114.97280636145 / (30))) is greater than 1.9765750658302461 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (116) + (30) - 2, .975) and 0.1506169198887558 = (83369.33597052428 - 70812.50337347305) / 83369.33597052428 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToUtf8Bytes(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 44.610267829663826 < 48.17566008633852. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 107.37122502276924 (T) = (0 -44619.163735084316) / Math.Sqrt((137973.27431450135 / (120)) + (45686.74772418065 / (30))) is greater than 1.9761224936134032 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (120) + (30) - 2, .975) and 0.11064101628215983 = (50170.02644821799 - 44619.163735084316) / 50170.02644821799 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToStream(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 48.66118543105501 < 52.21137401175213. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 82.52572842780316 (T) = (0 -48785.30070535986) / Math.Sqrt((314371.93935113866 / (119)) + (61362.89116854978 / (30))) is greater than 1.976233308895103 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (119) + (30) - 2, .975) and 0.10379254713061316 = (54435.27673103364 - 48785.30070535986) / 54435.27673103364 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in MicroBenchmarks.Serializers.Json_ToStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.80 μs 2.58 μs 0.92 0.01 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream<Location>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToStream<Location>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream<Location>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToStream<Location>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### MicroBenchmarks.Serializers.Json_ToStream<Location>.JsonNet_ #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.575940395330417 < 2.677344758753998. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 58.30441459737964 (T) = (0 -2560.843496239583) / Math.Sqrt((769.4437085566019 / (125)) + (276.801095812437 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.08197525255291115 = (2789.514665438994 - 2560.843496239583) / 2789.514665438994 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Half

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
28.79 ns 6.31 ns 0.22 0.01 True
67.89 ns 29.67 ns 0.44 0.01 True
66.37 ns 30.50 ns 0.46 0.01 True
70.90 ns 30.90 ns 0.44 0.01 True
67.54 ns 28.43 ns 0.42 0.01 True
28.93 ns 6.30 ns 0.22 0.01 True
28.89 ns 6.39 ns 0.22 0.02 True

graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Half*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Half*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Half*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Half*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Half.HalfToSingle(value: 6.1E-05) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.306197573858023 < 27.435432365867097. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1252.2133663781838 (T) = (0 -6.344470942249761) / Math.Sqrt((0.01569833693287043 / (124)) + (0.005957130993596911 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.7806578000453595 = (28.92499000904423 - 6.344470942249761) / 28.92499000904423 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.SingleToHalf(value: NaN) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 29.66630881627918 < 64.4311863995324. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1590.6923297920152 (T) = (0 -29.708746691142355) / Math.Sqrt((0.03448872659850996 / (124)) + (0.00894795702328773 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.5624543388119544 = (67.89862025022873 - 29.708746691142355) / 67.89862025022873 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.SingleToHalf(value: 12345) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 30.50058004128115 < 63.09582595473773. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1060.3570159382537 (T) = (0 -30.537746241154156) / Math.Sqrt((0.012167764885616458 / (125)) + (0.03142785777849749 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5402116594475951 = (66.41696525941718 - 30.537746241154156) / 66.41696525941718 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.SingleToHalf(value: 65520) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 30.902149826047495 < 67.36611767444876. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 2148.7837972871903 (T) = (0 -30.69985542531493) / Math.Sqrt((0.015612409508844476 / (125)) + (0.006750607946627737 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5669690918464867 = (70.89529834307245 - 30.69985542531493) / 70.89529834307245 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.SingleToHalf(value: 6.097555E-05) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 28.426990754202283 < 64.06070573608528. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1971.6208134561555 (T) = (0 -28.52759168764103) / Math.Sqrt((0.015649149610068313 / (125)) + (0.007911713669884971 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5768040172100047 = (67.40988300401101 - 28.52759168764103) / 67.40988300401101 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.HalfToSingle(value: 12344) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.300351399112794 < 27.453973068974058. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1815.224105640133 (T) = (0 -6.332541447361956) / Math.Sqrt((0.004416046856490753 / (125)) + (0.003581161297292906 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.7809570826557828 = (28.910048880560787 - 6.332541447361956) / 28.910048880560787 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Half.HalfToSingle(value: NaN) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.387883105924344 < 27.49177915438959. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 550.4895888087624 (T) = (0 -6.289921364029894) / Math.Sqrt((0.004492133839709429 / (125)) + (0.049596256586207005 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.782465651230476 = (28.914612334137708 - 6.289921364029894) / 28.914612334137708 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.85 μs 439.47 ns 0.24 0.10 True
1.74 μs 613.99 ns 0.35 0.01 True
105.48 ns 88.66 ns 0.84 0.01 True
95.23 ns 41.60 ns 0.44 0.01 True
279.49 ns 194.47 ns 0.70 0.01 True
288.65 ns 161.02 ns 0.56 0.01 True
107.24 ns 88.82 ns 0.83 0.01 True
194.21 ns 89.47 ns 0.46 0.02 True
94.74 ns 67.32 ns 0.71 0.01 True
27.37 ns 19.65 ns 0.72 0.12 True
233.66 ns 170.59 ns 0.73 0.01 True
731.42 ns 673.64 ns 0.92 0.01 False
112.63 ns 94.33 ns 0.84 0.01 True
22.18 ns 13.09 ns 0.59 0.01 True
944.33 ns 840.14 ns 0.89 0.01 True
294.43 ns 175.71 ns 0.60 0.01 True
106.81 ns 88.46 ns 0.83 0.01 True
106.45 ns 88.69 ns 0.83 0.01 True
116.71 ns 52.72 ns 0.45 0.02 True
242.89 ns 100.00 ns 0.41 0.03 True
100.96 ns 39.09 ns 0.39 0.01 True
291.62 ns 164.39 ns 0.56 0.01 True
37.87 ns 29.78 ns 0.79 0.04 True
94.09 ns 67.53 ns 0.72 0.03 True
294.74 ns 169.27 ns 0.57 0.01 True
3.37 μs 2.84 μs 0.84 0.11 False
27.60 ns 20.55 ns 0.74 0.01 True
113.17 ns 94.22 ns 0.83 0.01 True
113.24 ns 94.52 ns 0.83 0.01 True
47.81 ns 15.56 ns 0.33 0.04 True
22.08 ns 12.18 ns 0.55 0.11 True
113.03 ns 59.64 ns 0.53 0.05 True
112.39 ns 94.42 ns 0.84 0.01 True
23.46 ns 17.02 ns 0.73 0.05 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_String*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_String*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58lzfdql1fehvs91yzkt9xam7ahjbhvpd9edll13ab46i74ktwwgkgbi792e5gkuuzevo5qm8qt83edag7zovoe686gmtw730kms2i5xgji4xcp25287q68fvhwszd3mszht2uh7bchlgkj5qnq1x9m4lg7vwn8cq5l756akua6oyx9k71bmxbysnmhvxvlxde4k9maumfgxd8gxhxx4mwpph2ttyox9zilt3ylv1q9s4bopfuoa8qlrzodg2q67sh85wx4slcd6w7ufnendaxai633ove2ktbaxdt2sz6y6mo42473xd274gz833p6hj3mu77c4m4od9e5s8btxleh0efqnu9zj9rwtbk5758lio35b3q426j5fwwq1qyknfedrsmqyfw1m38mkkotdf7n0vr6p3erhy8dkzntr9fwjrslxjgrbegih0n6bpb5bfuy55bu65ce9kejcfifxwpcs05umrsb8kvd64q2iwugbbi7vd35g5ho0rff9rhombgzzaniyq7bbjbqr88jyw4ccgnoyl31of3a5thv0vg08gnrqzxas800hewtw8tnwgw5pav81ntdpdd62689x3iqpc317y82b3e2trbpdzieoxldaz009tz37gqmh4bdp1bv9lnl5s58udb11z0h7i2sdl5nbyhjyfzxwzezmp4qx0i3eyvsd3fg8sryq9jhlvkonnfcvb4snl4mcbimdzg49tzdhqjmfxfcq3p1st6b9x2xyevo17evpqp4yc4f2rm0f26ivr3t2f5m0boc44vituxaovcqy1jrkcs6im2kdu3jvcexx2k76egve63aon5a6nbxss4rcke90npmqp35qluf571ms160y2nhaqef835wah41qru8tauu362v0r8konl8", oldChar: 'b', newChar: '+') #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 439.4675270726428 < 1.753228187994131. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 306.23764088744394 (T) = (0 -468.88151483337657) / Math.Sqrt((183.99765069370036 / (124)) + (565.574523608078 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.7465348427194312 = (1849.8854827385849 - 468.88151483337657) / 1849.8854827385849 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.PadLeft(n: 2142) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 613.9930225023563 < 1.6200951433349113. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 593.3980197975095 (T) = (0 -616.8064025997315) / Math.Sqrt((181.36079962217198 / (125)) + (61.65261056023747 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.6430310640245731 = (1727.899378455137 - 616.8064025997315) / 1727.899378455137 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpperInvariant(s: "TeSt") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 88.65803926934952 < 101.25744789856954. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 249.9303312065759 (T) = (0 -88.80775052466399) / Math.Sqrt((0.24008157652345086 / (125)) + (0.09666274656534589 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16792907695417383 = (106.73098658414833 - 88.80775052466399) / 106.73098658414833 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.CtorCharCount(size: 100) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 41.599782902830206 < 90.36715085843458. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 386.68935231577376 (T) = (0 -41.00094115651385) / Math.Sqrt((0.8625421492132115 / (126)) + (0.37524407074039107 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.5674834574850192 = (94.7962381232938 - 41.00094115651385) / 94.7962381232938 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 194.46899843096236 < 265.05009401133776. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 107.92234244666868 (T) = (0 -199.42919410427976) / Math.Sqrt((65.1975230887607 / (122)) + (2.9790871060985986 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.3010863324992572 = (285.34167147914275 - 199.42919410427976) / 285.34167147914275 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 161.02286909982695 < 274.87387796047057. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 448.3023997226163 (T) = (0 -159.9141943417657) / Math.Sqrt((3.5931208612299934 / (124)) + (1.6004223340910928 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.4457853488673946 = (288.5419828129074 - 159.9141943417657) / 288.5419828129074 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLowerInvariant(s: "TEST") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 88.82421582712279 < 101.53978246363756. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 299.40971987732524 (T) = (0 -88.5737654137184) / Math.Sqrt((0.13163766784915135 / (125)) + (0.07794884877066081 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16961678753896708 = (106.66613207558657 - 88.5737654137184) / 106.66613207558657 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "b") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 89.46920755702051 < 183.28154303951422. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 262.1725184987832 (T) = (0 -88.22498967227439) / Math.Sqrt((13.77453403500639 / (125)) + (1.7374407655546549 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5492267639212287 = (195.71922778675648 - 88.22498967227439) / 195.71922778675648 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: RemoveEmptyEntries) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 67.31859049630319 < 89.94576571133308. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 208.4178884401466 (T) = (0 -67.46491698224045) / Math.Sqrt((0.10888953203514942 / (123)) + (0.4747050191206398 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.2853710886193777 = (94.40552419283189 - 67.46491698224045) / 94.40552419283189 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLower(s: "test") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.647805411719887 < 25.973021416113372. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 44.77237053062523 (T) = (0 -19.883342706491486) / Math.Sqrt((0.14382220908982 / (125)) + (0.8031479908255318 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.27339643917897793 = (27.36477465651889 - 19.883342706491486) / 27.36477465651889 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 170.59066975698252 < 221.6678965404454. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 125.42984130265991 (T) = (0 -171.0100839833597) / Math.Sqrt((30.51959638058452 / (124)) + (1.1313924661446104 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.28097099472311343 = (237.8347503763168 - 171.0100839833597) / 237.8347503763168 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Format_MultipleArgs #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 673.6445138777364 < 694.2044283881479. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 31.136482174890133 (T) = (0 -680.2317376631225) / Math.Sqrt((522.2708790678646 / (123)) + (21.997741434968763 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.09267496496058954 = (749.7111965323166 - 680.2317376631225) / 749.7111965323166 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpper(s: "test") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 94.32987940629843 < 107.34023647039452. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 176.9928238957133 (T) = (0 -94.54377856107321) / Math.Sqrt((0.41297039359466264 / (124)) + (0.2204231225838398 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.1620917626779763 = (112.83309358938583 - 94.54377856107321) / 112.83309358938583 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpperInvariant(s: "TEST") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.093300200094538 < 21.022272215299466. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 173.8826128263424 (T) = (0 -13.043944988100128) / Math.Sqrt((0.004066089093483493 / (124)) + (0.08118248702586811 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.4109479914562033 = (22.143961482019623 - 13.043944988100128) / 22.143961482019623 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Join_Enumerable #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 840.1370591007001 < 896.2919341706486. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 138.82173623971138 (T) = (0 -840.8101457964541) / Math.Sqrt((22.96892955444394 / (124)) + (9.39438813660275 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.10438993155476568 = (938.8127438719932 - 840.8101457964541) / 938.8127438719932 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 175.71010230203098 < 280.7345943791123. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 256.6612280694021 (T) = (0 -173.4949962042024) / Math.Sqrt((2.989000682758559 / (125)) + (6.0697753988644 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.41302456082905975 = (295.5745413287673 - 173.4949962042024) / 295.5745413287673 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpperInvariant(s: "test") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 88.4613465796456 < 101.17092651055349. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 206.59534508284665 (T) = (0 -88.72744499915457) / Math.Sqrt((0.2824973739848799 / (125)) + (0.16079729562232745 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16891955801086014 = (106.7615606339994 - 88.72744499915457) / 106.7615606339994 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLowerInvariant(s: "TeSt") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 88.68954950004748 < 101.18664333009866. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 180.7328542704836 (T) = (0 -88.73011837060811) / Math.Sqrt((0.28019701133603825 / (125)) + (0.22421714667525894 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16720084734262788 = (106.544438821149 - 88.73011837060811) / 106.544438821149 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.IndexOfAny #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 52.72486868155232 < 111.07427221129775. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 158.57594516945517 (T) = (0 -52.36136539782512) / Math.Sqrt((14.969953179299926 / (125)) + (1.9403369348739845 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5653348780354575 = (120.46369205140977 - 52.36136539782512) / 120.46369205140977 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+') #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 99.99795266161595 < 230.7189463709108. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 262.847514863109 (T) = (0 -100.36417802941135) / Math.Sqrt((16.76789047505291 / (124)) + (5.193818742995535 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.5925488874715543 = (246.32201249028267 - 100.36417802941135) / 246.32201249028267 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "bad", newValue: "nice") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 39.089737212763524 < 95.76185584835474. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 1193.3736080963267 (T) = (0 -39.14009332962389) / Math.Sqrt((0.16930171723472334 / (125)) + (0.03939330299826375 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.6116111970323623 = (100.77554509954608 - 39.14009332962389) / 100.77554509954608 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 164.391555321077 < 276.5363648315572. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 365.28925720843097 (T) = (0 -163.7415295533448) / Math.Sqrt((2.7134944284315936 / (124)) + (2.954703646099673 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.4363032142818473 = (290.47802595634323 - 163.7415295533448) / 290.47802595634323 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.PadLeft(n: 18) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 29.784154501918124 < 36.050656901079975. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 145.19876978985258 (T) = (0 -29.94697952780289) / Math.Sqrt((0.287875777610859 / (125)) + (0.024386343294775566 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2129983416621502 = (38.05199037451968 - 29.94697952780289) / 38.05199037451968 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 67.53135322833403 < 89.44103170042125. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 121.33069275531324 (T) = (0 -67.58743838516756) / Math.Sqrt((0.2194512352305856 / (125)) + (1.391023915853975 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2825395904173694 = (94.20371839678974 - 67.58743838516756) / 94.20371839678974 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 169.268217084208 < 279.92582254422524. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 456.67357854502876 (T) = (0 -166.99254148537347) / Math.Sqrt((3.790512645615591 / (124)) + (1.4318063822111267 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.4334932750683204 = (294.77592080749383 - 166.99254148537347) / 294.77592080749383 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Format_OneArg(s: "Testing {0}, {0:C}, {0:E} - {0:F4}{0:G}{0:N} , !!", o: 3.14159) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.8379314874775696 < 3.2143123454325093. IsChangePoint: Marked as a change because one of 9/27/2023 6:04:31 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.5361157247203 (T) = (0 -2827.340464032204) / Math.Sqrt((24412.88444001688 / (133)) + (407.36181759930304 / (22))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (133) + (22) - 2, .975) and 0.1479495731404675 = (3318.2783259121757 - 2827.340464032204) / 3318.2783259121757 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpper(s: "TEST") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.55413534781035 < 26.15042447911813. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 131.5790597660125 (T) = (0 -20.193648195172702) / Math.Sqrt((0.016333809403431637 / (124)) + (0.08738506990626758 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2644512922090108 = (27.453855851121766 - 20.193648195172702) / 27.453855851121766 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLower(s: "TeSt") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 94.22202179662825 < 107.27777832533377. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 119.109019790139 (T) = (0 -94.37432170924617) / Math.Sqrt((0.25771280040775635 / (124)) + (0.6521136595756218 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.16301824199474038 = (112.75552998211596 - 94.37432170924617) / 112.75552998211596 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToUpper(s: "TeSt") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 94.51771951917269 < 107.48314305256399. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 125.39268375684694 (T) = (0 -94.66765318512165) / Math.Sqrt((0.3056840958845675 / (124)) + (0.5593003881471823 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.16138473516522542 = (112.88567851644488 - 94.66765318512165) / 112.88567851644488 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'z', newChar: 'y') #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.564666174113436 < 45.613098379321634. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 381.2568975404986 (T) = (0 -15.671571178401729) / Math.Sqrt((0.022205211177269965 / (125)) + (0.21109978706239452 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.6738785696249318 = (48.054404644239554 - 15.671571178401729) / 48.054404644239554 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLowerInvariant(s: "test") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 12.177724845040617 < 20.94877108579518. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 70.98861863500444 (T) = (0 -12.771009440476567) / Math.Sqrt((0.005967926942164105 / (124)) + (0.5097133279032697 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.42048179008997594 = (22.037287564198184 - 12.771009440476567) / 22.037287564198184 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'i', newChar: 'I') #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 59.6420749872866 < 107.72798154715453. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 131.7721161915901 (T) = (0 -59.89994730912838) / Math.Sqrt((15.161537449244562 / (126)) + (1.9206130118851512 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.4857386433039956 = (116.47763637923326 - 59.89994730912838) / 116.47763637923326 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.ToLower(s: "TEST") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 94.41756555106068 < 107.22541828680474. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 176.4312956530637 (T) = (0 -94.45342925974776) / Math.Sqrt((0.3618986928231114 / (125)) + (0.23546801281793533 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16221029198395856 = (112.74121459837656 - 94.45342925974776) / 112.74121459837656 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_String.CtorCharCount(size: 10) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 17.018149677492268 < 22.439438778334875. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 61.573643427625264 (T) = (0 -17.681206363316246) / Math.Sqrt((0.24491161422095686 / (125)) + (0.21948072708371225 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.25115389543552796 = (23.611268397529575 - 17.681206363316246) / 23.611268397529575 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
323.31 μs 281.15 μs 0.87 0.01 True
356.65 μs 326.05 μs 0.91 0.01 False
105.59 μs 96.77 μs 0.92 0.01 True
389.59 μs 337.49 μs 0.87 0.01 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "repeated.xml") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 281.1456546275395 < 307.9329137974534. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 78.33467452436382 (T) = (0 -280412.5235639932) / Math.Sqrt((25238501.70012066 / (125)) + (3264286.08322467 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1347372117665623 = (324077.87249986443 - 280412.5235639932) / 324077.87249986443 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "deep.xml") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 326.05472354978355 < 338.973704061919. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 40.59202660959906 (T) = (0 -325746.6490546536) / Math.Sqrt((75855416.46439005 / (124)) + (2331936.804253638 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.09376853204212239 = (359451.928753587 - 325746.6490546536) / 359451.928753587 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "simple.xml") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 96.76808231951438 < 100.72171441857797. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 46.04738613387341 (T) = (0 -97626.20038736929) / Math.Sqrt((2160177.157179921 / (125)) + (360970.02590687305 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.07472168875876778 = (105510.09269460425 - 97626.20038736929) / 105510.09269460425 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "names.xml") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 337.49281671183377 < 369.01474378220007. IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 86.83998414327583 (T) = (0 -336831.4786386305) / Math.Sqrt((30808952.874823317 / (124)) + (2914749.484158273 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.1316181375793165 = (387884.05563847907 - 336831.4786386305) / 387884.05563847907 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.StreamReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
141.07 μs 108.75 μs 0.77 0.04 True
113.50 μs 81.58 μs 0.72 0.04 True
162.06 μs 128.79 μs 0.79 0.03 True
120.62 μs 87.83 μs 0.73 0.01 True
117.33 μs 84.76 μs 0.72 0.01 True
1.10 ms 935.86 μs 0.85 0.02 False
122.32 μs 90.78 μs 0.74 0.02 True
115.12 μs 82.79 μs 0.72 0.04 True
117.86 μs 86.33 μs 0.73 0.04 True
338.32 μs 294.95 μs 0.87 0.01 False
680.43 μs 597.02 μs 0.88 0.01 False
217.15 μs 181.05 μs 0.83 0.01 False
120.90 μs 88.02 μs 0.73 0.01 True

graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps #### Prerequisites (Build files either built locally or downloaded from payload above) - Libraries build extracted to `runtime/artifacts` or build instructions: [Libraries README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md) args: `-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0` - CoreCLR product build extracted to `runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release`, build instructions: [CoreCLR README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/README.md) args: `-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0` - Mono Runtime build extracted to `runtime/artifacts/bin/mono/$RunOS.$RunArch.Release`, build instructions: [MONO README](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/mono/README.md) args: `-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release ` - Dotnet SDK installed for dotnet commands - Running commands from the runtime folder Linux ```cmd # Set $RunDir to the runtime directory RunDir=`pwd` # Set the OS, arch, and OSId RunOS='linux' RunOSId='linux' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun # Create Core Root $RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance # One line run: python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" # Individual Commands: # Restore dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages ``` Windows ```cmd # Set $RunDir to the runtime directory $RunDir="FullPathHere" # Set the OS, arch, and OSId RunOS='windows' RunOSId='win' RunArch='x64' # Create mono dotnet mkdir -p $RunDir/artifacts/dotnet-mono $RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y # Create Core Root $RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release # Clone performance git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance # One line run: python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" # Individual Commands: # Restore dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Build dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 # Run dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 33, 128]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 108.74743425287356 < 134.06544771443342. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 50.29075096175901 (T) = (0 -108681.0526617665) / Math.Sqrt((48396796.89365105 / (124)) + (2172170.666386359 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.23940727587776303 = (142889.94519003585 - 108681.0526617665) / 142889.94519003585 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 1024]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 81.58168311631943 < 107.83518337452024. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.49867937599433 (T) = (0 -81268.94777851111) / Math.Sqrt((38537728.288321 / (124)) + (396168.68610362586 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.294147406405262 = (115135.86337428873 - 81268.94777851111) / 115135.86337428873 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 9, 32]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 128.78509740925057 < 154.09831007809214. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 43.73647829152867 (T) = (0 -128225.60801984352) / Math.Sqrt((78047304.16033567 / (123)) + (4426808.222364775 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.23174203444712702 = (166904.36516016675 - 128225.60801984352) / 166904.36516016675 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [1025, 2048]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 87.82954441287879 < 114.51748775274548. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 60.74017946097995 (T) = (0 -87723.19341291134) / Math.Sqrt((39638464.05075238 / (125)) + (243800.03185936387 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.28308998673639796 = (122362.90718491642 - 87723.19341291134) / 122362.90718491642 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 84.76176632289084 < 111.71445340708398. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 60.139988847350445 (T) = (0 -84446.77154438553) / Math.Sqrt((40266057.53102673 / (124)) + (339952.7858484653 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2922084661098558 = (119310.23119229969 - 84446.77154438553) / 119310.23119229969 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 935.860087254902 < 1.0440569321726192. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.4076405124631 (T) = (0 -939301.84058424) / Math.Sqrt((4852183545.399902 / (125)) + (28567013.255242866 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.18765719699939895 = (1156287.5144762562 - 939301.84058424) / 1156287.5144762562 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 33, 128]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 90.78433758665027 < 116.88642332732371. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 57.3988194322243 (T) = (0 -90533.4748382063) / Math.Sqrt((45173106.29443399 / (124)) + (471151.354295369 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2810049267359618 = (125916.68316614386 - 90533.4748382063) / 125916.68316614386 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 129, 1024]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 82.78753994236584 < 109.68488799297747. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.236063778322524 (T) = (0 -82866.34242314377) / Math.Sqrt((40484685.69209717 / (124)) + (367301.48601905804 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.29380600006194607 = (117342.17853792677 - 82866.34242314377) / 117342.17853792677 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 0, 1024]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 86.33489524785324 < 111.92283676890221. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 52.51958248116272 (T) = (0 -86482.3290727273) / Math.Sqrt((38946723.37808646 / (125)) + (2772962.3812481943 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2784994722331001 = (119864.54028023627 - 86482.3290727273) / 119864.54028023627 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 8]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 294.9545693396227 < 318.70998147836275. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 34.92076094274001 (T) = (0 -293310.0754791928) / Math.Sqrt((323669660.2163593 / (125)) + (3715025.4832344265 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16395560803072742 = (350830.7433153297 - 293310.0754791928) / 350830.7433153297 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 1]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 597.0210870192308 < 646.6522475223214. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 33.01889904175426 (T) = (0 -597803.6016437488) / Math.Sqrt((1429122505.1326122 / (126)) + (24729596.279270478 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.16153696480542912 = (712975.4998740338 - 597803.6016437488) / 712975.4998740338 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 9, 32]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 181.05174712643677 < 205.21867494936342. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 38.594737000213215 (T) = (0 -184299.17134288605) / Math.Sqrt((92465015.34922244 / (125)) + (2993518.912840045 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1609851318380846 = (219661.38901285775 - 184299.17134288605) / 219661.38901285775 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 129, 1024]) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 88.01965709796488 < 113.80484407104522. IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM. IsImprovementStdDev: Marked as improvement because 59.897350133329816 (T) = (0 -87401.71164246731) / Math.Sqrt((38754428.05208156 / (124)) + (329146.5960956423 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2804663668243501 = (121469.94610484196 - 87401.71164246731) / 121469.94610484196 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
matouskozak commented 8 months ago

Similar to https://github.com/dotnet/perf-autofiling-issues/issues/22688.