search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.14k stars 4.71k forks source link

[Perf] Linux/x64: 16 Regressions on 8/28/2023 7:02:26 PM #91621

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Net.Primitives.Tests.IPAddressPerformanceTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
11.85 ns 12.86 ns 1.09 0.26 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*'
### Payloads [Baseline]() [Compare]() ### System.Net.Primitives.Tests.IPAddressPerformanceTests.TryFormat(address: 143.24.20.36) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.8599746068092 > 12.056727412948058. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -11.07493886512902 (T) = (0 -12.724523881091898) / Math.Sqrt((0.19911632037081872 / (13)) + (0.05423179428190999 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.1300605723515152 = (11.260037021390588 - 12.724523881091898) / 11.260037021390588 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in SciMark2.kernel

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
493.53 ms 635.56 ms 1.29 0.11 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SciMark2.kernel*'
### Payloads [Baseline]() [Compare]() ### SciMark2.kernel.benchSparseMult #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 635.5553794615385 > 518.1746533384616. IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -20.80254686105018 (T) = (0 -635066391.063553) / Math.Sqrt((549931303529396.75 / (13)) + (938876866845.3124 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.2708801752398864 = (499705954.53161454 - 635066391.063553) / 499705954.53161454 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.63 ΞΌs 41.26 ΞΌs 1.13 0.32 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
### Payloads [Baseline]() [Compare]() ### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 41.25734353618421 > 38.49274458266977. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -3.827698670859208 (T) = (0 -42439.53143484542) / Math.Sqrt((12469277.456513325 / (13)) + (9232831.4236689 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.11601557816957614 = (38027.7231474244 - 42439.53143484542) / 38027.7231474244 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.GetService

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.76 ns 9.80 ns 1.12 0.34 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.GetService*'
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.DependencyInjection.GetService.Singleton #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.796466543791762 > 8.928597380551507. IsChangePoint: Marked as a change because one of 8/14/2023 9:42:18 AM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -11.328714609799167 (T) = (0 -10.006959380549333) / Math.Sqrt((0.10588144097847962 / (13)) + (0.23316557766638304 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.17596968156133477 = (8.509538585436228 - 10.006959380549333) / 8.509538585436228 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Invoke

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
11.92 ns 12.93 ns 1.09 0.32 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Invoke*'
### Payloads [Baseline]() [Compare]() ### System.Reflection.Invoke.Property_Get_class #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.934025162337663 > 11.931842525878434. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -5.8134313269420215 (T) = (0 -12.43298178735799) / Math.Sqrt((0.13497822992729458 / (13)) + (0.3557251851021524 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.07916202584447414 = (11.52095930880151 - 12.43298178735799) / 11.52095930880151 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
197.37 ΞΌs 262.86 ΞΌs 1.33 0.03 False
263.78 ΞΌs 282.60 ΞΌs 1.07 0.03 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*'
### Payloads [Baseline]() [Compare]() ### PerfLabTests.CastingPerf2.CastingPerf.ScalarValueTypeObj #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 262.86413593749995 > 207.03602456933592. IsChangePoint: Marked as a change because one of 8/15/2023 7:48:35 PM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -571.6960913178664 (T) = (0 -262736.10639719927) / Math.Sqrt((36460.566257674574 / (13)) + (259414.6093935372 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.3330097792672661 = (197099.90915567105 - 262736.10639719927) / 197099.90915567105 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.CastingPerf2.CastingPerf.FooObjCastIfIsa #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 282.5962487244898 > 276.74016504166667. IsChangePoint: Marked as a change because one of 8/15/2023 7:48:35 PM, 9/2/2023 3:43:46 AM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -74.61168385739026 (T) = (0 -282425.7198429051) / Math.Sqrt((583774.0204949128 / (32)) + (291455.2700233112 / (6))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (6) - 2, .025) and -0.07329441012996876 = (263139.09508641274 - 282425.7198429051) / 263139.09508641274 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
553.59 ns 623.16 ns 1.13 0.14 False
513.02 ns 679.64 ns 1.32 0.11 False
32.68 ns 43.39 ns 1.33 0.36 False
36.20 ns 39.66 ns 1.10 0.07 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 623.1633759593363 > 580.7947470256283. IsChangePoint: Marked as a change because one of 8/19/2023 11:06:43 PM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -17.772546313404245 (T) = (0 -624.3361230397583) / Math.Sqrt((157.56908967091178 / (13)) + (0.8081864634680289 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.11016987457320086 = (562.378909155485 - 624.3361230397583) / 562.378909155485 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.IndexOfAnyFiveValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 679.640959709437 > 538.9842832860049. IsChangePoint: Marked as a change because one of 8/19/2023 11:06:43 PM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -525.0385517488116 (T) = (0 -679.307678175634) / Math.Sqrt((0.8420108968969636 / (13)) + (0.8752767012868232 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.323006116403176 = (513.4577004242815 - 679.307678175634) / 513.4577004242815 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.IndexOfAnyFiveValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 43.38838435545172 > 37.49482912108297. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -7.501317357406745 (T) = (0 -44.13679175536036) / Math.Sqrt((11.363373458839362 / (13)) + (0.5015702049745557 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.19148514243077333 = (37.043509972198216 - 44.13679175536036) / 37.043509972198216 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 39.66469474644949 > 38.01209476983544. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -81.56370161870893 (T) = (0 -39.86311581142425) / Math.Sqrt((0.009244118016176117 / (13)) + (0.030914726401205114 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.09926253540069081 = (36.26350805896773 - 39.86311581142425) / 36.26350805896773 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
17.80 ns 19.69 ns 1.11 0.09 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Int32.ToStringHex(value: 2147483647) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 19.68549505341147 > 18.863384749456166. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -15.032296546459138 (T) = (0 -20.33268632700021) / Math.Sqrt((0.1423980119275122 / (13)) + (0.2898717442461349 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.12488107189761949 = (18.075409778831073 - 20.33268632700021) / 18.075409778831073 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
878.11 ns 1.32 ΞΌs 1.50 0.07 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*'
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_VectorConvert.Widen_float #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.320969514442833 > 926.9380103399895. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -813.2144417580388 (T) = (0 -1320.5381608067803) / Math.Sqrt((2.2969982946450807 / (13)) + (2.8296111997410787 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.49601818195949177 = (882.7019462271061 - 1320.5381608067803) / 882.7019462271061 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.76 ΞΌs 6.57 ΞΌs 1.38 0.12 True
5.97 ΞΌs 7.63 ΞΌs 1.28 0.11 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'
### Payloads [Baseline]() [Compare]() ### System.MathBenchmarks.Single.ILogB #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.572198914035087 > 4.994373118578768. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -27.359992932869652 (T) = (0 -6706.209542008301) / Math.Sqrt((12.808989858092003 / (13)) + (126996.80894369919 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.4100567534574738 = (4755.985548499807 - 6706.209542008301) / 4755.985548499807 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.MathBenchmarks.Single.ScaleB #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.629899337557603 > 6.418856593990998. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -30.421500523990122 (T) = (0 -7626.589581188343) / Math.Sqrt((33167.80888767883 / (13)) + (303.2005721844885 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.25307102914785307 = (6086.318655355699 - 7626.589581188343) / 6086.318655355699 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6
Compare 4c013865c31cb154ccf88f7628f2d26758b00d3e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.96 ΞΌs 8.44 ΞΌs 1.06 0.08 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Double*'
### Payloads [Baseline]() [Compare]() ### System.MathBenchmarks.Double.ScaleB #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 8.439246719280314 > 8.406949340281045. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -19.49026983725177 (T) = (0 -8466.03511544965) / Math.Sqrt((1295.97645959424 / (13)) + (11640.797142937692 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.057907324648037065 = (8002.624538275389 - 8466.03511544965) / 8002.624538275389 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
cincuranet commented 1 year ago

Few are noise. But otherwise mostly valid. Seems only Linux x64 though.

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 22.04 Queue | TigerUbuntu Baseline | [a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6](https://github.com/dotnet/runtime/commit/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6) Compare | [4c013865c31cb154ccf88f7628f2d26758b00d3e](https://github.com/dotnet/runtime/commit/4c013865c31cb154ccf88f7628f2d26758b00d3e) Diff | [Diff](https://github.com/dotnet/runtime/compare/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6...4c013865c31cb154ccf88f7628f2d26758b00d3e) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Net.Primitives.Tests.IPAddressPerformanceTests Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [TryFormat - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 11.85 ns | 12.86 ns | 1.09 | 0.26 | True | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' ```
### Payloads [Baseline]() [Compare]() ### System.Net.Primitives.Tests.IPAddressPerformanceTests.TryFormat(address: 143.24.20.36) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.8599746068092 > 12.056727412948058. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -11.07493886512902 (T) = (0 -12.724523881091898) / Math.Sqrt((0.19911632037081872 / (13)) + (0.05423179428190999 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.1300605723515152 = (11.260037021390588 - 12.724523881091898) / 11.260037021390588 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 22.04 Queue | TigerUbuntu Baseline | [a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6](https://github.com/dotnet/runtime/commit/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6) Compare | [4c013865c31cb154ccf88f7628f2d26758b00d3e](https://github.com/dotnet/runtime/commit/4c013865c31cb154ccf88f7628f2d26758b00d3e) Diff | [Diff](https://github.com/dotnet/runtime/compare/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6...4c013865c31cb154ccf88f7628f2d26758b00d3e) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in SciMark2.kernel Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [benchSparseMult - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 493.53 ms | 635.56 ms | 1.29 | 0.11 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SciMark2.kernel*' ```
### Payloads [Baseline]() [Compare]() ### SciMark2.kernel.benchSparseMult #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 635.5553794615385 > 518.1746533384616. IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -20.80254686105018 (T) = (0 -635066391.063553) / Math.Sqrt((549931303529396.75 / (13)) + (938876866845.3124 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.2708801752398864 = (499705954.53161454 - 635066391.063553) / 499705954.53161454 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 22.04 Queue | TigerUbuntu Baseline | [a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6](https://github.com/dotnet/runtime/commit/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6) Compare | [4c013865c31cb154ccf88f7628f2d26758b00d3e](https://github.com/dotnet/runtime/commit/4c013865c31cb154ccf88f7628f2d26758b00d3e) Diff | [Diff](https://github.com/dotnet/runtime/compare/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6...4c013865c31cb154ccf88f7628f2d26758b00d3e) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [Count - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 36.63 ΞΌs | 41.26 ΞΌs | 1.13 | 0.32 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' ```
### Payloads [Baseline]() [Compare]() ### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 41.25734353618421 > 38.49274458266977. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -3.827698670859208 (T) = (0 -42439.53143484542) / Math.Sqrt((12469277.456513325 / (13)) + (9232831.4236689 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.11601557816957614 = (38027.7231474244 - 42439.53143484542) / 38027.7231474244 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 22.04 Queue | TigerUbuntu Baseline | [a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6](https://github.com/dotnet/runtime/commit/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6) Compare | [4c013865c31cb154ccf88f7628f2d26758b00d3e](https://github.com/dotnet/runtime/commit/4c013865c31cb154ccf88f7628f2d26758b00d3e) Diff | [Diff](https://github.com/dotnet/runtime/compare/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6...4c013865c31cb154ccf88f7628f2d26758b00d3e) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in Microsoft.Extensions.DependencyInjection.GetService Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [Singleton - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 8.76 ns | 9.80 ns | 1.12 | 0.34 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.GetService*' ```
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.DependencyInjection.GetService.Singleton #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.796466543791762 > 8.928597380551507. IsChangePoint: Marked as a change because one of 8/14/2023 9:42:18 AM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -11.328714609799167 (T) = (0 -10.006959380549333) / Math.Sqrt((0.10588144097847962 / (13)) + (0.23316557766638304 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.17596968156133477 = (8.509538585436228 - 10.006959380549333) / 8.509538585436228 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 22.04 Queue | TigerUbuntu Baseline | [a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6](https://github.com/dotnet/runtime/commit/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6) Compare | [4c013865c31cb154ccf88f7628f2d26758b00d3e](https://github.com/dotnet/runtime/commit/4c013865c31cb154ccf88f7628f2d26758b00d3e) Diff | [Diff](https://github.com/dotnet/runtime/compare/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6...4c013865c31cb154ccf88f7628f2d26758b00d3e) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Reflection.Invoke Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [Property_Get_class - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 11.92 ns | 12.93 ns | 1.09 | 0.32 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Invoke*' ```
### Payloads [Baseline]() [Compare]() ### System.Reflection.Invoke.Property_Get_class #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.934025162337663 > 11.931842525878434. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -5.8134313269420215 (T) = (0 -12.43298178735799) / Math.Sqrt((0.13497822992729458 / (13)) + (0.3557251851021524 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.07916202584447414 = (11.52095930880151 - 12.43298178735799) / 11.52095930880151 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 22.04 Queue | TigerUbuntu Baseline | [a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6](https://github.com/dotnet/runtime/commit/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6) Compare | [4c013865c31cb154ccf88f7628f2d26758b00d3e](https://github.com/dotnet/runtime/commit/4c013865c31cb154ccf88f7628f2d26758b00d3e) Diff | [Diff](https://github.com/dotnet/runtime/compare/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6...4c013865c31cb154ccf88f7628f2d26758b00d3e) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in PerfLabTests.CastingPerf2.CastingPerf Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [ScalarValueTypeObj - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 197.37 ΞΌs | 262.86 ΞΌs | 1.33 | 0.03 | False | | | |
  • [FooObjCastIfIsa - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 263.78 ΞΌs | 282.60 ΞΌs | 1.07 | 0.03 | False | | | ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*' ```
### Payloads [Baseline]() [Compare]() ### PerfLabTests.CastingPerf2.CastingPerf.ScalarValueTypeObj #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 262.86413593749995 > 207.03602456933592. IsChangePoint: Marked as a change because one of 8/15/2023 7:48:35 PM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -571.6960913178664 (T) = (0 -262736.10639719927) / Math.Sqrt((36460.566257674574 / (13)) + (259414.6093935372 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.3330097792672661 = (197099.90915567105 - 262736.10639719927) / 197099.90915567105 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### PerfLabTests.CastingPerf2.CastingPerf.FooObjCastIfIsa #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 282.5962487244898 > 276.74016504166667. IsChangePoint: Marked as a change because one of 8/15/2023 7:48:35 PM, 9/2/2023 3:43:46 AM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -74.61168385739026 (T) = (0 -282425.7198429051) / Math.Sqrt((583774.0204949128 / (32)) + (291455.2700233112 / (6))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (6) - 2, .025) and -0.07329441012996876 = (263139.09508641274 - 282425.7198429051) / 263139.09508641274 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 22.04 Queue | TigerUbuntu Baseline | [a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6](https://github.com/dotnet/runtime/commit/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6) Compare | [4c013865c31cb154ccf88f7628f2d26758b00d3e](https://github.com/dotnet/runtime/commit/4c013865c31cb154ccf88f7628f2d26758b00d3e) Diff | [Diff](https://github.com/dotnet/runtime/compare/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6...4c013865c31cb154ccf88f7628f2d26758b00d3e) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Memory.Span<Int32> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [IndexOfAnyFourValues - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 553.59 ns | 623.16 ns | 1.13 | 0.14 | False | | | |
  • [IndexOfAnyFiveValues - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 513.02 ns | 679.64 ns | 1.32 | 0.11 | False | | | |
  • [IndexOfAnyFiveValues - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 32.68 ns | 43.39 ns | 1.33 | 0.36 | False | | | |
  • [IndexOfAnyFourValues - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 36.20 ns | 39.66 ns | 1.10 | 0.07 | True | | | ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span<Int32>*' ```
### Payloads [Baseline]() [Compare]() ### System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 623.1633759593363 > 580.7947470256283. IsChangePoint: Marked as a change because one of 8/19/2023 11:06:43 PM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -17.772546313404245 (T) = (0 -624.3361230397583) / Math.Sqrt((157.56908967091178 / (13)) + (0.8081864634680289 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.11016987457320086 = (562.378909155485 - 624.3361230397583) / 562.378909155485 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.IndexOfAnyFiveValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 679.640959709437 > 538.9842832860049. IsChangePoint: Marked as a change because one of 8/19/2023 11:06:43 PM, 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -525.0385517488116 (T) = (0 -679.307678175634) / Math.Sqrt((0.8420108968969636 / (13)) + (0.8752767012868232 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.323006116403176 = (513.4577004242815 - 679.307678175634) / 513.4577004242815 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.IndexOfAnyFiveValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 43.38838435545172 > 37.49482912108297. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -7.501317357406745 (T) = (0 -44.13679175536036) / Math.Sqrt((11.363373458839362 / (13)) + (0.5015702049745557 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.19148514243077333 = (37.043509972198216 - 44.13679175536036) / 37.043509972198216 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 33) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 39.66469474644949 > 38.01209476983544. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -81.56370161870893 (T) = (0 -39.86311581142425) / Math.Sqrt((0.009244118016176117 / (13)) + (0.030914726401205114 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.09926253540069081 = (36.26350805896773 - 39.86311581142425) / 36.26350805896773 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 22.04 Queue | TigerUbuntu Baseline | [a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6](https://github.com/dotnet/runtime/commit/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6) Compare | [4c013865c31cb154ccf88f7628f2d26758b00d3e](https://github.com/dotnet/runtime/commit/4c013865c31cb154ccf88f7628f2d26758b00d3e) Diff | [Diff](https://github.com/dotnet/runtime/compare/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6...4c013865c31cb154ccf88f7628f2d26758b00d3e) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Tests.Perf_Int32 Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [ToStringHex - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 17.80 ns | 19.69 ns | 1.11 | 0.09 | True | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*' ```
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Int32.ToStringHex(value: 2147483647) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 19.68549505341147 > 18.863384749456166. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -15.032296546459138 (T) = (0 -20.33268632700021) / Math.Sqrt((0.1423980119275122 / (13)) + (0.2898717442461349 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.12488107189761949 = (18.075409778831073 - 20.33268632700021) / 18.075409778831073 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 22.04 Queue | TigerUbuntu Baseline | [a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6](https://github.com/dotnet/runtime/commit/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6) Compare | [4c013865c31cb154ccf88f7628f2d26758b00d3e](https://github.com/dotnet/runtime/commit/4c013865c31cb154ccf88f7628f2d26758b00d3e) Diff | [Diff](https://github.com/dotnet/runtime/compare/a7d4b4b4761b68c85e27d9c4c8dacb86c285ddc6...4c013865c31cb154ccf88f7628f2d26758b00d3e) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Numerics.Tests.Perf_VectorConvert Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [Widen_float - Duration of single invocation]()
  • πŸ“ - [Benchmark Source]()
  • [πŸ“ˆ - ADX Test Multi Config Graph]()
| 878.11 ns | 1.32 ΞΌs | 1.50 | 0.07 | True | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' ```
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_VectorConvert.Widen_float #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.320969514442833 > 926.9380103399895. IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/4/2023 7:24:21 PM falls between 8/27/2023 6:19:41 AM and 9/4/2023 7:24:21 PM. IsRegressionStdDev: Marked as regression because -813.2144417580388 (T) = (0 -1320.5381608067803) / Math.Sqrt((2.2969982946450807 / (13)) + (2.8296111997410787 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.49601818195949177 = (882.7019462271061 - 1320.5381608067803) / 882.7019462271061 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: -
Labels: `area-System.Net`, `os-linux`, `tenet-performance`, `tenet-performance-benchmarks`, `arch-x64`, `untriaged`, `runtime-coreclr`, `needs-area-label`
Milestone: -
karelz commented 1 year ago

The Networking test at the top regressed in continuous runs, so it is worth looking into:

Benchmark source code

Continuous benchmark results: TryFormat - Duration of single invocation image

@MihaZupan it looks related to the perf test you changed recently (ToString - #88671). Can you please take a look at this one as well?

@DrewScoggins @cincuranet what should we do with the rest of the regressions? Is someone going to take a deeper look if perhaps some low-level change in Runtime (e.g. JIT) caused all of them?

cincuranet commented 1 year ago

@karelz @MihaZupan Worth mentioning that it seems the regression to be only on Linux/Ubuntu 22.04/x64/Intel: image

@karelz You can leave it open. I think at least @EgorBo will take a look.

MihaZupan commented 1 year ago

I'm not having much luck reproducing the IPAddress.TryFormat regression on a Ubuntu VM with the cited commit range (do you know which CPU the TigerUbuntu queue uses?). Regardless, there weren't any changes to IPAddress code in that timeframe.

cincuranet commented 1 year ago

It's i7-8700.

wfurt commented 4 months ago

triage: does not seems critical for 9.0, moving to future