search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 12 Regressions on 7/22/2023 10:20:55 PM #20240

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d1e28432105391f5e0d5a4573f7858fa9d798a7
Compare d40ca00ca2fb5411ff159b2e84b800e8dfbd681e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
5.03 ns 6.38 ns 1.27 0.17 False
9.45 ns 11.87 ns 1.26 0.14 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt32*'
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_UInt32.TryParseHex(value: "0") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.383962888157527 > 5.279850259921415. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 7/12/2023 2:12:47 AM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -19.362213085790856 (T) = (0 -5.807219041272775) / Math.Sqrt((1.0549475888703852E-05 / (13)) + (0.045387115134848804 / (28))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (28) - 2, .025) and -0.15509609354939188 = (5.027476998409967 - 5.807219041272775) / 5.027476998409967 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_UInt32.Parse(value: "12345") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 11.871047584258637 > 9.915227705406458. IsChangePoint: Marked as a change because one of 7/6/2023 1:53:22 PM, 7/12/2023 2:12:47 AM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -23.503561431415676 (T) = (0 -11.864123087999989) / Math.Sqrt((0.12434837540595652 / (11)) + (0.034270571804483875 / (29))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (29) - 2, .025) and -0.28430149824057316 = (9.237802108191282 - 11.864123087999989) / 9.237802108191282 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d1e28432105391f5e0d5a4573f7858fa9d798a7
Compare d40ca00ca2fb5411ff159b2e84b800e8dfbd681e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
482.48 ns 519.55 ns 1.08 0.28 False
482.31 ns 518.49 ns 1.08 0.29 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_BitOperations.Log2_uint #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 519.5474060981982 > 506.648711118448. IsChangePoint: Marked as a change because one of 6/22/2023 7:55:14 AM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -137.84957498091387 (T) = (0 -519.8787806418002) / Math.Sqrt((0.18767363284293206 / (13)) + (1.5101095442853227 / (26))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (26) - 2, .025) and -0.07689504181832056 = (482.7571494469813 - 519.8787806418002) / 482.7571494469813 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Numerics.Tests.Perf_BitOperations.Log2_ulong #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 518.4894521989587 > 506.555015627358. IsChangePoint: Marked as a change because one of 5/26/2023 6:22:05 AM, 6/5/2023 4:27:01 PM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -184.34450001951376 (T) = (0 -519.2167049137184) / Math.Sqrt((0.280293613405838 / (12)) + (0.4677451828035761 / (29))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (29) - 2, .025) and -0.07590722622442893 = (482.58501500705825 - 519.2167049137184) / 482.58501500705825 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d1e28432105391f5e0d5a4573f7858fa9d798a7
Compare d40ca00ca2fb5411ff159b2e84b800e8dfbd681e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
14.21 ns 16.97 ns 1.19 0.11 False
13.75 ns 17.60 ns 1.28 0.13 False
13.43 ns 17.31 ns 1.29 0.15 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Int32.ParseSpan(value: "2147483647") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.969562583405196 > 14.64774881202224. IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 7/12/2023 2:12:47 AM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -20.067828973051235 (T) = (0 -17.07143165775283) / Math.Sqrt((0.07166939010791126 / (12)) + (0.6051062546950927 / (29))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (29) - 2, .025) and -0.23850994563578107 = (13.783847047743585 - 17.07143165775283) / 13.783847047743585 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Int32.TryParse(value: "2147483647") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 17.59590802485162 > 14.466013717102578. IsChangePoint: Marked as a change because one of 5/2/2023 9:04:55 PM, 7/12/2023 2:12:47 AM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -24.221097947314327 (T) = (0 -17.31771724729266) / Math.Sqrt((0.21967154513214399 / (11)) + (0.04951004898176549 / (29))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (29) - 2, .025) and -0.2593250381055437 = (13.751586543013898 - 17.31771724729266) / 13.751586543013898 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Int32.Parse(value: "2147483647") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 17.3149939103932 > 14.261447181828464. IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 7/12/2023 2:12:47 AM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -21.244420695917146 (T) = (0 -17.440827781120056) / Math.Sqrt((0.13636324189771215 / (11)) + (0.6008767363395293 / (28))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (28) - 2, .025) and -0.2888749474320726 = (13.53182309569194 - 17.440827781120056) / 13.53182309569194 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 3c7027a58ba1b6e3a8861ae628c68293b5cb5f2b
Compare d40ca00ca2fb5411ff159b2e84b800e8dfbd681e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
413.90 ns 477.21 ns 1.15 0.02 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_BigInteger.Divide(arguments: 1024,512 bits) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 477.2114405597018 > 435.9005073342891. IsChangePoint: Marked as a change because one of 6/30/2023 8:34:57 AM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -22.547295931786472 (T) = (0 -470.4655647725425) / Math.Sqrt((16.101163063158513 / (11)) + (119.47140648765273 / (29))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (29) - 2, .025) and -0.12770678386452264 = (417.18784661409114 - 470.4655647725425) / 417.18784661409114 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 3c7027a58ba1b6e3a8861ae628c68293b5cb5f2b
Compare d40ca00ca2fb5411ff159b2e84b800e8dfbd681e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Base64Tests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
740.45 ns 813.62 ns 1.10 0.20 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Base64Tests*'
### Payloads [Baseline]() [Compare]() ### System.Buffers.Text.Tests.Base64Tests.ConvertTryFromBase64Chars(NumberOfBytes: 1000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 813.6157927444027 > 778.233971617166. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -13.747289088534414 (T) = (0 -838.4951578776457) / Math.Sqrt((1.2238268085262383 / (11)) + (1512.1211389635162 / (30))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (30) - 2, .025) and -0.13189698082747447 = (740.7875204903038 - 838.4951578776457) / 740.7875204903038 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 3c7027a58ba1b6e3a8861ae628c68293b5cb5f2b
Compare d40ca00ca2fb5411ff159b2e84b800e8dfbd681e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
515.18 ns 619.48 ns 1.20 0.28 False
510.14 ns 673.22 ns 1.32 0.18 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 619.4830617526976 > 540.9150788396315. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 6/13/2023 5:43:42 PM, 6/19/2023 6:37:24 PM, 6/30/2023 11:17:52 PM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -32.02369739371876 (T) = (0 -618.0190998179264) / Math.Sqrt((65.51921080123542 / (11)) + (101.87355557102401 / (27))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (27) - 2, .025) and -0.19278565354434404 = (518.1308963446133 - 618.0190998179264) / 518.1308963446133 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.Span<Int32>.IndexOfAnyFiveValues(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 673.2167627997409 > 536.3416151484862. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/22/2023 3:21:21 PM, 5/26/2023 9:55:14 AM, 6/8/2023 2:41:32 AM, 6/13/2023 11:57:38 AM, 6/20/2023 3:32:08 AM, 7/1/2023 1:53:48 AM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -10.15118506663606 (T) = (0 -667.1879864852413) / Math.Sqrt((2047.3145160704928 / (13)) + (1070.4418987631036 / (25))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (25) - 2, .025) and -0.2744276626421445 = (523.5196991110712 - 667.1879864852413) / 523.5196991110712 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d1e28432105391f5e0d5a4573f7858fa9d798a7
Compare d40ca00ca2fb5411ff159b2e84b800e8dfbd681e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
46.47 ns 52.92 ns 1.14 0.20 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Version.ToStringL #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 52.91982327327766 > 48.96296831577288. IsChangePoint: Marked as a change because one of 4/25/2023 1:57:48 PM, 5/20/2023 3:57:43 PM, 6/22/2023 7:55:14 AM, 7/12/2023 4:12:15 PM, 7/22/2023 12:54:19 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM. IsRegressionStdDev: Marked as regression because -8.032829843779254 (T) = (0 -51.009634854981776) / Math.Sqrt((0.6987370687468637 / (13)) + (7.2659614858013075 / (27))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (27) - 2, .025) and -0.09827267343012046 = (46.445328276873816 - 51.009634854981776) / 46.445328276873816 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
cincuranet commented 1 year ago

Most of them point to https://github.com/dotnet/runtime/pull/89350. cc @AndyAyersMS