search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 16 Regressions on 8/24/2023 4:35:49 AM #21090

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.Concurrent.IsEmpty<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.41 ns 3.86 ns 1.60 0.81 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Concurrent.IsEmpty&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.Concurrent.IsEmpty<Int32>.Stack(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.861611957128495 > 2.6469558077623283. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -6.392589768722784 (T) = (0 -4.1382287469776085) / Math.Sqrt((0.3111417351054016 / (12)) + (0.7937080550762446 / (23))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (23) - 2, .025) and -0.6122937894927502 = (2.566671641326332 - 4.1382287469776085) / 2.566671641326332 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
62.99 ns 74.19 ns 1.18 0.31 True
1.09 ΞΌs 1.46 ΞΌs 1.35 0.15 True
92.77 ns 122.12 ns 1.32 0.16 True
66.69 ns 73.98 ns 1.11 0.26 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'
### Payloads [Baseline]() [Compare]() ### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 74.19019023019614 > 66.35833830693676. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -20.15765643745673 (T) = (0 -74.8979565060387) / Math.Sqrt((2.7554193588427256 / (12)) + (2.9365175561199393 / (20))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (20) - 2, .025) and -0.1977882017042211 = (62.53021727837474 - 74.8979565060387) / 62.53021727837474 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.4636819273835395 > 1.1427487800417178. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -22.22163864710237 (T) = (0 -1471.6746344016606) / Math.Sqrt((2892.14426299914 / (12)) + (830.6577684200965 / (23))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (23) - 2, .025) and -0.3357637455621169 = (1101.74769998893 - 1471.6746344016606) / 1101.74769998893 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 122.12075541171478 > 98.91790791988419. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -33.37521579083975 (T) = (0 -120.22393484160006) / Math.Sqrt((5.404795670691668 / (12)) + (2.926966700013869 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.2691115783275086 = (94.73078403400628 - 120.22393484160006) / 94.73078403400628 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 73.97898478135475 > 70.63656918202317. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -8.764373447311673 (T) = (0 -76.02063619074711) / Math.Sqrt((6.098089242096664 / (11)) + (7.428082266339512 / (23))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (23) - 2, .025) and -0.12105984673513098 = (67.81139866184883 - 76.02063619074711) / 67.81139866184883 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare fd6d5fd0a330a8a6fd286a3d924f4e832aaeb479
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_VectorOf<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.23 ns 2.94 ns 2.39 0.94 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_VectorOf<Int32>.EqualsAnyBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.9403112171681483 > 1.3720362405156785. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -3.818066772875991 (T) = (0 -2.4704894469930814) / Math.Sqrt((0.4933504559395109 / (11)) + (1.2622677368750461 / (22))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (22) - 2, .025) and -0.9767909208459592 = (1.249747467443773 - 2.4704894469930814) / 1.249747467443773 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.Sort<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
22.00 ΞΌs 28.53 ΞΌs 1.30 0.06 True
21.77 ΞΌs 27.08 ΞΌs 1.24 0.22 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.Sort<Int32>.List(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 28.53384282857143 > 22.968971486. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -42.51277491499385 (T) = (0 -27757.060169654396) / Math.Sqrt((49361.65784437044 / (12)) + (345420.2456902875 / (23))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (23) - 2, .025) and -0.26878991826635795 = (21876.79754547619 - 27757.060169654396) / 21876.79754547619 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Sort<Int32>.Array(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.076720000000005 > 22.932545978999997. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -25.255063366577538 (T) = (0 -27787.990339786873) / Math.Sqrt((334422.5434481456 / (12)) + (501850.58842491015 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.25723183827836693 = (22102.518798632478 - 27787.990339786873) / 22102.518798632478 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.23 ΞΌs 43.56 ΞΌs 1.20 0.06 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.CtorFromCollection<Int32>.SortedSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 43.55832150649953 > 38.42366544996278. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -26.957943920350083 (T) = (0 -42024.74143099952) / Math.Sqrt((115203.31936506095 / (12)) + (753464.9623224655 / (23))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (23) - 2, .025) and -0.15210755366728784 = (36476.4047394968 - 42024.74143099952) / 36476.4047394968 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.06 ms 1.32 ms 1.25 0.46 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_File*'
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_File.CopyTo(size: 1048576) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.3173375052083336 > 1.1969065450111607. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -11.222246051940422 (T) = (0 -1294247.4205220372) / Math.Sqrt((1579258639.378689 / (11)) + (1421291062.096205 / (23))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (23) - 2, .025) and -0.1418890154049397 = (1133426.6317143508 - 1294247.4205220372) / 1133426.6317143508 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Strings

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
83.60 ms 92.01 ms 1.10 0.04 True
84.01 ms 92.71 ms 1.10 0.04 True
84.08 ms 91.62 ms 1.09 0.14 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Strings*'
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: False, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 92.0095 > 88.76197750000001. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -12.039930077713231 (T) = (0 -92455568.61259954) / Math.Sqrt((3331807955050.6636 / (12)) + (980457261432.5946 / (23))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (23) - 2, .025) and -0.07956144280426741 = (85641784.66066468 - 92455568.61259954) / 85641784.66066468 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 92.71230773076923 > 88.6788328125. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -12.470511233231342 (T) = (0 -93402933.83180153) / Math.Sqrt((3463775627115.541 / (11)) + (1708061448963.1633 / (22))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (22) - 2, .025) and -0.09128448246880973 = (85589903.76230435 - 93402933.83180153) / 85589903.76230435 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: True, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 91.62105358928572 > 88.42046502625. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -6.803314096725538 (T) = (0 -92435239.93820761) / Math.Sqrt((14174569450025.408 / (12)) + (1302847976711.2366 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.08929236462909108 = (84858062.8486111 - 92435239.93820761) / 84858062.8486111 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
9.86 ns 12.75 ns 1.29 0.60 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_Matrix4x4.GetDeterminantBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.747941045003538 > 11.492634786487347. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -3.9737208234552233 (T) = (0 -12.581143091852198) / Math.Sqrt((2.0133519397487265 / (12)) + (0.7145649903926837 / (23))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (23) - 2, .025) and -0.16393246066513245 = (10.809169360791492 - 12.581143091852198) / 10.809169360791492 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Vector3

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
3.72 ns 5.63 ns 1.51 0.86 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*'
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_Vector3.MultiplyByScalarOperatorBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 5.634527270083949 > 3.9731195754643127. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -5.107393801680558 (T) = (0 -6.237835992936999) / Math.Sqrt((1.336142570523421 / (12)) + (2.3257206290910823 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.6167242676238925 = (3.8583177835913705 - 6.237835992936999) / 3.8583177835913705 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.IO.Tests.Perf_FileStream

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
968.89 ΞΌs 1.11 ms 1.14 0.40 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_FileStream*'
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_FileStream.CopyToFile(fileSize: 1048576, options: None) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.1064178508928575 > 1022.7532392923177. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/30/2023 5:40:03 PM falls between 8/22/2023 3:00:25 AM and 8/30/2023 5:40:03 PM. IsRegressionStdDev: Marked as regression because -9.428413660202333 (T) = (0 -1079456.516863273) / Math.Sqrt((1360691371.6221502 / (12)) + (1239646534.7649739 / (23))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (23) - 2, .025) and -0.127358760299521 = (957509.3172438549 - 1079456.516863273) / 957509.3172438549 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

probably v8 version change https://github.com/dotnet/runtime/pull/90961