search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 19 Regressions on 8/24/2023 4:35:49 AM #21000

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.10 μs 43.17 μs 1.20 0.07 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.CtorFromCollection<Int32>.SortedSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 43.1719758079409 > 38.163746741382894. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -22.80925326217689 (T) = (0 -41867.081061811965) / Math.Sqrt((204146.94537839672 / (17)) + (578245.4328273193 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.14389154276042923 = (36600.56875740045 - 41867.081061811965) / 36600.56875740045 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Vector3

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.70 ns 7.86 ns 1.67 0.88 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*'
### Payloads [Baseline]() [Compare]() ### System.Numerics.Tests.Perf_Vector3.MultiplyByScalarOperatorBenchmark #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.8597629029816565 > 4.334578800067891. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -4.613637116519953 (T) = (0 -6.369621971481842) / Math.Sqrt((0.9505145280729234 / (17)) + (3.3889685698609444 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.6539561714199584 = (3.8511431448714744 - 6.369621971481842) / 3.8511431448714744 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.Sort<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
23.13 μs 27.21 μs 1.18 0.22 True
22.19 μs 29.00 μs 1.31 0.02 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.Sort<Int32>.Array(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.213200093333334 > 23.104136986. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -22.008059098648868 (T) = (0 -27785.06972930966) / Math.Sqrt((361817.1139864115 / (17)) + (577246.7715112759 / (13))) is less than -2.048407141790485 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (13) - 2, .025) and -0.25471529043377716 = (22144.521503124324 - 27785.06972930966) / 22144.521503124324 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Sort<Int32>.List(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 28.999160040000003 > 22.74871518769231. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -40.60601830044777 (T) = (0 -27660.542373888013) / Math.Sqrt((52013.48850820273 / (17)) + (246531.16020695615 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.26751052179338 = (21822.731960245634 - 27660.542373888013) / 21822.731960245634 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Strings

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
83.18 ms 95.64 ms 1.15 0.17 False
85.19 ms 94.36 ms 1.11 0.12 True
84.58 ms 92.46 ms 1.09 0.05 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Strings*'
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: True, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 95.64451926923077 > 87.92122282125. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -8.394708410578174 (T) = (0 -92476412.0726261) / Math.Sqrt((11201716518812.012 / (17)) + (1720129072205.903 / (13))) is less than -2.048407141790485 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (13) - 2, .025) and -0.08784044891372404 = (85009168.54578216 - 92476412.0726261) / 85009168.54578216 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 94.36498210714285 > 89.72690123250001. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -9.770192239192358 (T) = (0 -93387641.6256639) / Math.Sqrt((6448035109321.386 / (17)) + (1374630228641.6484 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.07792594228017545 = (86636417.18105204 - 93387641.6256639) / 86636417.18105204 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: False, Escaped: AllEscaped) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 92.46061530769231 > 88.960321846875. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -14.767557879046036 (T) = (0 -92569258.26663396) / Math.Sqrt((2924448878179.232 / (17)) + (931936194592.3062 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.08450925226508292 = (85355895.3723039 - 92569258.26663396) / 85355895.3723039 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.Concurrent.IsEmpty<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.18 ns 3.44 ns 2.91 0.81 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Concurrent.IsEmpty&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.Concurrent.IsEmpty<Int32>.Stack(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.4356194051881976 > 2.7875160771930245. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -5.481037498625704 (T) = (0 -4.2528743632135875) / Math.Sqrt((0.24017425720077068 / (17)) + (1.15827115549443 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.6697362508881172 = (2.5470336174060564 - 4.2528743632135875) / 2.5470336174060564 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
519.87 μs 617.86 μs 1.19 0.26 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile*'
### Payloads [Baseline]() [Compare]() ### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity: 1000000) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 617.8582540064101 > 594.3814894381011. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -8.379201195316528 (T) = (0 -609456.2219034954) / Math.Sqrt((364780748.0924335 / (17)) + (191218042.63226727 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.08869955202315292 = (559802.0324073159 - 609456.2219034954) / 559802.0324073159 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.14 ms 1.31 ms 1.15 0.41 True
1.13 ms 1.24 ms 1.10 0.36 True
6.48 μs 7.61 μs 1.17 0.37 True
505.27 μs 542.64 μs 1.07 0.40 False

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_File*'
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_File.CopyTo(size: 1048576) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.3118902781249997 > 1.2034362204627407. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -12.482765264490022 (T) = (0 -1303248.3924806223) / Math.Sqrt((2370899254.2293286 / (16)) + (983987190.3933704 / (14))) is less than -2.048407141790485 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (14) - 2, .025) and -0.1649192535004412 = (1118745.688651397 - 1303248.3924806223) / 1118745.688651397 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_File.CopyToOverwrite(size: 1048576) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.24111367032967 > 1.1375824832374657. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -8.338993559215428 (T) = (0 -1234984.2180619726) / Math.Sqrt((1526513881.1767826 / (17)) + (1677316288.9604676 / (11))) is less than -2.055529438636913 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (11) - 2, .025) and -0.11744531849607494 = (1105185.3702551536 - 1234984.2180619726) / 1105185.3702551536 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_File.Delete #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.605263157894736 > 7.19332894736842. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -11.578594746774199 (T) = (0 -7779.200452110669) / Math.Sqrt((93642.87270423655 / (17)) + (26555.5785864936 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.14689765801548463 = (6782.820069204152 - 7779.200452110669) / 6782.820069204152 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.IO.Tests.Perf_File.WriteAllBytes(size: 1048576) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 542.6423853448276 > 531.7637603830644. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -4.5197626096750865 (T) = (0 -536227.060065516) / Math.Sqrt((330182703.61230916 / (17)) + (257686880.13013005 / (13))) is less than -2.048407141790485 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (13) - 2, .025) and -0.055746335255287514 = (507912.7837425571 - 536227.060065516) / 507912.7837425571 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
93.57 ns 118.50 ns 1.27 0.19 True
60.69 ns 77.62 ns 1.28 0.33 True
67.85 ns 76.46 ns 1.13 0.29 True
1.08 μs 1.47 μs 1.36 0.19 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'
### Payloads [Baseline]() [Compare]() ### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 118.49993139285961 > 101.5919688473991. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -33.94753220980857 (T) = (0 -120.47379095303954) / Math.Sqrt((5.588417113548103 / (17)) + (3.0448272174934132 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.26303539161049355 = (95.38433503389298 - 120.47379095303954) / 95.38433503389298 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 77.62409447149443 > 67.28852865317626. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -16.73876839739948 (T) = (0 -74.50352418625192) / Math.Sqrt((4.129839314673953 / (17)) + (2.5982493706633787 / (11))) is less than -2.055529438636913 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (11) - 2, .025) and -0.184155612609908 = (62.91700465114068 - 74.50352418625192) / 62.91700465114068 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 76.45587313309461 > 70.28233372406801. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -8.020928702605353 (T) = (0 -76.20482902653085) / Math.Sqrt((6.268879597300544 / (16)) + (10.97948670780107 / (14))) is less than -2.048407141790485 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (14) - 2, .025) and -0.1288525499880191 = (67.50645071168917 - 76.20482902653085) / 67.50645071168917 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.470759332299609 > 1.143467370276492. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -25.87187742816119 (T) = (0 -1476.8385721089383) / Math.Sqrt((2038.4358863577665 / (17)) + (1323.9828336982512 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.3450958373582731 = (1097.943009777953 - 1476.8385721089383) / 1097.943009777953 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 3a9412266ac60dc5e6156909ab2ddb9e89eb0df8
Compare e3368e865281f97585316af5702e7bac5d4c1a3a
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.08 μs 8.97 μs 1.11 0.21 True
67.33 ns 74.27 ns 1.10 0.33 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'
### Payloads [Baseline]() [Compare]() ### System.Collections.Tests.Perf_BitArray.BitArrayGet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 8.974231464737796 > 8.536990757640698. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -6.453458831014721 (T) = (0 -9035.144609790756) / Math.Sqrt((147708.09659936762 / (17)) + (58978.54691620554 / (14))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (14) - 2, .025) and -0.0882922280786489 = (8302.13096875833 - 9035.144609790756) / 8302.13096875833 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Tests.Perf_BitArray.BitArrayGet(Size: 4) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 74.27173603162552 > 70.71071950156339. IsChangePoint: Marked as a change because one of 8/22/2023 4:48:44 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM. IsRegressionStdDev: Marked as regression because -7.255595531722063 (T) = (0 -75.43236349867215) / Math.Sqrt((6.291669597320374 / (16)) + (8.055518907586796 / (15))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (15) - 2, .025) and -0.10225914416153926 = (68.43432771524127 - 75.43236349867215) / 68.43432771524127 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

range is https://github.com/dotnet/runtime/compare/7cffd46ec956fbbb0c1e4b0fbe0a3a1b335ef006...c4f8c3b27a8fb2e452a57a6516050c8c2ee677f4

might be v8 version change https://github.com/dotnet/runtime/pull/90961