search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 17 Improvements on 4/4/2023 4:23:54 PM #15753

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Buffers.Tests.ReadOnlySequenceTests<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IterateGetPositionMemory - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Buffers.Tests.ReadOnlySequenceTests(Byte).IterateGetPositionMemory.html>) 607.55 ns 519.91 ns 0.86 0.36 False
[SliceMemory - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Buffers.Tests.ReadOnlySequenceTests(Byte).SliceMemory.html>) 98.31 ns 89.22 ns 0.91 0.44 False
[FirstSpanMemory - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Buffers.Tests.ReadOnlySequenceTests(Byte).FirstSpanMemory.html>) 44.95 ns 41.14 ns 0.92 0.46 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Tests.ReadOnlySequenceTests<Byte>.IterateGetPositionMemory ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 519.9104089785693 < 574.7026480137283. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 6.893641555396788 (T) = (0 -546.8364488959405) / Math.Sqrt((702.8960069636994 / (299)) + (1530.0709238959043 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.07881914978419967 = (593.6255066177678 - 546.8364488959405) / 593.6255066177678 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### System.Buffers.Tests.ReadOnlySequenceTests<Byte>.SliceMemory ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 89.22252975670435 < 95.9808274826169. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 7.88834978416433 (T) = (0 -91.61557770526596) / Math.Sqrt((22.433840329743322 / (299)) + (24.190061867956214 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.06926237084155705 = (98.43330153966505 - 91.61557770526596) / 98.43330153966505 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### System.Buffers.Tests.ReadOnlySequenceTests<Byte>.FirstSpanMemory ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 41.13831123824913 < 41.98885616956758. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 7.0483740960520915 (T) = (0 -41.41313154725547) / Math.Sqrt((5.863747040459582 / (299)) + (8.280565359936299 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.07931418259884557 = (44.98074236024779 - 41.41313154725547) / 44.98074236024779 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableList - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Collections.ContainsFalse(Int32).ImmutableList(Size%3a%20512).html>) 9.33 ms 8.22 ms 0.88 0.28 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsFalse<Int32>.ImmutableList(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.218421883928572 < 8.539648318072915. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 17.78228965791915 (T) = (0 -8298402.473631477) / Math.Sqrt((153797935077.26236 / (299)) + (83159528990.84027 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.10121717705887615 = (9232933.9878529 - 8298402.473631477) / 9232933.9878529 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GenericClassGenericStaticMethod - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod.html>) 664.23 μs 481.31 μs 0.72 0.24 True
[ForeachOverList100Elements - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.ForeachOverList100Elements.html>) 149.07 ms 126.39 ms 0.85 0.31 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'
### Payloads [Baseline]() [Compare]() ### Histogram #### PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 481.3068178904429 < 631.0821894890873. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 47.62286973210246 (T) = (0 -493284.3144356327) / Math.Sqrt((683950515.575239 / (299)) + (453053783.2325094 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.2690219106486562 = (674827.7706563872 - 493284.3144356327) / 674827.7706563872 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### PerfLabTests.LowLevelPerf.ForeachOverList100Elements ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 126.39487495833333 < 138.48188012666665. IsChangePoint: Marked as a change because one of 3/8/2023 4:46:51 PM, 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 25.61018617269349 (T) = (0 -127062125.00838035) / Math.Sqrt((65138741843427.74 / (299)) + (8662171567312.181 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.11933340221904905 = (144279487.07097965 - 127062125.00838035) / 144279487.07097965 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Trim - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Memory.ReadOnlySpan.Trim(input%3a%20%22abcdefg%22).html>) 20.52 ns 15.10 ns 0.74 0.62 False
[Trim - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Memory.ReadOnlySpan.Trim(input%3a%20%22%22).html>) 11.12 ns 3.45 ns 0.31 0.73 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.ReadOnlySpan.Trim(input: "abcdefg") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.098116631485276 < 20.092520697863034. IsChangePoint: Marked as a change because one of 3/29/2023 6:23:42 PM, 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 15.19100824834803 (T) = (0 -13.944687709191356) / Math.Sqrt((3.3794987665825778 / (299)) + (2.6165034269691843 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.23994821157829938 = (18.347023086608942 - 13.944687709191356) / 18.347023086608942 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### System.Memory.ReadOnlySpan.Trim(input: "") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.4490915263543545 < 10.651649153937834. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 25.15629854216386 (T) = (0 -4.666230960612478) / Math.Sqrt((2.6026666044475544 / (299)) + (2.805458067763238 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.6134233011390016 = (12.070647233423443 - 4.666230960612478) / 12.070647233423443 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[HeapSort - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Collections.Tests.Perf_PriorityQueue(Int32%2c%20Int32).HeapSort(Size%3a%201000).html>) 341.70 μs 302.67 μs 0.89 0.33 False
[HeapSort - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Collections.Tests.Perf_PriorityQueue(Int32%2c%20Int32).HeapSort(Size%3a%20100).html>) 22.29 μs 20.47 μs 0.92 0.36 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.HeapSort(Size: 1000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 302.6687088235294 < 323.64812552933716. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 20.98768045269195 (T) = (0 -309566.53053637425) / Math.Sqrt((241226231.31334615 / (299)) + (75384989.90018791 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.10260633699210307 = (344961.796920612 - 309566.53053637425) / 344961.796920612 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.HeapSort(Size: 100) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.468731128719277 < 21.586615721696344. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 9.09799255511979 (T) = (0 -20960.055130973386) / Math.Sqrt((1063856.1884160698 / (299)) + (1700304.657795189 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.08807793291230191 = (22984.48067816928 - 20960.055130973386) / 22984.48067816928 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Memory.ReadOnlySequence

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Slice_Repeat_StartPosition_And_EndPosition - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Memory.ReadOnlySequence.Slice_Repeat_StartPosition_And_EndPosition(Segment%3a%20Multiple).html>) 173.25 ns 153.35 ns 0.89 0.20 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySequence*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.ReadOnlySequence.Slice_Repeat_StartPosition_And_EndPosition(Segment: Multiple) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 153.35392896097744 < 161.88788121285165. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 17.587168402270457 (T) = (0 -155.9234687070134) / Math.Sqrt((28.679565382277175 / (299)) + (23.674083702238747 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.08821537921825962 = (171.00910144034748 - 155.9234687070134) / 171.00910144034748 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count00ForX - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/LinqBenchmarks.Count00ForX.html>) 1.27 secs 1.15 secs 0.91 0.28 False
[Where00ForX - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/LinqBenchmarks.Where00ForX.html>) 1.65 secs 1.55 secs 0.94 0.25 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'LinqBenchmarks*'
### Payloads [Baseline]() [Compare]() ### Histogram #### LinqBenchmarks.Count00ForX ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.1537936924615386 < 1.2098932875376605. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 9.803183011658717 (T) = (0 -1157884852.8326705) / Math.Sqrt((5229861548877935 / (299)) + (6739861432254647 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.10684688589567429 = (1296401293.963828 - 1157884852.8326705) / 1296401293.963828 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### LinqBenchmarks.Where00ForX ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.5516488569285714 < 1.563918025. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 14.53907673048894 (T) = (0 -1537485551.933403) / Math.Sqrt((7500740835256007 / (299)) + (3511476805840458 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.09386286616967109 = (1696747097.6874146 - 1537485551.933403) / 1696747097.6874146 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Collections.Tests.DictionarySequentialKeys

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryGetValue_17_Int_Int - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Collections.Tests.DictionarySequentialKeys.TryGetValue_17_Int_Int.html>) 41.31 ns 35.31 ns 0.85 0.59 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.DictionarySequentialKeys*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Tests.DictionarySequentialKeys.TryGetValue_17_Int_Int ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 35.308983290640995 < 38.24582306418471. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 8.394371650202222 (T) = (0 -35.760332310907394) / Math.Sqrt((6.293697970438516 / (299)) + (5.761053315773744 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.0898246404727795 = (39.28949727828565 - 35.760332310907394) / 39.28949727828565 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Collections.IterateForEach<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[List - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Collections.IterateForEach(Int32).List(Size%3a%20512).html>) 9.68 μs 8.51 μs 0.88 0.33 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.IterateForEach<Int32>.List(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 8.510658714138916 < 8.828153959383917. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 6.9545901097765075 (T) = (0 -8752.64536412411) / Math.Sqrt((222681.29890112265 / (299)) + (184654.08106400806 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.05740198839030869 = (9285.660754977684 - 8752.64536412411) / 9285.660754977684 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[NewJsonSerializerContext - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Text.Json.Serialization.Tests.ColdStartSerialization(SimpleStructWithProperties).NewJsonSerializerContext.html>) 58.36 μs 6.94 μs 0.12 0.03 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ColdStartSerialization&lt;SimpleStructWithProperties&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewJsonSerializerContext ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 6.939579643737673 < 57.00305782112419. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 216.46148693729037 (T) = (0 -7106.982594806703) / Math.Sqrt((22208437.1648675 / (299)) + (31018.20509806569 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.8930194176422265 = (66432.45379838119 - 7106.982594806703) / 66432.45379838119 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[BitArraySetAll - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size%3a%204).html>) 27.29 ns 19.87 ns 0.73 0.78 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 4) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.867860195305386 < 25.0439281567451. IsChangePoint: Marked as a change because one of 3/1/2023 11:57:24 AM, 3/30/2023 4:24:24 AM, 4/11/2023 5:52:15 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 5:52:15 AM. IsImprovementStdDev: Marked as improvement because 3.461710880581645 (T) = (0 -20.579277390840712) / Math.Sqrt((17.60948468049285 / (299)) + (4.782200160568731 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.0686032043321287 = (22.09506999225185 - 20.579277390840712) / 22.09506999225185 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
lewing commented 1 year ago

range is https://github.com/dotnet/runtime/compare/57ae91cf6fc5672e34705b1a272cf268761d505a...56196485b31275919f772bc36402995c5550715d