search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/arm64: 23 Improvements on 7/26/2023 2:29:36 PM #20357

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 2c4b4c0831d1d9f505ff5d3055c38bb41646021b
Compare 08acbb7ad7154e9d18227e4b00e4c9f41af1fffa
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Collections.ContainsTrueComparer<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
35.64 μs 32.93 μs 0.92 0.01 False
320.48 μs 288.42 μs 0.90 0.04 False
311.55 μs 282.63 μs 0.91 0.04 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrueComparer&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsTrueComparer<String>.HashSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 32.92608386729692 < 33.802480495916754. IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 5/19/2023 4:22:12 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 41.12034316305407 (T) = (0 -33285.82616768127) / Math.Sqrt((940855.9962857029 / (299)) + (46094.42574070243 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.08751720955943856 = (36478.30569123428 - 33285.82616768127) / 36478.30569123428 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrueComparer<String>.ImmutableSortedSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 288.41616507575753 < 302.79589421326534. IsChangePoint: Marked as a change because one of 5/19/2023 1:23:34 PM, 7/6/2023 2:30:01 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 54.06961006348331 (T) = (0 -290583.27272382175) / Math.Sqrt((218053800.94096765 / (299)) + (4565560.5783203 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.15784392664942673 = (345046.81723390904 - 290583.27272382175) / 345046.81723390904 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrueComparer<String>.SortedSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 282.632038939394 < 294.6882146752451. IsChangePoint: Marked as a change because one of 5/19/2023 1:23:34 PM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 49.85803590235254 (T) = (0 -284675.0290266751) / Math.Sqrt((172901184.21253505 / (299)) + (6116786.463104932 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.1465042733900111 = (333540.07542296627 - 284675.0290266751) / 333540.07542296627 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8aff8937fef4d5db8742a2710371aae366813c44
Compare 08acbb7ad7154e9d18227e4b00e4c9f41af1fffa
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.04 ms 947.98 μs 0.91 0.26 False
1.58 ms 627.42 μs 0.40 0.16 True
1.04 ms 940.89 μs 0.91 0.30 False
1.13 ms 949.30 μs 0.84 0.22 False
1.04 ms 942.35 μs 0.90 0.24 False
1.04 ms 940.54 μs 0.91 0.25 False

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsFalse<String>.Queue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 947.9761666666666 < 1.055800233455128. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 47.4461015059777 (T) = (0 -948286.4173442214) / Math.Sqrt((4953137184.903069 / (299)) + (439172.9104297828 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.1693039967372602 = (1141556.4943368207 - 948286.4173442214) / 1141556.4943368207 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<String>.LinkedList(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 627.4188313333334 < 1.50234890440625. IsChangePoint: Marked as a change because one of 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 260.52939469965304 (T) = (0 -626493.0856562787) / Math.Sqrt((4102995690.5528507 / (299)) + (4799114.494708541 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.6089507256825233 = (1602082.2100993209 - 626493.0856562787) / 1602082.2100993209 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<String>.ICollection(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 940.8919454185522 < 1.031048091568376. IsChangePoint: Marked as a change because one of 3/19/2023 2:30:51 PM, 7/25/2023 10:40:47 PM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 49.31827443148945 (T) = (0 -941893.4780645573) / Math.Sqrt((4376817882.676417 / (299)) + (377252.5628281775 / (17))) is greater than 1.9675476980854363 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (17) - 2, .975) and 0.16700230656420462 = (1130727.5944301942 - 941893.4780645573) / 1130727.5944301942 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<String>.ImmutableArray(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 949.3046306372548 < 0.9949896556009615. IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 49.21515810826831 (T) = (0 -949580.292357015) / Math.Sqrt((4256798735.8915 / (299)) + (1369638.449397613 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.1639802166860332 = (1135834.7150505176 - 949580.292357015) / 1135834.7150505176 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<String>.Stack(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 942.3459485294115 < 1.0746556984455127. IsChangePoint: Marked as a change because one of 4/12/2023 2:59:13 PM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 56.6688793772054 (T) = (0 -944676.0397709864) / Math.Sqrt((3493617667.463929 / (299)) + (2425813.718926716 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.17107239974689964 = (1139636.3681008378 - 944676.0397709864) / 1139636.3681008378 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsFalse<String>.List(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 940.5386110294119 < 1.0823308727306546. IsChangePoint: Marked as a change because one of 3/17/2023 1:30:52 PM, 3/23/2023 11:51:53 PM, 3/28/2023 4:36:21 PM, 4/4/2023 10:11:10 PM, 4/12/2023 6:21:10 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 48.749147212118075 (T) = (0 -941578.601571001) / Math.Sqrt((4625189218.107032 / (299)) + (1931846.4521627487 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.1697263245099614 = (1134058.117662551 - 941578.601571001) / 1134058.117662551 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8aff8937fef4d5db8742a2710371aae366813c44
Compare 08acbb7ad7154e9d18227e4b00e4c9f41af1fffa
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
25.11 μs 19.54 μs 0.78 0.26 False
24.65 μs 19.30 μs 0.78 0.25 False
26.51 μs 20.94 μs 0.79 0.27 False
26.18 μs 20.75 μs 0.79 0.16 False
25.14 μs 19.97 μs 0.79 0.23 False

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>.SerializeToStream(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.540826528415614 < 23.6995833875766. IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 38.6101029385391 (T) = (0 -19375.469716810305) / Math.Sqrt((5384607.487046316 / (299)) + (313253.3251622496 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.2786753866776219 = (26860.957409408293 - 19375.469716810305) / 26860.957409408293 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>.SerializeToWriter(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.300069377572022 < 23.31989752338053. IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 35.551206765556756 (T) = (0 -19265.362663222233) / Math.Sqrt((6403654.861909505 / (299)) + (356621.045363154 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.27838778709186934 = (26697.667138395467 - 19265.362663222233) / 26697.667138395467 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>.SerializeObjectProperty(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 25.134996167494883 (T) = (0 -21114.94152637211) / Math.Sqrt((6723778.926299332 / (299)) + (1245492.88401924 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.27381328137996436 = (29076.463373630122 - 21114.94152637211) / 29076.463373630122 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>.SerializeToString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 20.74583257744411 < 24.874973931297856. IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 40.37303159981057 (T) = (0 -20708.421859619506) / Math.Sqrt((5818974.081644385 / (299)) + (269263.2296525334 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.27081739541034977 = (28399.500659060886 - 20708.421859619506) / 28399.500659060886 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>.SerializeToUtf8Bytes(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 19.972529768595447 < 23.911669657508014. IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 35.13647511374268 (T) = (0 -19666.599167431566) / Math.Sqrt((5928301.340202052 / (299)) + (454591.54085422965 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.2818152773549099 = (27383.761513331905 - 19666.599167431566) / 27383.761513331905 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8aff8937fef4d5db8742a2710371aae366813c44
Compare 08acbb7ad7154e9d18227e4b00e4c9f41af1fffa
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Collections.ContainsTrue<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
566.95 μs 478.29 μs 0.84 0.26 False
793.12 μs 344.30 μs 0.43 0.18 True
573.27 μs 485.23 μs 0.85 0.28 False
611.50 μs 478.43 μs 0.78 0.25 False
569.82 μs 479.33 μs 0.84 0.27 False
567.96 μs 478.29 μs 0.84 0.25 False

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsTrue<String>.ICollection(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 478.2890672348485 < 541.7607052441621. IsChangePoint: Marked as a change because one of 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 55.45577726976159 (T) = (0 -478112.3795302354) / Math.Sqrt((1282146573.366509 / (299)) + (116079.16023201191 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.19380221182122515 = (593046.0074943963 - 478112.3795302354) / 593046.0074943963 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<String>.LinkedList(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 344.30180163043474 < 755.6905446874998. IsChangePoint: Marked as a change because one of 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 397.81568410869295 (T) = (0 -343938.8034488619) / Math.Sqrt((331790665.2372863 / (299)) + (2451909.5005405187 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.5651856686259762 = (791001.5347516419 - 343938.8034488619) / 791001.5347516419 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<String>.ImmutableArray(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 485.23163156565664 < 567.466482970441. IsChangePoint: Marked as a change because one of 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 59.42009699975714 (T) = (0 -485869.313763347) / Math.Sqrt((1034024452.0357552 / (299)) + (2995696.5339054833 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.18930084820852708 = (599321.3545242758 - 485869.313763347) / 599321.3545242758 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<String>.Queue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 478.430046969697 < 582.8774009708333. IsChangePoint: Marked as a change because one of 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 57.73395414105236 (T) = (0 -478449.25862665975) / Math.Sqrt((1191069681.634305 / (299)) + (72240.1263838631 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.19418285849800043 = (593744.205707583 - 478449.25862665975) / 593744.205707583 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<String>.Stack(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 479.3277768939393 < 553.1215570409393. IsChangePoint: Marked as a change because one of 4/11/2023 8:14:56 PM, 5/19/2023 4:22:12 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 61.62221072726938 (T) = (0 -479932.4648279671) / Math.Sqrt((1117836909.5971863 / (299)) + (587499.0092208093 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.19966608860877458 = (599665.2871970619 - 479932.4648279671) / 599665.2871970619 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.ContainsTrue<String>.List(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 478.2895922619048 < 541.7892139229167. IsChangePoint: Marked as a change because one of 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 59.098996254149306 (T) = (0 -478330.430935189) / Math.Sqrt((1157417892.6822734 / (299)) + (181981.5320709987 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.195781898760115 = (594777.0016587962 - 478330.430935189) / 594777.0016587962 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 2c4b4c0831d1d9f505ff5d3055c38bb41646021b
Compare 08acbb7ad7154e9d18227e4b00e4c9f41af1fffa
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
331.91 ms 246.10 ms 0.74 0.26 False
333.79 ms 248.74 ms 0.75 0.28 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'LinqBenchmarks*'
### Payloads [Baseline]() [Compare]() ### LinqBenchmarks.Where01LinqQueryX #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 5/16/2023 7:34:13 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 29.32466491025083 (T) = (0 -249344739.21048534) / Math.Sqrt((5843395228419071 / (299)) + (7993785161269.892 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.34491341942567466 = (380628678.10829014 - 249344739.21048534) / 380628678.10829014 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### LinqBenchmarks.Where01LinqMethodX #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 29.750723555634014 (T) = (0 -248886506.5763164) / Math.Sqrt((6125938080188687 / (299)) + (7950158588656.998 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.3538313253891831 = (385172659.02160776 - 248886506.5763164) / 385172659.02160776 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 2c4b4c0831d1d9f505ff5d3055c38bb41646021b
Compare 08acbb7ad7154e9d18227e4b00e4c9f41af1fffa
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
38.03 ns 30.37 ns 0.80 0.19 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Buffers.Tests.RentReturnArrayPoolTests<Object>.SingleSerial(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: True) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 5/6/2023 4:44:37 AM, 5/8/2023 3:52:55 PM, 7/26/2023 12:02:55 AM, 8/2/2023 4:26:52 PM falls between 7/24/2023 8:03:13 PM and 8/2/2023 4:26:52 PM. IsImprovementStdDev: Marked as improvement because 11.103921901160042 (T) = (0 -31.70843730906188) / Math.Sqrt((3.478669154745382 / (299)) + (5.060558884786034 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.16703637889432038 = (38.067013379254135 - 31.70843730906188) / 38.067013379254135 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)