dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 13 Improvements on 4/4/2023 4:23:54 PM #15647

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Improvements in System.Text.Json.Serialization.Tests.WriteJson<HashSet<String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SerializeToString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Json.Serialization.Tests.WriteJson(HashSet(String)).SerializeToString(Mode%3a%20SourceGen).html>) 699.66 μs 93.58 μs 0.13 0.14 False

graph_1.png>) Test Report.html>)

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Serialization.Tests.WriteJson<HashSet<String>>.SerializeToString(Mode: SourceGen) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 93.58485543912177 < 153.22729523161573. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/29/2023 9:08:46 AM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 3.7572682080861908 (T) = (0 -103862.81926152333) / Math.Sqrt((3103961463.660788 / (299)) + (1893315293.3787196 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.22304239093760436 = (133678.87520512377 - 103862.81926152333) / 133678.87520512377 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Improvements in System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SerializeToWriter - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String%2c%20String)).SerializeToWriter(Mode%3a%20Reflection).html>) 1.81 ms 296.20 μs 0.16 0.09 False
[SerializeToUtf8Bytes - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String%2c%20String)).SerializeToUtf8Bytes(Mode%3a%20SourceGen).html>) 374.78 μs 244.34 μs 0.65 0.08 False

graph_1.png>) graph_2.png>) Test Report.html>)

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>.SerializeToWriter(Mode: Reflection) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 296.20382683236573 < 425.8405816557103. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/29/2023 6:23:42 PM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 3.928187167590073 (T) = (0 -314429.7122894626) / Math.Sqrt((14027831090.173513 / (299)) + (9755775790.38979 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.18217226115346707 = (384469.36604636983 - 314429.7122894626) / 384469.36604636983 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>.SerializeToUtf8Bytes(Mode: SourceGen) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 244.3417976888021 < 363.2350120192308. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/29/2023 6:23:42 PM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 4.596162080417311 (T) = (0 -258325.37227750497) / Math.Sqrt((11034120291.208994 / (299)) + (5617907144.92138 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.19817082112484716 = (322170.0819617179 - 258325.37227750497) / 322170.0819617179 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Improvements in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[BitArrayGet - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Collections.Tests.Perf_BitArray.BitArrayGet(Size%3a%204).html>) 4.02 μs 186.38 ns 0.05 0.21 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Tests.Perf_BitArray.BitArrayGet(Size: 4) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 186.37801737169838 < 2.290940816124766. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/9/2023 8:37:21 PM, 3/29/2023 3:17:08 PM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 13.085627543241362 (T) = (0 -188.44214150210547) / Math.Sqrt((163050.08334386104 / (299)) + (26.6329830028466 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.6187122352344008 = (494.2255139447035 - 188.44214150210547) / 494.2255139447035 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Improvements in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Trim - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Memory.ReadOnlySpan.Trim(input%3a%20%22%22).html>) 81.73 ns 21.02 ns 0.26 0.28 False
[IndexOfString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Memory.ReadOnlySpan.IndexOfString(input%3a%20%22string1%22%2c%20value%3a%20%22string2%22%2c%20comparisonType%3a%20InvariantCulture).html>) 575.90 ns 356.94 ns 0.62 0.12 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.ReadOnlySpan.Trim(input: "") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 21.01963231608891 < 75.27776504132558. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/22/2023 10:50:22 PM, 3/23/2023 12:23:24 PM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 22.35451007590831 (T) = (0 -22.451858387675696) / Math.Sqrt((1410.2327743145574 / (299)) + (40.27632309840417 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.7063279957717482 = (76.45215772840695 - 22.451858387675696) / 76.45215772840695 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 356.9375823257675 < 549.1588318111429. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/9/2023 8:37:21 PM, 3/22/2023 6:15:16 AM, 3/22/2023 3:40:18 PM, 4/8/2023 3:09:32 AM, 4/8/2023 9:16:19 PM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 14.268538818177548 (T) = (0 -358.62467363577616) / Math.Sqrt((31914.99723635924 / (299)) + (35.40133534430311 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.29506058108157307 = (508.73119591752453 - 358.62467363577616) / 508.73119591752453 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Improvements in System.Net.Primitives.Tests.CredentialCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetCredential_Uri - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_Uri(uriString%3a%20%22http%3a%2f%2fnotfound%22%2c%20uriCount%3a%200).html>) 8.70 μs 2.09 μs 0.24 0.17 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_Uri(uriString: "http://notfound", uriCount: 0) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.0903906946309374 < 4.922657971896628. IsChangePoint: Marked as a change because one of 3/6/2023 4:48:38 PM, 3/29/2023 4:47:17 AM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 6.074515193632856 (T) = (0 -2318.535406220822) / Math.Sqrt((1001852.5514415408 / (299)) + (481700.42691849277 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.2531140737479984 = (3104.269774978383 - 2318.535406220822) / 3104.269774978383 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Improvements in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[WriteBasicUtf16 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted%3a%20False%2c%20SkipValidation%3a%20False%2c%20DataSize%3a%2010).html>) 66.26 μs 15.10 μs 0.23 0.08 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 15.103645029419813 < 40.18542781046049. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/15/2023 6:58:43 PM, 3/29/2023 3:17:08 PM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 6.222406853280491 (T) = (0 -16059.286826439047) / Math.Sqrt((59869296.11032878 / (299)) + (36017482.891367555 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.29803535668349335 = (22877.629207313403 - 16059.286826439047) / 22877.629207313403 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Improvements in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EncodeUtf16 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments%3a%20Url%2c%ef%bf%bd2020%2c512).html>) 89.12 μs 3.92 μs 0.04 0.11 False
[EncodeUtf16 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments%3a%20UnsafeRelaxed%2chello%20%22there%22%2c512).html>) 88.99 μs 3.52 μs 0.04 0.12 False
[EncodeUtf16 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments%3a%20UnsafeRelaxed%2chello%20%22there%22%2c16).html>) 5.37 μs 567.58 ns 0.11 0.16 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: Url,�2020,512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.922184062263675 < 16.009624813420878. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/29/2023 6:23:42 PM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 3.92421670347765 (T) = (0 -4035.2170016453633) / Math.Sqrt((49837434.45761609 / (299)) + (172483.29324756094 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.2870894393134731 = (5660.200906211128 - 4035.2170016453633) / 5660.200906211128 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: UnsafeRelaxed,hello "there",512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.5193168741288208 < 15.691704788362996. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/29/2023 6:23:42 PM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 4.829011050273645 (T) = (0 -3603.7082764971537) / Math.Sqrt((26863825.612024132 / (299)) + (53806.348148662764 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.2882489089869977 = (5063.158064665785 - 3603.7082764971537) / 5063.158064665785 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: UnsafeRelaxed,hello "there",16) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 567.5833634694185 < 1.1700878778273027. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/9/2023 8:37:21 PM, 3/29/2023 6:23:42 PM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 11.00018223175923 (T) = (0 -589.4326769334319) / Math.Sqrt((106376.94341194519 / (299)) + (16264.690791933222 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.34655153516716886 = (902.0339149227689 - 589.4326769334319) / 902.0339149227689 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Improvements in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EscapeDataString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_Uri.EscapeDataString(input%3a%20%22aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.html>) 34.10 μs 5.67 μs 0.17 0.20 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Uri.EscapeDataString(input: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 5.672176155654217 < 32.40080200491592. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/9/2023 8:37:21 PM, 3/29/2023 4:47:17 AM, 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 13.874769143671228 (T) = (0 -5738.496185393197) / Math.Sqrt((199129128.9558876 / (299)) + (12861.965353526997 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.663715922605281 = (17064.430257450284 - 5738.496185393197) / 17064.430257450284 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 2d90e251f9f6c4509de6641a6bc39ac7a35cc032
Compare bc887b38db30f11d7fd35619c35e3043d3b3a6f9
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Improvements in System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[NewJsonSerializerContext - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Json.Serialization.Tests.ColdStartSerialization(SimpleStructWithProperties).NewJsonSerializerContext.html>) 204.04 μs 28.14 μs 0.14 0.09 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ColdStartSerialization&lt;SimpleStructWithProperties&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewJsonSerializerContext ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 28.13720911445094 < 192.8615700443787. IsChangePoint: Marked as a change because one of 3/30/2023 4:24:24 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM. IsImprovementStdDev: Marked as improvement because 60.25215202467182 (T) = (0 -29183.876233453382) / Math.Sqrt((3861742205.2645116 / (299)) + (28414063.02199335 / (36))) is greater than 1.9671134448822736 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (36) - 2, .975) and 0.8842999428220019 = (252237.3535957342 - 29183.876233453382) / 252237.3535957342 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
lewing commented 1 year ago

diff is https://github.com/dotnet/runtime/compare/57ae91cf6fc5672e34705b1a272cf268761d505a...56196485b31275919f772bc36402995c5550715d :(