search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 7 Improvements on 4/20/2023 12:14:19 AM #17060

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 33b18692fdf2ef438de3da38bc419c6616a746c5
Compare 7222061f18b8342a0bd4edbd37df318fd43c47d3
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Ctor - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_Uri.Ctor(input%3a%20%22https%3a%2f%2fa.much.longer.domain.name%22).html>) 410.41 ns 364.89 ns 0.89 0.08 False
[Ctor - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_Uri.Ctor(input%3a%20%22http%3a%2f%2fh%c3%b6st.with.%c3%bcnicode%22).html>) 1.25 μs 1.06 μs 0.84 0.27 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Uri.Ctor(input: "https://a.much.longer.domain.name") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 364.89463643789446 < 388.9025260008287. IsChangePoint: Marked as a change because one of 3/2/2023 11:15:04 AM, 4/19/2023 8:38:27 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsImprovementStdDev: Marked as improvement because 16.933554056252213 (T) = (0 -369.5412789040005) / Math.Sqrt((166.54410642862229 / (299)) + (46.006277151429636 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.06827500803490213 = (396.6205501524675 - 369.5412789040005) / 396.6205501524675 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Uri.Ctor(input: "http://höst.with.ünicode") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.0593218531341109 < 1.1614960796346543. IsChangePoint: Marked as a change because one of 3/11/2023 1:35:17 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsImprovementStdDev: Marked as improvement because 22.1725352857483 (T) = (0 -1115.4114076652997) / Math.Sqrt((2821.474482401777 / (299)) + (1275.239985697304 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.11865814343699432 = (1265.5831552300283 - 1115.4114076652997) / 1265.5831552300283 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 33b18692fdf2ef438de3da38bc419c6616a746c5
Compare 7222061f18b8342a0bd4edbd37df318fd43c47d3
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToLowerInvariant - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_String.ToLowerInvariant(s%3a%20%22This%20is%20a%20much%20longer%20piece%20of%20text%20that%20might%20benefit%20more%20from%20vectorization.%22).html>) 278.63 ns 214.46 ns 0.77 0.18 False
[Concat_str_str - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_String.Concat_str_str(size%3a%201000).html>) 530.52 ns 477.29 ns 0.90 0.03 False
[ToLower - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_String.ToLower(s%3a%20%22This%20is%20a%20much%20longer%20piece%20of%20text%20that%20might%20benefit%20more%20from%20vectorization.%22).html>) 320.30 ns 274.97 ns 0.86 0.23 False
[ToUpperInvariant - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_String.ToUpperInvariant(s%3a%20%22This%20is%20a%20much%20longer%20piece%20of%20text%20that%20might%20benefit%20more%20from%20vectorization.%22).html>) 265.13 ns 222.61 ns 0.84 0.22 False

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 214.45963642370026 < 276.3277209058668. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 4/19/2023 8:38:27 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsImprovementStdDev: Marked as improvement because 9.044792719323247 (T) = (0 -219.43259829069942) / Math.Sqrt((2431.116231480251 / (299)) + (535.3116094325152 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.1876475979139465 = (270.119959917906 - 219.43259829069942) / 270.119959917906 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Concat_str_str(size: 1000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 477.29353310678613 < 497.5502179047296. IsChangePoint: Marked as a change because one of 2/14/2023 11:21:41 AM, 3/10/2023 8:06:53 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsImprovementStdDev: Marked as improvement because 30.175451051270922 (T) = (0 -480.50584399348145) / Math.Sqrt((952.4446921327024 / (299)) + (46.39243047034044 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.11685410607929513 = (544.0843322729921 - 480.50584399348145) / 544.0843322729921 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 274.9684342540415 < 317.8845530456082. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 4/19/2023 8:38:27 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsImprovementStdDev: Marked as improvement because 8.029164269045607 (T) = (0 -272.07402373560404) / Math.Sqrt((2994.762040272043 / (299)) + (222.91770603186137 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.11697145702200322 = (308.11464238521626 - 272.07402373560404) / 308.11464238521626 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 222.6112856440589 < 254.213208486426. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 4/19/2023 8:38:27 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsImprovementStdDev: Marked as improvement because 15.159819568783325 (T) = (0 -216.05769560995213) / Math.Sqrt((2562.596638620794 / (299)) + (91.95993219266492 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.1992022751567803 = (269.8030837341002 - 216.05769560995213) / 269.8030837341002 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 33b18692fdf2ef438de3da38bc419c6616a746c5
Compare 7222061f18b8342a0bd4edbd37df318fd43c47d3
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Tests.Perf_DateTime

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_DateTime.ToString(format%3a%20%22s%22).html>) 487.85 ns 162.65 ns 0.33 0.13 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_DateTime*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_DateTime.ToString(format: "s") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 162.6512292283251 < 463.8037888581452. IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 10:53:42 PM, 4/19/2023 8:38:27 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsImprovementStdDev: Marked as improvement because 85.43025550893505 (T) = (0 -159.2572811785712) / Math.Sqrt((1937.088128457099 / (299)) + (18.08483844104 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.5911456201290549 = (389.520790333323 - 159.2572811785712) / 389.520790333323 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
lewing commented 1 year ago

@stephentoub https://github.com/dotnet/runtime/pull/84963