dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 7 Improvements on 3/14/2023 8:42:09 PM #14328

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 0fd457128fe50dc80ec250104f11c1c1433ed4c5
Compare 002875823f4aa37d557bf8d4fb5053d0479f0c85
Diff Diff
Configs CompliationMode:wasm, RunKind:micro

Improvements in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Log2_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.Log2_ulong.html>) 57.40 μs 11.00 μs 0.19 0.09 False
[PopCount_uint - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.PopCount_uint.html>) 45.44 μs 2.44 μs 0.05 0.12 False
[PopCount_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.PopCount_ulong.html>) 86.48 μs 10.99 μs 0.13 0.10 False
[TrailingZeroCount_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_ulong.html>) 7.21 μs 2.42 μs 0.34 0.18 False
[Log2_uint - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.Log2_uint.html>) 50.04 μs 11.05 μs 0.22 0.11 False
[LeadingZeroCount_uint - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_uint.html>) 55.28 μs 2.41 μs 0.04 0.14 False
[LeadingZeroCount_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong.html>) 55.85 μs 2.41 μs 0.04 0.12 False

graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BitOperations.Log2_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 11.00332384817018 < 54.543177629924244. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/14/2023 5:06:59 PM, 3/21/2023 9:37:45 AM falls between 3/12/2023 3:57:11 PM and 3/21/2023 9:37:45 AM. IsImprovementStdDev: Marked as improvement because 9.543490728339696 (T) = (0 -20877.563773391328) / Math.Sqrt((76635413.10293184 / (299)) + (548836209.4136646 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.6525976569510107 = (60096.20888033922 - 20877.563773391328) / 60096.20888033922 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.PopCount_uint ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 2.443394446973459 < 43.20560543115767. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/14/2023 5:06:59 PM, 3/21/2023 9:37:45 AM falls between 3/12/2023 3:57:11 PM and 3/21/2023 9:37:45 AM. IsImprovementStdDev: Marked as improvement because 11.224045890897813 (T) = (0 -10443.19731543443) / Math.Sqrt((41905602.49952611 / (299)) + (314331366.6334067 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.7696579941879864 = (45337.78925220144 - 10443.19731543443) / 45337.78925220144 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.PopCount_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 10.98774164746544 < 82.32595242912066. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/14/2023 5:06:59 PM, 3/21/2023 9:37:45 AM falls between 3/12/2023 3:57:11 PM and 3/21/2023 9:37:45 AM. IsImprovementStdDev: Marked as improvement because 33.698141253532015 (T) = (0 -13118.768979350429) / Math.Sqrt((379169987.48265237 / (299)) + (137573142.36296198 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.8569289171067561 = (91694.06363646055 - 13118.768979350429) / 91694.06363646055 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 2.420173150121343 < 6.845668739069994. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/16/2023 8:14:39 PM, 3/17/2023 8:17:18 AM, 3/21/2023 9:37:45 AM falls between 3/12/2023 3:57:11 PM and 3/21/2023 9:37:45 AM. IsImprovementStdDev: Marked as improvement because 30.42476174979974 (T) = (0 -2421.482674466103) / Math.Sqrt((142326239.95294872 / (299)) + (148.51324825435833 / (16))) is greater than 1.9675720194858106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (16) - 2, .975) and 0.8965741145227546 = (23412.733314222885 - 2421.482674466103) / 23412.733314222885 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.Log2_uint ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 11.047105844191245 < 47.509368770417275. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/14/2023 5:06:59 PM, 3/21/2023 9:37:45 AM falls between 3/12/2023 3:57:11 PM and 3/21/2023 9:37:45 AM. IsImprovementStdDev: Marked as improvement because 13.206505561268221 (T) = (0 -15185.658646000928) / Math.Sqrt((49874847.11074319 / (299)) + (258535127.753229 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.7109811597505055 = (52542.10636542573 - 15185.658646000928) / 52542.10636542573 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_uint ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 2.4133177810859308 < 50.94321837445737. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/14/2023 5:06:59 PM, 3/21/2023 9:37:45 AM falls between 3/12/2023 3:57:11 PM and 3/21/2023 9:37:45 AM. IsImprovementStdDev: Marked as improvement because 12.551791573204566 (T) = (0 -12223.580918201747) / Math.Sqrt((53716239.88264945 / (299)) + (423188828.7717239 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.7873638411543894 = (57485.89978563788 - 12223.580918201747) / 57485.89978563788 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 2.4144499021909227 < 52.413066703937574. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/14/2023 5:06:59 PM, 3/21/2023 9:37:45 AM falls between 3/12/2023 3:57:11 PM and 3/21/2023 9:37:45 AM. IsImprovementStdDev: Marked as improvement because 18.1810075619401 (T) = (0 -10202.8587504139) / Math.Sqrt((89651693.1204691 / (299)) + (288994257.676181 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.842837223637375 = (64919.053904167675 - 10202.8587504139) / 64919.053904167675 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
kg commented 1 year ago

Expected from https://github.com/dotnet/runtime/commit/6ddb03f8e8c1876fa14f0289a42c128dd95e08a4 afaik