search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Changes at 7/13/2022 11:09:36 AM #6936

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline d5442952a7cdb6fe1eb63b7a6c977be2090959e7
Compare 750157dbe53c0083e98e7be47a6d3eaeeb7ecc91
Diff Diff

Regressions in System.Diagnostics.Perf_Process

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetProcessesByName - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Diagnostics.Perf_Process.GetProcessesByName.html>) 742.08 μs 1.01 ms 1.37 0.62 False
[GetProcesses - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Diagnostics.Perf_Process.GetProcesses.html>) 868.88 μs 1.19 ms 1.37 0.67 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Diagnostics.Perf_Process*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Diagnostics.Perf_Process.GetProcessesByName ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.013178392857143 > 734.0845683620109. IsChangePoint: Marked as a change because one of 4/27/2022 2:48:42 PM, 6/4/2022 3:35:14 AM, 6/8/2022 9:45:10 AM, 7/13/2022 5:51:45 AM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM. IsRegressionStdDev: Marked as regression because -9.065718570320533 (T) = (0 -951425.0398315875) / Math.Sqrt((7904907401.178629 / (34)) + (3709548340.9750423 / (20))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (20) - 2, .025) and -0.24193694786631287 = (766081.5965465605 - 951425.0398315875) / 766081.5965465605 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Diagnostics.Perf_Process.GetProcesses ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.188464678030303 > 929.1934250919118. IsChangePoint: Marked as a change because one of 5/23/2022 1:47:22 PM, 5/26/2022 4:20:35 PM, 6/2/2022 5:58:32 PM, 6/8/2022 9:45:10 AM, 7/11/2022 5:09:40 AM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM. IsRegressionStdDev: Marked as regression because -6.731096075883674 (T) = (0 -1099548.348439357) / Math.Sqrt((11536310180.355911 / (25)) + (10638687983.521454 / (29))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (29) - 2, .025) and -0.21386316926994628 = (905825.6122069 - 1099548.348439357) / 905825.6122069 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline d5442952a7cdb6fe1eb63b7a6c977be2090959e7
Compare 750157dbe53c0083e98e7be47a6d3eaeeb7ecc91
Diff Diff

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Ctor_ByteArray - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Numerics.Tests.Perf_BigInteger.Ctor_ByteArray(numberString%3a%201234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123.html>) 157.87 ns 200.84 ns 1.27 0.03 False
[Parse - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Numerics.Tests.Perf_BigInteger.Parse(numberString%3a%201234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012.html>) 2.33 μs 2.71 μs 1.16 0.01 False
[Parse - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Numerics.Tests.Perf_BigInteger.Parse(numberString%3a%20-2147483648).html>) 203.39 ns 222.24 ns 1.09 0.01 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BigInteger.Ctor_ByteArray(numberString: 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 200.83515123660575 > 165.71851376052396. IsChangePoint: Marked as a change because one of 3/17/2022 11:30:59 AM, 5/24/2022 10:45:34 PM, 5/31/2022 5:02:26 PM, 6/6/2022 8:15:49 PM, 6/11/2022 3:29:22 PM, 6/14/2022 10:05:19 AM, 6/17/2022 2:19:07 PM, 7/13/2022 5:51:45 AM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM. IsRegressionStdDev: Marked as regression because -131.09833231275312 (T) = (0 -199.3754105373878) / Math.Sqrt((0.6109001374770696 / (34)) + (1.629018433091358 / (20))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (20) - 2, .025) and -0.2615568972568911 = (158.0391744287606 - 199.3754105373878) / 158.0391744287606 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BigInteger.Parse(numberString: 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.7077002365716263 > 2.4422227854452974. IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/6/2022 3:55:24 AM, 7/12/2022 1:46:57 PM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM. IsRegressionStdDev: Marked as regression because -20.074959147596587 (T) = (0 -2743.7680826378537) / Math.Sqrt((10626.405625172134 / (31)) + (1114.1016161543653 / (23))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (23) - 2, .025) and -0.1692044818312588 = (2346.6965148307036 - 2743.7680826378537) / 2346.6965148307036 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BigInteger.Parse(numberString: -2147483648) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 222.2421756226367 > 212.5152029666535. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 7/2/2022 7:35:06 PM, 7/6/2022 3:55:24 AM, 7/12/2022 1:46:57 PM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM. IsRegressionStdDev: Marked as regression because -7.294962940240493 (T) = (0 -222.0961039092985) / Math.Sqrt((141.52183744477387 / (31)) + (1.770528924985671 / (23))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (23) - 2, .025) and -0.07615886533603575 = (206.3785478735502 - 222.0961039092985) / 206.3785478735502 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
DrewScoggins commented 2 years ago

https://github.com/dotnet/runtime/issues/74158