search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 3 Regressions on 11/8/2022 2:29:58 AM #9760

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bfa4812aec806f09f1e2927939ba98118e5e5bf7
Compare f33badf332c43c0d325e8c802466ea5fd362fb91
Diff Diff

Regressions in System.IO.Tests.StreamReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadLine - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange%3a%20%5b%20%20%201%2c%20%20%20%201%5d).html>) 3.05 ms 4.78 ms 1.57 0.07 False
[ReadLine - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange%3a%20%5b%20%20%201%2c%20%20%20%208%5d).html>) 1.49 ms 1.98 ms 1.33 0.04 False
[ReadLine - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange%3a%20%5b%20%20%200%2c%20%20%20%200%5d).html>) 3.16 ms 6.27 ms 1.98 0.11 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*'

Related Issues

Regressions

Improvements

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 1]) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 4.781691966517857 > 3.2166848428125. IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 11/14/2022 6:23:13 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 6:23:13 PM. IsRegressionStdDev: Marked as regression because -57.09251996792648 (T) = (0 -4600162.224617006) / Math.Sqrt((4605864924.373739 / (10)) + (8595748692.52886 / (36))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (36) - 2, .025) and -0.48856679159305294 = (3090329.739046474 - 4600162.224617006) / 3090329.739046474 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 8]) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 1.9781770828125 > 1.5480122912478145. IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 11/14/2022 6:23:13 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 6:23:13 PM. IsRegressionStdDev: Marked as regression because -89.13452586937301 (T) = (0 -1981355.6460121113) / Math.Sqrt((221593302.941373 / (10)) + (424404172.5873213 / (36))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (36) - 2, .025) and -0.355225713472459 = (1462011.5500430819 - 1981355.6460121113) / 1462011.5500430819 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0]) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 6.270424681089743 > 3.330590737103366. IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 11/14/2022 6:23:13 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 6:23:13 PM. IsRegressionStdDev: Marked as regression because -143.24380047732507 (T) = (0 -6254404.661465553) / Math.Sqrt((964507010.0760417 / (10)) + (13373204433.892677 / (37))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (37) - 2, .025) and -0.961104183815278 = (3189226.106946429 - 6254404.661465553) / 3189226.106946429 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

Looks like https://github.com/dotnet/runtime/pull/69888. @GrabYourPitchforks @stephentoub please take a look

stephentoub commented 1 year ago

This is the same as https://github.com/dotnet/runtime/issues/78401 and https://github.com/dotnet/runtime/issues/78176. The regressions are for tests involving empty or otherwise really short lines, such that the overhead of the IndexOf now being used is observable. This is acceptable.

As an aside, is there anything we can do to better group such issues, rather than having at least three different issues created for the same tests? @DrewScoggins