search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 3 Regressions on 11/5/2022 3:05:12 PM #9639

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline ae1ada9ec291e24749cfdce8f4cad03c624bbf58
Compare d3ab95d3be895a1950a46c559397780dbb3e9807
Diff Diff

Regressions in System.IO.Tests.BinaryWriterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[WriteNonAsciiChar - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.IO.Tests.BinaryWriterTests.WriteNonAsciiChar.html>) 22.59 ns 31.08 ns 1.38 0.64 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.BinaryWriterTests*'

Related Issues

Regressions

Improvements

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.IO.Tests.BinaryWriterTests.WriteNonAsciiChar ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 31.07720332520468 > 24.080739655076236. IsChangePoint: Marked as a change because one of 9/23/2022 10:31:05 PM, 11/5/2022 1:28:30 PM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM. IsRegressionStdDev: Marked as regression because -3.531833865313926 (T) = (0 -29.46730582874035) / Math.Sqrt((11.234504408801987 / (29)) + (13.341747260336112 / (8))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (8) - 2, .025) and -0.20746939708202788 = (24.40418440413569 - 29.46730582874035) / 24.40418440413569 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a20b31ab0c8b9b0aad8fbf72d1c5e91a2095adf0
Compare d3ab95d3be895a1950a46c559397780dbb3e9807
Diff Diff

Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadStructAndReverseLE - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructAndReverseLE.html>) 9.32 ns 13.60 ns 1.46 0.61 False
[ReadStructFieldByFieldBE - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldBE.html>) 31.22 ns 34.99 ns 1.12 0.50 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*'

Related Issues

Regressions

Improvements

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructAndReverseLE ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 13.602822849988591 > 10.213792960872157. IsChangePoint: Marked as a change because one of 9/16/2022 12:02:07 PM, 9/23/2022 4:51:48 AM, 11/3/2022 8:45:56 PM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM. IsRegressionStdDev: Marked as regression because -6.080708498967696 (T) = (0 -14.045544522004981) / Math.Sqrt((3.4814547112647634 / (27)) + (1.1835777602762014 / (8))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (8) - 2, .025) and -0.2950149383110965 = (10.845855214861524 - 14.045544522004981) / 10.845855214861524 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldBE ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 34.987769801752634 > 30.610675718097777. IsChangePoint: Marked as a change because one of 9/7/2022 3:19:02 AM, 9/23/2022 10:31:05 PM, 11/3/2022 8:45:56 PM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM. IsRegressionStdDev: Marked as regression because -6.695647096178333 (T) = (0 -33.77458128857416) / Math.Sqrt((4.63921388309199 / (28)) + (1.2644919688081449 / (8))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (8) - 2, .025) and -0.12714052368267928 = (29.964836308274393 - 33.77458128857416) / 29.964836308274393 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

Looks like https://github.com/dotnet/runtime/pull/76339. @stephentoub do you see similar regressions in coreclr or is it mono specific?

stephentoub commented 1 year ago

Looks like https://github.com/dotnet/runtime/pull/76339.

That PR only added new APIs. Other than changing a few existing methods from internal to public, it didn't touch any existing code.