search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 3 Regressions on 9/26/2022 10:49:59 PM #8825

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3f0106aed2ece86c56f9f49f0191e94ee5030bff
Compare 7fc8fb56327e9696ce9baa80f997e3f50675af1f
Diff Diff

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadAllBytesAsync - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.IO.Tests.Perf_File.ReadAllBytesAsync(size%3a%2016384).html>) 7.49 μs 9.07 μs 1.21 0.21 False
[ReadAllBytesAsync - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.IO.Tests.Perf_File.ReadAllBytesAsync(size%3a%20512).html>) 5.73 μs 6.86 μs 1.20 0.33 False
[ReadAllBytesAsync - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.IO.Tests.Perf_File.ReadAllBytesAsync(size%3a%204096).html>) 6.26 μs 7.19 μs 1.15 0.30 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_File*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.IO.Tests.Perf_File.ReadAllBytesAsync(size: 16384) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.071659091237708 > 8.009796614346554. IsChangePoint: Marked as a change because one of 9/26/2022 7:45:50 PM, 10/4/2022 5:30:44 AM falls between 9/25/2022 5:06:47 PM and 10/4/2022 5:30:44 AM. IsRegressionStdDev: Marked as regression because -30.999701638133054 (T) = (0 -9180.135568115762) / Math.Sqrt((25117.2310893202 / (21)) + (61770.49996130184 / (44))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (44) - 2, .025) and -0.20799603617997242 = (7599.4749098233515 - 9180.135568115762) / 7599.4749098233515 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.Perf_File.ReadAllBytesAsync(size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.859836955212443 > 5.730097537180211. IsChangePoint: Marked as a change because one of 9/26/2022 7:45:50 PM, 10/4/2022 5:30:44 AM falls between 9/25/2022 5:06:47 PM and 10/4/2022 5:30:44 AM. IsRegressionStdDev: Marked as regression because -28.19638584414468 (T) = (0 -7010.224773917447) / Math.Sqrt((37086.76700717027 / (21)) + (49525.936737476135 / (43))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (43) - 2, .025) and -0.2775713284657154 = (5487.149419935947 - 7010.224773917447) / 5487.149419935947 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.Perf_File.ReadAllBytesAsync(size: 4096) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.19043715006136 > 6.1301181689163915. IsChangePoint: Marked as a change because one of 9/26/2022 7:45:50 PM, 10/4/2022 5:30:44 AM falls between 9/25/2022 5:06:47 PM and 10/4/2022 5:30:44 AM. IsRegressionStdDev: Marked as regression because -30.749075888994337 (T) = (0 -7362.661810045072) / Math.Sqrt((30751.697813429928 / (21)) + (39672.42720035179 / (44))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (44) - 2, .025) and -0.25493316849951186 = (5866.975226137659 - 7362.661810045072) / 5866.975226137659 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS alpine 3.15
Baseline 3f0106aed2ece86c56f9f49f0191e94ee5030bff
Compare 7fc8fb56327e9696ce9baa80f997e3f50675af1f
Diff Diff

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadAllBytesAsync - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_alpine 3.15/System.IO.Tests.Perf_File.ReadAllBytesAsync(size%3a%2016384).html>) 7.41 μs 8.63 μs 1.16 0.00 False
[ReadAllBytesAsync - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_alpine 3.15/System.IO.Tests.Perf_File.ReadAllBytesAsync(size%3a%204096).html>) 5.57 μs 6.88 μs 1.24 0.01 False
[ReadAllBytesAsync - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_alpine 3.15/System.IO.Tests.Perf_File.ReadAllBytesAsync(size%3a%20512).html>) 5.24 μs 6.70 μs 1.28 0.01 False
[CopyTo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_alpine 3.15/System.IO.Tests.Perf_File.CopyTo(size%3a%20512).html>) 20.83 μs 28.30 μs 1.36 0.05 False

graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_File*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.IO.Tests.Perf_File.ReadAllBytesAsync(size: 16384) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 8.630060742009134 > 7.656850039002599. IsChangePoint: Marked as a change because one of 9/26/2022 7:45:50 PM, 10/4/2022 5:30:44 AM falls between 9/25/2022 5:06:47 PM and 10/4/2022 5:30:44 AM. IsRegressionStdDev: Marked as regression because -62.96600963785591 (T) = (0 -8736.592691588665) / Math.Sqrt((7643.92214375535 / (21)) + (7804.7313476064255 / (46))) is less than -1.997137908391408 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (46) - 2, .025) and -0.19975118936784977 = (7282.0037762929705 - 8736.592691588665) / 7282.0037762929705 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.Perf_File.ReadAllBytesAsync(size: 4096) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.88245885969764 > 5.876985550739924. IsChangePoint: Marked as a change because one of 9/26/2022 7:45:50 PM, 10/4/2022 5:30:44 AM falls between 9/25/2022 5:06:47 PM and 10/4/2022 5:30:44 AM. IsRegressionStdDev: Marked as regression because -64.22772281668412 (T) = (0 -7026.840759558374) / Math.Sqrt((6590.276567770871 / (21)) + (7161.525416722021 / (46))) is less than -1.997137908391408 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (46) - 2, .025) and -0.24696697055450964 = (5635.145858300987 - 7026.840759558374) / 5635.145858300987 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.Perf_File.ReadAllBytesAsync(size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.702499779443026 > 5.357592018097292. IsChangePoint: Marked as a change because one of 9/26/2022 7:45:50 PM, 10/4/2022 5:30:44 AM falls between 9/25/2022 5:06:47 PM and 10/4/2022 5:30:44 AM. IsRegressionStdDev: Marked as regression because -65.92669393992934 (T) = (0 -6620.849482964704) / Math.Sqrt((7740.064917293306 / (21)) + (5096.324906678931 / (46))) is less than -1.997137908391408 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (46) - 2, .025) and -0.27879198409020833 = (5177.424917685172 - 6620.849482964704) / 5177.424917685172 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.Perf_File.CopyTo(size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 28.295095495129875 > 21.673361387767482. IsChangePoint: Marked as a change because one of 9/26/2022 7:45:50 PM, 10/4/2022 5:30:44 AM falls between 9/25/2022 5:06:47 PM and 10/4/2022 5:30:44 AM. IsRegressionStdDev: Marked as regression because -77.1385976441343 (T) = (0 -28184.06490938403) / Math.Sqrt((114268.50933185674 / (21)) + (180363.02948914902 / (46))) is less than -1.997137908391408 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (46) - 2, .025) and -0.3602206292584903 = (20720.21575260792 - 28184.06490938403) / 20720.21575260792 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)