search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 30 Regressions on 5/3/2023 11:54:24 PM #17567

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline edd6d6326f853be985172c3b7514acf33675ec82
Compare cb5fe56312c40244524f2bd5d89006279e52d1da
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.IO.Tests.Perf_FileStream

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Read - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.Read(fileSize%3a%201024%2c%20userBufferSize%3a%201024%2c%20options%3a%20Asynchronous).html>) 8.49 μs 10.45 μs 1.23 0.26 False
[CopyToFile - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.CopyToFile(fileSize%3a%201024%2c%20options%3a%20None).html>) 19.37 μs 24.12 μs 1.25 0.15 False
[OpenClose - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.OpenClose(fileSize%3a%201024%2c%20options%3a%20Asynchronous).html>) 6.89 μs 8.08 μs 1.17 0.27 False
[Write - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.Write(fileSize%3a%201024%2c%20userBufferSize%3a%201024%2c%20options%3a%20Asynchronous).html>) 8.31 μs 10.53 μs 1.27 0.27 False
[Write - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.Write(fileSize%3a%201024%2c%20userBufferSize%3a%201024%2c%20options%3a%20None).html>) 8.47 μs 10.17 μs 1.20 0.30 False
[Read - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.Read(fileSize%3a%201024%2c%20userBufferSize%3a%201024%2c%20options%3a%20None).html>) 8.71 μs 10.43 μs 1.20 0.23 False
[OpenClose - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.OpenClose(fileSize%3a%201024%2c%20options%3a%20None).html>) 6.92 μs 7.96 μs 1.15 0.23 False
[LockUnlock - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.LockUnlock(fileSize%3a%201024%2c%20options%3a%20None).html>) 10.79 μs 11.54 μs 1.07 0.30 False
[ReadByte - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.ReadByte(fileSize%3a%201024%2c%20options%3a%20Asynchronous).html>) 23.20 μs 24.75 μs 1.07 0.17 False
[LockUnlock - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.LockUnlock(fileSize%3a%201024%2c%20options%3a%20Asynchronous).html>) 10.68 μs 12.36 μs 1.16 0.30 False
[WriteByte - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.WriteByte(fileSize%3a%201024%2c%20options%3a%20Asynchronous).html>) 22.45 μs 24.11 μs 1.07 0.21 False

graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_FileStream*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_FileStream.Read(fileSize: 1024, userBufferSize: 1024, options: Asynchronous) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 10.450880915551839 > 9.243163377604345. IsChangePoint: Marked as a change because one of 3/10/2023 8:23:08 AM, 4/17/2023 1:30:24 AM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -16.254036360558214 (T) = (0 -10454.018443980429) / Math.Sqrt((62979.08456091162 / (27)) + (105911.35385723332 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.17248239602734097 = (8916.141069069537 - 10454.018443980429) / 8916.141069069537 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.CopyToFile(fileSize: 1024, options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 24.12115965181771 > 21.259996546151815. IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -19.154846844693317 (T) = (0 -23422.98423785152) / Math.Sqrt((195255.26104723552 / (27)) + (302327.0510313284 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.15231934188587873 = (20326.81686963408 - 23422.98423785152) / 20326.81686963408 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.OpenClose(fileSize: 1024, options: Asynchronous) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 8.077539670723286 > 7.256028116762241. IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -10.749173715194274 (T) = (0 -8245.74462383664) / Math.Sqrt((20482.963409828568 / (27)) + (236232.5532253561 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.19391356707415996 = (6906.483728167949 - 8245.74462383664) / 6906.483728167949 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.Write(fileSize: 1024, userBufferSize: 1024, options: Asynchronous) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 10.529785776901164 > 8.789147911988438. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -26.02445447454076 (T) = (0 -10193.071353152896) / Math.Sqrt((35772.56799249821 / (26)) + (53912.671254127534 / (16))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (16) - 2, .025) and -0.21341372225921654 = (8400.326423023089 - 10193.071353152896) / 8400.326423023089 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.Write(fileSize: 1024, userBufferSize: 1024, options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 10.1680873852459 > 8.84148017469419. IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -16.1971536696331 (T) = (0 -10332.117386642833) / Math.Sqrt((132393.12946788085 / (27)) + (105901.24455653214 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.2023198780292643 = (8593.484625387979 - 10332.117386642833) / 8593.484625387979 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.Read(fileSize: 1024, userBufferSize: 1024, options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 10.429970903150133 > 9.332135175965167. IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -13.494848512702573 (T) = (0 -10646.120411505737) / Math.Sqrt((78961.89464422535 / (27)) + (166309.983091556 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.17135610310571953 = (9088.713827740976 - 10646.120411505737) / 9088.713827740976 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.OpenClose(fileSize: 1024, options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.964085587839314 > 7.253869675606825. IsChangePoint: Marked as a change because one of 5/3/2023 1:06:54 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -14.329789933533643 (T) = (0 -8029.410069116254) / Math.Sqrt((32558.529119307383 / (25)) + (75580.8721432245 / (17))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (17) - 2, .025) and -0.15648176525832863 = (6942.962967792828 - 8029.410069116254) / 6942.962967792828 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.LockUnlock(fileSize: 1024, options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 11.537681744271987 > 10.813151470739243. IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -12.342399206010864 (T) = (0 -11935.178809501702) / Math.Sqrt((121169.57514443964 / (27)) + (173724.28845528592 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.14692535954212346 = (10406.238479430322 - 11935.178809501702) / 10406.238479430322 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.ReadByte(fileSize: 1024, options: Asynchronous) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 24.747154284963194 > 24.54572821183628. IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -8.048174018903248 (T) = (0 -25225.583129883955) / Math.Sqrt((376001.7256659369 / (27)) + (336251.09364449495 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.06342103127919319 = (23721.162538547884 - 25225.583129883955) / 23721.162538547884 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.LockUnlock(fileSize: 1024, options: Asynchronous) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.363636062533121 > 11.09988779656821. IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -13.46213219471004 (T) = (0 -11996.190570100352) / Math.Sqrt((111266.83507436156 / (27)) + (122590.31675455437 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.13871063463123243 = (10534.889378621881 - 11996.190570100352) / 10534.889378621881 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_FileStream.WriteByte(fileSize: 1024, options: Asynchronous) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 24.11290732307947 > 23.748805882304513. IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -9.593289062654216 (T) = (0 -24259.023780229785) / Math.Sqrt((459250.5478244729 / (27)) + (156717.80270468054 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.06922522283762467 = (22688.41331281784 - 24259.023780229785) / 22688.41331281784 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline edd6d6326f853be985172c3b7514acf33675ec82
Compare cb5fe56312c40244524f2bd5d89006279e52d1da
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadAllBytes - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.ReadAllBytes(size%3a%204096).html>) 14.86 μs 16.87 μs 1.14 0.16 False
[CopyTo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.CopyTo(size%3a%204096).html>) 37.92 μs 41.32 μs 1.09 0.09 False
[CopyTo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.CopyTo(size%3a%20512).html>) 37.50 μs 42.48 μs 1.13 0.14 False
[WriteAllBytes - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.WriteAllBytes(size%3a%20512).html>) 6.68 μs 7.73 μs 1.16 0.22 False
[WriteAllBytes - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.WriteAllBytes(size%3a%204096).html>) 8.43 μs 11.81 μs 1.40 0.24 False
[WriteAllText - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.WriteAllText(size%3a%2010000).html>) 25.38 μs 29.79 μs 1.17 0.13 False
[CopyToOverwrite - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.CopyToOverwrite(size%3a%20512).html>) 25.98 μs 27.52 μs 1.06 0.11 False
[AppendAllLines - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.AppendAllLines.html>) 14.43 μs 17.83 μs 1.24 0.18 False
[WriteAllText - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.WriteAllText(size%3a%20100).html>) 7.43 μs 8.55 μs 1.15 0.28 False
[ReadAllBytes - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.ReadAllBytes(size%3a%20512).html>) 14.45 μs 16.19 μs 1.12 0.20 False
[AppendAllText - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.AppendAllText(size%3a%20100).html>) 8.38 μs 10.11 μs 1.21 0.27 False
[WriteAllBytes - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.WriteAllBytes(size%3a%2016384).html>) 14.21 μs 17.42 μs 1.23 0.22 False

graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_File*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_File.ReadAllBytes(size: 4096) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.86908296886249 > 15.54394920340428. IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -14.90471966451719 (T) = (0 -16595.082971263553) / Math.Sqrt((151209.21830289558 / (27)) + (103387.65744614066 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.10943525908461872 = (14958.135533709241 - 16595.082971263553) / 14958.135533709241 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.CopyTo(size: 4096) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 41.321709508828896 > 39.55563748946543. IsChangePoint: Marked as a change because one of 3/1/2023 11:57:24 AM, 5/3/2023 1:06:54 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -12.723725912805625 (T) = (0 -40652.83560012588) / Math.Sqrt((721176.5866374169 / (25)) + (417890.07169486064 / (17))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (17) - 2, .025) and -0.07798750570935632 = (37711.78736749345 - 40652.83560012588) / 37711.78736749345 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.CopyTo(size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 42.48108146711554 > 39.947611838908806. IsChangePoint: Marked as a change because one of 3/1/2023 11:57:24 AM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -8.904067176281227 (T) = (0 -40778.2329776526) / Math.Sqrt((2344288.9275432294 / (27)) + (507885.55521136813 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.08130001912072016 = (37712.228111132565 - 40778.2329776526) / 37712.228111132565 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.WriteAllBytes(size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.7330533053307535 > 6.957282725995071. IsChangePoint: Marked as a change because one of 5/3/2023 1:06:54 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -26.832956071463233 (T) = (0 -7900.311355562448) / Math.Sqrt((21585.220228946524 / (26)) + (34413.657938385004 / (17))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (17) - 2, .025) and -0.2216942453119961 = (6466.684594675215 - 7900.311355562448) / 6466.684594675215 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.WriteAllBytes(size: 4096) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 11.809282963557402 > 8.750779602187695. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -21.42201820120111 (T) = (0 -12445.670672638698) / Math.Sqrt((66171.96170276376 / (27)) + (542891.7319118282 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.4887795926067867 = (8359.646205820758 - 12445.670672638698) / 8359.646205820758 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.WriteAllText(size: 10000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 29.78515821585213 > 26.925497053139065. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -23.215879061345404 (T) = (0 -29262.88681045999) / Math.Sqrt((402772.86666633975 / (27)) + (118165.12883104278 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.13440524553032032 = (25795.79645436138 - 29262.88681045999) / 25795.79645436138 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.CopyToOverwrite(size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.522793816137565 > 26.11207919694889. IsChangePoint: Marked as a change because one of 3/1/2023 11:57:24 AM, 3/10/2023 11:41:11 AM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -9.6625989320974 (T) = (0 -27965.71331348423) / Math.Sqrt((719835.503834481 / (27)) + (753071.3124463336 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.10353021792353308 = (25342.045790196982 - 27965.71331348423) / 25342.045790196982 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.AppendAllLines ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 17.82582564615385 > 14.984098966085973. IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -11.594675417583716 (T) = (0 -16637.61419100029) / Math.Sqrt((211679.9808324433 / (27)) + (398325.5635676167 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.14428083420406446 = (14539.799753417206 - 16637.61419100029) / 14539.799753417206 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.WriteAllText(size: 100) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 8.554132798102982 > 7.480210980470133. IsChangePoint: Marked as a change because one of 3/10/2023 11:41:11 AM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -19.568287899148963 (T) = (0 -8837.747798801609) / Math.Sqrt((50836.88306534708 / (27)) + (80691.71972397612 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.22589604103537728 = (7209.21473189305 - 8837.747798801609) / 7209.21473189305 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.ReadAllBytes(size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.192894345238095 > 15.085891584474656. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/10/2023 8:06:53 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -11.965435221758101 (T) = (0 -15950.784354587331) / Math.Sqrt((92994.41899361354 / (26)) + (201324.0753690322 / (16))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (16) - 2, .025) and -0.10541040139460017 = (14429.73970071533 - 15950.784354587331) / 14429.73970071533 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.AppendAllText(size: 100) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 10.112679983999998 > 9.056960261607141. IsChangePoint: Marked as a change because one of 3/11/2023 1:35:17 AM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -16.52832174913082 (T) = (0 -10007.09059460817) / Math.Sqrt((43974.732443739005 / (27)) + (91693.03885995688 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.16508293446111402 = (8589.165885634357 - 10007.09059460817) / 8589.165885634357 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.Tests.Perf_File.WriteAllBytes(size: 16384) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 17.416230128968255 > 15.143016330566672. IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -10.349629846922774 (T) = (0 -17144.339124779894) / Math.Sqrt((225748.447832284 / (27)) + (860683.530935875 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.17716367199271957 = (14564.108231234923 - 17144.339124779894) / 14564.108231234923 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline edd6d6326f853be985172c3b7514acf33675ec82
Compare cb5fe56312c40244524f2bd5d89006279e52d1da
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[CreateNew - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity%3a%201000000).html>) 29.83 μs 33.28 μs 1.12 0.13 False
[CreateFromFile_Read - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile_Read(capacity%3a%2010000).html>) 12.94 μs 14.10 μs 1.09 0.23 False
[CreateNew - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity%3a%20100000).html>) 25.12 μs 27.74 μs 1.10 0.18 False
[CreateFromFile - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity%3a%2010000).html>) 22.09 μs 23.89 μs 1.08 0.19 False
[CreateNew - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity%3a%2010000).html>) 23.91 μs 29.46 μs 1.23 0.12 False

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity: 1000000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 33.27591387056415 > 31.118855855040426. IsChangePoint: Marked as a change because one of 3/10/2023 11:41:11 AM, 4/12/2023 8:39:35 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -15.703232231661138 (T) = (0 -34041.27311801008) / Math.Sqrt((803838.0844610789 / (27)) + (533138.1119375235 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.13105625531765996 = (30096.887717091937 - 34041.27311801008) / 30096.887717091937 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile_Read(capacity: 10000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 14.102148690422663 > 13.274289960690885. IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -16.6137310717646 (T) = (0 -14240.413388848006) / Math.Sqrt((69637.59329780644 / (27)) + (97616.85248388223 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.12195054474249174 = (12692.550001940097 - 14240.413388848006) / 12692.550001940097 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity: 100000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.744713945954487 > 26.232392663255204. IsChangePoint: Marked as a change because one of 4/18/2023 7:36:21 AM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -17.191359839551662 (T) = (0 -28923.64789331277) / Math.Sqrt((391988.27479376516 / (27)) + (540004.5406843078 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.15019677919248553 = (25146.69525819668 - 28923.64789331277) / 25146.69525819668 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity: 10000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.894926765267172 > 23.401835900823816. IsChangePoint: Marked as a change because one of 3/11/2023 1:35:17 AM, 4/12/2023 8:39:35 PM, 5/4/2023 2:44:24 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -7.1626760265692875 (T) = (0 -25691.369852185097) / Math.Sqrt((710359.2703707451 / (29)) + (2526689.1760705253 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.14445593652182745 = (22448.544353979243 - 25691.369852185097) / 22448.544353979243 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity: 10000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 29.45737099724758 > 25.21937730067507. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -33.389181138724226 (T) = (0 -30073.504855617168) / Math.Sqrt((442459.776645712 / (26)) + (286839.019007268 / (16))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (16) - 2, .025) and -0.26190338500613625 = (23831.860040117834 - 30073.504855617168) / 23831.860040117834 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline edd6d6326f853be985172c3b7514acf33675ec82
Compare cb5fe56312c40244524f2bd5d89006279e52d1da
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Reflection.Attributes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IsDefinedMethodBaseMiss - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Reflection.Attributes.IsDefinedMethodBaseMiss.html>) 878.70 ns 976.35 ns 1.11 0.20 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Attributes*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Reflection.Attributes.IsDefinedMethodBaseMiss ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 976.3532333566327 > 897.729649155377. IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -11.272402293565257 (T) = (0 -962.8439592098872) / Math.Sqrt((1764.9268870361893 / (27)) + (377.2232478176466 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.12411681551223483 = (856.5337213385077 - 962.8439592098872) / 856.5337213385077 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline edd6d6326f853be985172c3b7514acf33675ec82
Compare cb5fe56312c40244524f2bd5d89006279e52d1da
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.ComponentModel.Tests.Perf_TypeDescriptorTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetConverter - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.ComponentModel.Tests.Perf_TypeDescriptorTests.GetConverter(typeToConvert%3a%20typeof(System.ComponentModel.Tests.DerivedClass)).html>) 1.08 μs 1.18 μs 1.09 0.23 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.ComponentModel.Tests.Perf_TypeDescriptorTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.ComponentModel.Tests.Perf_TypeDescriptorTests.GetConverter(typeToConvert: typeof(System.ComponentModel.Tests.DerivedClass)) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.1784748784828531 > 1.10673774120205. IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/10/2023 11:41:11 AM, 5/3/2023 6:00:09 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM. IsRegressionStdDev: Marked as regression because -12.199111677203003 (T) = (0 -1188.3827062518105) / Math.Sqrt((651.711156122352 / (27)) + (1814.4015051324216 / (16))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (16) - 2, .025) and -0.13686464581123936 = (1045.3159139308173 - 1188.3827062518105) / 1045.3159139308173 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

range is https://github.com/dotnet/runtime/compare/c8f43d52f699c1da289a5cb36d371aec15901ab8...d819cc97b954dc64b9120cb42b2ffbbe10d7a3ae

@kotlarmilos do you see similar regression on other platforms?

kotlarmilos commented 1 year ago

I haven't noticed similar regressions on other platforms. Could be bimodal data though.

It would be useful to get instructions stream before/after https://github.com/dotnet/runtime/pull/85695.