search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 18 Improvements on 11/8/2022 6:11:08 PM #9822

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 134968ad4cc78550f09e5ac629644d088f396120
Compare e0294993ba0c384a9ab5a778c2ba96731511bf9d
Diff Diff

Improvements in Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryGetValueHit - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.TryGetValueHit.html>) 234.38 ns 195.59 ns 0.83 0.26 False
[AddThenRemove_NoExpiration - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_NoExpiration.html>) 105.09 μs 83.28 μs 0.79 0.24 False
[AddThenRemove_AbsoluteExpiration - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_AbsoluteExpiration.html>) 119.10 μs 86.35 μs 0.73 0.23 False
[SetOverride - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.SetOverride.html>) 663.08 ns 495.83 ns 0.75 0.21 False
[AddThenRemove_ExpirationTokens - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_ExpirationTokens.html>) 139.66 μs 108.56 μs 0.78 0.16 False
[AddThenRemove_SlidingExpiration - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_SlidingExpiration.html>) 113.55 μs 85.54 μs 0.75 0.27 False
[GetMiss - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.GetMiss.html>) 191.83 ns 161.80 ns 0.84 0.37 False
[AddThenRemove_RelativeExpiration - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_RelativeExpiration.html>) 118.74 μs 91.77 μs 0.77 0.27 False

graph graph graph graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.TryGetValueHit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 25.577123214747633 (T) = (0 -187.63703233855222) / Math.Sqrt((100.8551985487386 / (299)) + (50.319851320240616 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16211272596659526 = (223.94066380231422 - 187.63703233855222) / 223.94066380231422 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_NoExpiration ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 37.31757308333736 (T) = (0 -83101.43320788909) / Math.Sqrt((21088111.844609655 / (299)) + (10229895.660837602 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.21692132094529346 = (106121.43508772961 - 83101.43320788909) / 106121.43508772961 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_AbsoluteExpiration ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 86.35378198587127 < 108.17698307039596. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 35.116719681537475 (T) = (0 -87554.938625627) / Math.Sqrt((17718323.120107546 / (299)) + (16261814.681694131 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.23457443434755593 = (114387.26710283801 - 87554.938625627) / 114387.26710283801 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.SetOverride ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 495.8342166483889 < 613.3025967541606. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 52.031486216503694 (T) = (0 -476.8295399318843) / Math.Sqrt((443.59435656233325 / (299)) + (372.00765931788396 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.2804385105612291 = (662.6668421399153 - 476.8295399318843) / 662.6668421399153 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_ExpirationTokens ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 52.628785755902136 (T) = (0 -110560.52397690358) / Math.Sqrt((19930186.060402922 / (299)) + (5286491.169871745 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.1848199739453386 = (135627.1258411452 - 110560.52397690358) / 135627.1258411452 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_SlidingExpiration ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 85.54368279569891 < 105.09486795517803. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 40.86193328792615 (T) = (0 -86918.06381894415) / Math.Sqrt((17683414.00250422 / (299)) + (10121824.00240961 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.2236391707066888 = (111955.75631766743 - 86918.06381894415) / 111955.75631766743 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.GetMiss ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 161.80237744402172 < 186.3062454146483. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 20.525550469724248 (T) = (0 -163.01884826479284) / Math.Sqrt((64.8496006807808 / (299)) + (59.902234788969274 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.1539575588337882 = (192.68400771961566 - 163.01884826479284) / 192.68400771961566 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_RelativeExpiration ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 91.7719089853427 < 112.56988020525104. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 44.180999231637166 (T) = (0 -87751.34286353888) / Math.Sqrt((23346150.17160149 / (299)) + (13753799.477619035 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.2640622830222964 = (119237.45833262878 - 87751.34286353888) / 119237.45833262878 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 134968ad4cc78550f09e5ac629644d088f396120
Compare e0294993ba0c384a9ab5a778c2ba96731511bf9d
Diff Diff

Improvements in Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Get - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount%3a%208%2c%20KeysCountPerProvider%3a%2040).html>) 8.16 ms 6.30 ms 0.77 0.13 False
[Get - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount%3a%2032%2c%20KeysCountPerProvider%3a%2010).html>) 26.68 ms 18.28 ms 0.69 0.11 False
[Get - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount%3a%2032%2c%20KeysCountPerProvider%3a%2040).html>) 128.27 ms 81.69 ms 0.64 0.13 False
[Get - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount%3a%2016%2c%20KeysCountPerProvider%3a%2010).html>) 5.52 ms 4.25 ms 0.77 0.09 False
[Get - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount%3a%208%2c%20KeysCountPerProvider%3a%2010).html>) 1.67 ms 1.39 ms 0.84 0.09 False
[Get - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount%3a%208%2c%20KeysCountPerProvider%3a%2020).html>) 3.65 ms 2.85 ms 0.78 0.09 False
[Get - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount%3a%2016%2c%20KeysCountPerProvider%3a%2040).html>) 28.50 ms 22.12 ms 0.78 0.12 False

graph graph graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 8, KeysCountPerProvider: 40) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 6.299546339250494 < 7.623692499575586. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 61.3587770259914 (T) = (0 -6228076.238367175) / Math.Sqrt((74075460994.0807 / (269)) + (17766629570.217976 / (33))) is greater than 1.9679030112610798 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (269) + (33) - 2, .975) and 0.21938495478471676 = (7978421.9847435225 - 6228076.238367175) / 7978421.9847435225 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 32, KeysCountPerProvider: 10) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 18.2760069375 < 24.04582297. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 88.29563263232555 (T) = (0 -17907056.876972273) / Math.Sqrt((455614038389.1237 / (272)) + (164999546958.34012 / (32))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (272) + (32) - 2, .975) and 0.2895387453837558 = (25204832.438955132 - 17907056.876972273) / 25204832.438955132 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 32, KeysCountPerProvider: 40) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 81.69119642857143 < 116.02850333333332. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 95.99551775447829 (T) = (0 -84282777.50925325) / Math.Sqrt((12349139252771.912 / (271)) + (3423801094560.6055 / (33))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (271) + (33) - 2, .975) and 0.3056136552853774 = (121377354.48108733 - 84282777.50925325) / 121377354.48108733 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 16, KeysCountPerProvider: 10) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 4.253429805418719 < 5.37231764604928. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 60.27263027876276 (T) = (0 -4356986.204877636) / Math.Sqrt((16902392198.105955 / (268)) + (14647299186.404726 / (33))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (268) + (33) - 2, .975) and 0.2374927251167795 = (5714025.752141076 - 4356986.204877636) / 5714025.752141076 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 8, KeysCountPerProvider: 10) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 1.392291703111495 < 1.6005829529895832. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 36.58384307433552 (T) = (0 -1402491.0318628177) / Math.Sqrt((8499027091.430579 / (272)) + (1310372717.01434 / (33))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (272) + (33) - 2, .975) and 0.18014316863154722 = (1710653.5899955472 - 1402491.0318628177) / 1710653.5899955472 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 8, KeysCountPerProvider: 20) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 2.8528844968992253 < 3.4702939393656713. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 48.37941752679751 (T) = (0 -2890100.736446896) / Math.Sqrt((7277798294.078646 / (271)) + (7329314529.383271 / (32))) is greater than 1.9678765312856872 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (271) + (32) - 2, .975) and 0.2112202357323609 = (3664014.8078979654 - 2890100.736446896) / 3664014.8078979654 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 16, KeysCountPerProvider: 40) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 22.12172003333334 < 26.80700529517857. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 98.2069618707379 (T) = (0 -20819173.972619805) / Math.Sqrt((675064713780.3893 / (270)) + (131486770945.43747 / (33))) is greater than 1.9678765312856872 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (270) + (33) - 2, .975) and 0.2752886763833258 = (28727540.600196023 - 20819173.972619805) / 28727540.600196023 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 134968ad4cc78550f09e5ac629644d088f396120
Compare e0294993ba0c384a9ab5a778c2ba96731511bf9d
Diff Diff

Improvements in Benchmark.GetChildKeysTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[AddChainedConfigurationEmpty - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty.html>) 68.22 ms 30.74 ms 0.45 0.02 False
[AddChainedConfigurationWithCommonPaths - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Benchmark.GetChildKeysTests.AddChainedConfigurationWithCommonPaths.html>) 1.27 ms 992.51 μs 0.78 0.18 False
[AddChainedConfigurationWithSplitting - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Benchmark.GetChildKeysTests.AddChainedConfigurationWithSplitting.html>) 1.25 ms 937.13 μs 0.75 0.19 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchmark.GetChildKeysTests*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 9/23/2022 6:26:22 PM, 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 143.84681811139438 (T) = (0 -31140619.051431622) / Math.Sqrt((18864317935721.316 / (263)) + (140353730527.46573 / (30))) is greater than 1.9681495542477299 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (263) + (30) - 2, .975) and 0.5607947817297397 = (70902206.43113938 - 31140619.051431622) / 70902206.43113938 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Benchmark.GetChildKeysTests.AddChainedConfigurationWithCommonPaths ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 38.854625251977886 (T) = (0 -948044.9596159066) / Math.Sqrt((11194051560.141932 / (258)) + (856496348.1609758 / (33))) is greater than 1.9682064355043956 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (258) + (33) - 2, .975) and 0.2544443290114181 = (1271595.1289845742 - 948044.9596159066) / 1271595.1289845742 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Benchmark.GetChildKeysTests.AddChainedConfigurationWithSplitting ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 937.1284136029411 < 1.2022820936298073. IsChangePoint: Marked as a change because one of 11/8/2022 3:14:34 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 67.00115335760786 (T) = (0 -947771.2300125851) / Math.Sqrt((1688662773.5652092 / (263)) + (535113633.82080144 / (32))) is greater than 1.9680934526838334 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (263) + (32) - 2, .975) and 0.2537794561507502 = (1270095.3328404373 - 947771.2300125851) / 1270095.3328404373 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
lewing commented 1 year ago

this looks like it is probably due to a v8 update between builds from 10.9.183 to 10.9.184

lewing commented 1 year ago

the relevant diff is https://github.com/dotnet/runtime/compare/d0d21fdb83aa7287e2141395ebecd50ae1cc851e...3e0d59994dd278ee002ec44f7dc42bb3f86c7a01 and it doesn't appear to have any related changes