DrewScoggins / performance-2

This repo contains benchmarks used for testing the performance of all .NET Runtimes
MIT License
0 stars 0 forks source link

[Perf] Windows/arm64: 2 Improvements on 9/29/2022 6:03:53 PM #8191

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS Windows 10.0.25094
Baseline da9b348cbd164aa9fed9242c52d8a101b1f00561
Compare bfef7ccc856ac307fd5102c00ea54ddbe0b06fa3
Diff Diff

Improvements in Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ForEach_Array - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.25094/Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array.html>) 9.30 ns 5.04 ns 0.54 0.81 False
[ForEach_String - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.25094/Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_String.html>) 2.44 ns 0.32 ns 0.13 0.92 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark*'
### Related Issues #### Regressions #### Improvements ### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 5.041343666539489 < 9.028745885796194. IsChangePoint: Marked as a change because one of 9/29/2022 1:36:12 PM, 10/6/2022 12:41:43 AM falls between 9/27/2022 9:53:04 AM and 10/6/2022 12:41:43 AM. IsImprovementStdDev: Marked as improvement because 27.61387760042512 (T) = (0 -4.550505960015658) / Math.Sqrt((2.118443199048198 / (299)) + (0.7852780966323764 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.5253272108281845 = (9.586616430984268 - 4.550505960015658) / 9.586616430984268 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_String ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 0.31652441504687234 < 1.767043052510001. IsChangePoint: Marked as a change because one of 10/1/2022 9:48:16 AM, 10/6/2022 12:41:43 AM falls between 9/27/2022 9:53:04 AM and 10/6/2022 12:41:43 AM. IsImprovementStdDev: Marked as improvement because 16.120463115853138 (T) = (0 -0.6651350004389033) / Math.Sqrt((0.9711967162192905 / (299)) + (0.22328676007738799 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.73720910177827 = (2.531042760383943 - 0.6651350004389033) / 2.531042760383943 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
DrewScoggins commented 2 years ago

Seems related to https://github.com/dotnet/runtime/pull/76283