dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 3 Improvements on 11/21/2022 12:46:21 PM #10024

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 0ff0f7d0d2a9bbdd736e06ba379dd797435fe5bc
Compare 1ab2c4a1add9afe3d7ea3bf44fb9cd5da63079f8
Diff Diff

Improvements in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Remainder - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BigInteger.Remainder(arguments%3a%2065536%2c32768%20bits).html>) 1.96 ms 1.73 ms 0.88 0.02 False
[ToStringD - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString%3a%20123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678.html>) 2.30 μs 1.00 μs 0.44 0.00 False
[Divide - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BigInteger.Divide(arguments%3a%2065536%2c32768%20bits).html>) 1.96 ms 1.73 ms 0.88 0.02 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BigInteger.Remainder(arguments: 65536,32768 bits) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 1.734563888888889 < 1.8632973485742188. IsChangePoint: Marked as a change because one of 9/20/2022 8:04:40 PM, 11/8/2022 6:56:37 AM, 11/11/2022 3:15:52 PM, 11/21/2022 3:20:13 AM, 11/22/2022 2:20:42 AM falls between 11/13/2022 1:28:42 PM and 11/22/2022 2:20:42 AM. IsImprovementStdDev: Marked as improvement because 33.973311887622806 (T) = (0 -1734009.1725308644) / Math.Sqrt((5312037680.26004 / (299)) + (266063.34383069916 / (3))) is greater than 1.9679030112610798 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (3) - 2, .975) and 0.07645742974016746 = (1877562.7982617114 - 1734009.1725308644) / 1877562.7982617114 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString: 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 1.0021852988082154 < 2.1866053166936523. IsChangePoint: Marked as a change because one of 11/14/2022 7:25:10 PM, 11/22/2022 2:20:42 AM falls between 11/13/2022 1:28:42 PM and 11/22/2022 2:20:42 AM. IsImprovementStdDev: Marked as improvement because 851.6341240853565 (T) = (0 -1010.798805847022) / Math.Sqrt((114.32745448737593 / (299)) + (52.89919417144707 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.5594445467951784 = (2294.373610618059 - 1010.798805847022) / 2294.373610618059 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BigInteger.Divide(arguments: 65536,32768 bits) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 1.7336025630341882 < 1.862963261830357. IsChangePoint: Marked as a change because one of 9/20/2022 8:04:40 PM, 11/8/2022 6:56:37 AM, 11/11/2022 3:15:52 PM, 11/21/2022 3:20:13 AM, 11/22/2022 2:20:42 AM falls between 11/13/2022 1:28:42 PM and 11/22/2022 2:20:42 AM. IsImprovementStdDev: Marked as improvement because 36.02568793536901 (T) = (0 -1735009.0570563695) / Math.Sqrt((3769826349.601997 / (299)) + (3552922.784255823 / (3))) is greater than 1.9679030112610798 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (3) - 2, .975) and 0.07159279450327878 = (1868801.8003135768 - 1735009.0570563695) / 1868801.8003135768 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
lewing commented 1 year ago

probably a v8 bump