dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Changes at 5/12/2022 5:35:05 PM #5424

Closed performanceautofiler[bot] closed 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline 35e47797a36e29475d28af61f0fa0fb3bf067838
Compare 04e65d2b871211832f9999256435787c727afd0f
Diff Diff

Improvements in System.Net.Security.Tests.SslStreamTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[HandshakeECDSA512CertAsync - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Net.Security.Tests.SslStreamTests.HandshakeECDSA512CertAsync(protocol%3a%20Tls13).html>) 29.27 ms 25.18 ms 0.86 0.35 False
[HandshakeECDSA512CertAsync - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Net.Security.Tests.SslStreamTests.HandshakeECDSA512CertAsync(protocol%3a%20Tls12).html>) 29.62 ms 25.38 ms 0.86 0.33 False
[HandshakeECDSA256CertAsync - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Net.Security.Tests.SslStreamTests.HandshakeECDSA256CertAsync(protocol%3a%20Tls13).html>) 5.73 ms 4.52 ms 0.79 0.61 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Security.Tests.SslStreamTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Net.Security.Tests.SslStreamTests.HandshakeECDSA512CertAsync(protocol: Tls13) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 25.181839805000006 < 27.243851344537042. IsChangePoint: Marked as a change because one of 3/22/2022 12:26:23 PM, 5/12/2022 12:13:09 PM, 5/18/2022 4:10:31 AM falls between 5/9/2022 3:38:36 PM and 5/18/2022 4:10:31 AM. IsImprovementStdDev: Marked as improvement because 7.735186088358221 (T) = (0 -24584574.257345986) / Math.Sqrt((4447468905604.916 / (286)) + (869364639817.689 / (23))) is greater than 1.9677212881555053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (286) + (23) - 2, .975) and 0.0677492037529852 = (26371202.209015768 - 24584574.257345986) / 26371202.209015768 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Net.Security.Tests.SslStreamTests.HandshakeECDSA512CertAsync(protocol: Tls12) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 25.383129844444444 < 28.28076772592592. IsChangePoint: Marked as a change because one of 3/22/2022 12:26:23 PM, 5/12/2022 12:13:09 PM, 5/18/2022 4:10:31 AM falls between 5/9/2022 3:38:36 PM and 5/18/2022 4:10:31 AM. IsImprovementStdDev: Marked as improvement because 9.022072456318611 (T) = (0 -26066454.143691834) / Math.Sqrt((4391467536437.5864 / (285)) + (757856840889.4608 / (23))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (285) + (23) - 2, .975) and 0.0707301615855341 = (28050468.29903229 - 26066454.143691834) / 28050468.29903229 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Net.Security.Tests.SslStreamTests.HandshakeECDSA256CertAsync(protocol: Tls13) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 4.5177316888888885 < 5.640464019429865. IsChangePoint: Marked as a change because one of 3/22/2022 12:26:23 PM, 5/12/2022 12:13:09 PM, 5/18/2022 4:10:31 AM falls between 5/9/2022 3:38:36 PM and 5/18/2022 4:10:31 AM. IsImprovementStdDev: Marked as improvement because 8.506472461270683 (T) = (0 -4841105.252297311) / Math.Sqrt((587459391075.4972 / (299)) + (135238322674.4278 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.13467089395291623 = (5594524.925218336 - 4841105.252297311) / 5594524.925218336 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
kunalspathak commented 2 years ago

Related to https://github.com/dotnet/performance/pull/2405