dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] ubuntu 18.04/x64 : Regression on 8/25/2022 6:55:57 AM #8157

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8583df7ef4a856c88a2a23cfaef614f4a77aa50
Compare 0e13ffaae62f26c0682bee3eb61de1a991a62334
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[DataContractSerializer_ - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/MicroBenchmarks.Serializers.XmlToStream(MyEventsListerViewModel).DataContractSerializer.html>) 0.00 ns 3.91 ms 0.10 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;MyEventsListerViewModel&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### MicroBenchmarks.Serializers.Xml_ToStream<MyEventsListerViewModel>.DataContractSerializer_ ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.907090140625 > 0. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/25/2022 4:02:22 AM, 8/30/2022 4:40:43 AM falls between 8/21/2022 6:52:55 AM and 8/30/2022 4:40:43 AM. IsRegressionStdDev: Marked as regression because -236.2832973510978 (T) = (0 -3908818.803564092) / Math.Sqrt((0 / (35)) + (6568047329.97412 / (24))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (24) - 2, .025) and -∞ = (0 - 3908818.803564092) / 0 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8583df7ef4a856c88a2a23cfaef614f4a77aa50
Compare 0e13ffaae62f26c0682bee3eb61de1a991a62334
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[DataContractSerializer_ - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/MicroBenchmarks.Serializers.XmlToStream(IndexViewModel).DataContractSerializer.html>) 0.00 ns 286.48 μs 0.06 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;IndexViewModel&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### MicroBenchmarks.Serializers.Xml_ToStream<IndexViewModel>.DataContractSerializer_ ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 286.47900906666666 > 0. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/25/2022 4:02:22 AM, 8/30/2022 4:40:43 AM falls between 8/21/2022 6:52:55 AM and 8/30/2022 4:40:43 AM. IsRegressionStdDev: Marked as regression because -355.44073809668015 (T) = (0 -288237.33003317815) / Math.Sqrt((0 / (36)) + (15782554.220708556 / (24))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (24) - 2, .025) and -∞ = (0 - 288237.33003317815) / 0 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8583df7ef4a856c88a2a23cfaef614f4a77aa50
Compare 0e13ffaae62f26c0682bee3eb61de1a991a62334
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[DataContractJsonSerializer_ - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/MicroBenchmarks.Serializers.JsonFromStream(IndexViewModel).DataContractJsonSerializer.html>) 0.00 ns 2.03 ms 0.04 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>.DataContractJsonSerializer_ ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.0261208338541667 > 0. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/25/2022 4:02:22 AM, 8/30/2022 4:40:43 AM falls between 8/21/2022 6:52:55 AM and 8/30/2022 4:40:43 AM. IsRegressionStdDev: Marked as regression because -446.58945654206235 (T) = (0 -2045812.3760016283) / Math.Sqrt((0 / (36)) + (503646608.03449595 / (24))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (24) - 2, .025) and -∞ = (0 - 2045812.3760016283) / 0 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
naricc commented 2 years ago

@drewnoakes Why are these reporting 0 as the baseline? New benchmarks?