search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 18 Regressions on 3/11/2023 1:35:17 AM #14045

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b9002422acc79f65a2f1b83f342ce52eb1ce2326
Compare 2bacbca96ddcf7ae809a0e32a0d5dbffd6354199
Diff Diff

Regressions in Benchstone.BenchI.Fib

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Test - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/Benchstone.BenchI.Fib.Test.html>) 2.09 ms 2.27 ms 1.09 0.17 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.Fib*'
### Payloads [Baseline]() [Compare]() ### Histogram #### Benchstone.BenchI.Fib.Test ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.266422619047619 > 2.1899359705528845. IsChangePoint: Marked as a change because one of 1/13/2023 4:18:46 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -11.319119613617927 (T) = (0 -2263626.8315476193) / Math.Sqrt((1801730785.6133575 / (48)) + (2088520345.331355 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.07844057075249286 = (2098981.522892959 - 2263626.8315476193) / 2098981.522892959 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b9002422acc79f65a2f1b83f342ce52eb1ce2326
Compare 2bacbca96ddcf7ae809a0e32a0d5dbffd6354199
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetPinnableReference - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).GetPinnableReference(Size%3a%204).html>) 48.11 ns 52.67 ns 1.09 0.07 False
[GetPinnableReference - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).GetPinnableReference(Size%3a%20512).html>) 48.33 ns 51.30 ns 1.06 0.07 False
[GetPinnableReference - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).GetPinnableReference(Size%3a%2033).html>) 48.03 ns 51.28 ns 1.07 0.05 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Span<Byte>.GetPinnableReference(Size: 4) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 52.667982595065624 > 50.50280039352934. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/18/2023 11:30:09 AM, 2/17/2023 12:34:41 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -15.469259435235243 (T) = (0 -51.67797230036326) / Math.Sqrt((0.2850509205390661 / (48)) + (0.43533599711819626 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.0655349836577399 = (48.49955477103577 - 51.67797230036326) / 48.49955477103577 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.GetPinnableReference(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 51.29778339413507 > 50.80444936254753. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/18/2023 11:30:09 AM, 2/16/2023 3:28:45 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -14.421087047021448 (T) = (0 -51.784031725067656) / Math.Sqrt((0.37385255360164504 / (48)) + (0.5202599524821231 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.06721226891908169 = (48.52271027348359 - 51.784031725067656) / 48.52271027348359 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.GetPinnableReference(Size: 33) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 51.277651934074235 > 50.93682447101759. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/18/2023 11:30:09 AM, 2/17/2023 12:34:41 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -17.48319395514449 (T) = (0 -51.773782619936064) / Math.Sqrt((0.36335687710642867 / (48)) + (0.3101796556676777 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.06579232794581882 = (48.57774001781711 - 51.773782619936064) / 48.57774001781711 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b9002422acc79f65a2f1b83f342ce52eb1ce2326
Compare 2bacbca96ddcf7ae809a0e32a0d5dbffd6354199
Diff Diff

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetPinnableReference - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Char).GetPinnableReference(Size%3a%20512).html>) 48.33 ns 51.33 ns 1.06 0.08 False
[GetPinnableReference - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Char).GetPinnableReference(Size%3a%204).html>) 48.39 ns 51.37 ns 1.06 0.05 False
[GetPinnableReference - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Char).GetPinnableReference(Size%3a%2033).html>) 47.78 ns 52.22 ns 1.09 0.04 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Span<Char>.GetPinnableReference(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 51.327035258926706 > 50.469823882118156. IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/3/2023 5:57:54 PM, 2/17/2023 12:34:41 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -23.946223110244137 (T) = (0 -51.54681124233891) / Math.Sqrt((0.24531855542387265 / (48)) + (0.1430827279279217 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.06454493669214545 = (48.421451707346456 - 51.54681124233891) / 48.421451707346456 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Char>.GetPinnableReference(Size: 4) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 51.366849349746786 > 50.438681861900456. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/18/2023 11:30:09 AM, 2/17/2023 12:34:41 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -12.373137970557304 (T) = (0 -51.9742173180475) / Math.Sqrt((0.3315274809315786 / (48)) + (0.9203196484644269 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.07392097562545819 = (48.39668699810747 - 51.9742173180475) / 48.39668699810747 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Char>.GetPinnableReference(Size: 33) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 52.21522142463035 > 50.465143874616594. IsChangePoint: Marked as a change because one of 1/13/2023 4:18:46 AM, 1/18/2023 11:30:09 AM, 2/17/2023 12:34:41 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -19.668367478669516 (T) = (0 -51.668983975887905) / Math.Sqrt((0.35983712494547004 / (48)) + (0.23894002964235447 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.06725878031896139 = (48.41279821604849 - 51.668983975887905) / 48.41279821604849 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b9002422acc79f65a2f1b83f342ce52eb1ce2326
Compare 2bacbca96ddcf7ae809a0e32a0d5dbffd6354199
Diff Diff

Regressions in System.Memory.Slice<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadOnlySpanStart - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Slice(Byte).ReadOnlySpanStart.html>) 48.84 ns 51.98 ns 1.06 0.07 False
[ReadOnlySpanStartLength - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Slice(Byte).ReadOnlySpanStartLength.html>) 46.96 ns 54.29 ns 1.16 0.14 False
[SpanStartLength - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Slice(Byte).SpanStartLength.html>) 46.15 ns 55.02 ns 1.19 0.17 False
[SpanStart - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Slice(Byte).SpanStart.html>) 48.73 ns 52.85 ns 1.08 0.09 False

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Slice&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Slice<Byte>.ReadOnlySpanStart ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 51.98042808521323 > 51.25801455652943. IsChangePoint: Marked as a change because one of 1/12/2023 10:41:19 PM, 2/3/2023 8:13:11 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -15.050749833749904 (T) = (0 -52.36406818332557) / Math.Sqrt((0.2748014218059079 / (48)) + (0.5316887530776225 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.06870838726943823 = (48.99752711505928 - 52.36406818332557) / 48.99752711505928 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Slice<Byte>.ReadOnlySpanStartLength ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 54.294216092384666 > 49.15458624912584. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -9.805574649533408 (T) = (0 -54.289660924924014) / Math.Sqrt((1.571181016186533 / (48)) + (6.786331141339011 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.16238725361305847 = (46.70531335936022 - 54.289660924924014) / 46.70531335936022 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Slice<Byte>.SpanStartLength ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 55.020717755570246 > 48.37298017090156. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -10.76849416831492 (T) = (0 -53.931641041983085) / Math.Sqrt((1.545632083673282 / (48)) + (5.268983109563734 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.15884662945557304 = (46.539067095807326 - 53.931641041983085) / 46.539067095807326 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Slice<Byte>.SpanStart ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 52.851834228397635 > 51.199674661072336. IsChangePoint: Marked as a change because one of 1/12/2023 10:41:19 PM, 2/1/2023 12:50:52 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -5.757533354547897 (T) = (0 -53.080804443078954) / Math.Sqrt((0.2578186000808916 / (48)) + (6.3704257658731045 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.08628220796400243 = (48.8646541883138 - 53.080804443078954) / 48.8646541883138 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b9002422acc79f65a2f1b83f342ce52eb1ce2326
Compare 2bacbca96ddcf7ae809a0e32a0d5dbffd6354199
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EmptyStaticFunction5Arg - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.EmptyStaticFunction5Arg.html>) 32.89 ms 37.49 ms 1.14 0.06 False
[GenericClassGenericInstanceMethod - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.GenericClassGenericInstanceMethod.html>) 3.09 ms 3.40 ms 1.10 0.10 False
[GenericGenericMethod - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.GenericGenericMethod.html>) 2.84 ms 3.03 ms 1.07 0.09 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'
### Payloads [Baseline]() [Compare]() ### Histogram #### PerfLabTests.LowLevelPerf.EmptyStaticFunction5Arg ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 37.48988778571428 > 34.50121421576923. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/17/2023 12:34:41 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -19.400994182591752 (T) = (0 -37800968.98614164) / Math.Sqrt((365412563329.6146 / (47)) + (708020582596.5198 / (12))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (12) - 2, .025) and -0.15290684850543365 = (32787530.957201596 - 37800968.98614164) / 32787530.957201596 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### PerfLabTests.LowLevelPerf.GenericClassGenericInstanceMethod ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.402079466071428 > 3.2426906616927087. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -11.61781902155298 (T) = (0 -3349310.266535981) / Math.Sqrt((4505166019.332722 / (48)) + (3781405586.4303846 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.07544037062553252 = (3114361.668037296 - 3349310.266535981) / 3114361.668037296 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### PerfLabTests.LowLevelPerf.GenericGenericMethod ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.0257708215277774 > 2.985019088854167. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/17/2023 6:13:48 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -16.62875330060637 (T) = (0 -3035684.726689242) / Math.Sqrt((3655384951.1262836 / (48)) + (316299130.86804336 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.058717951603747286 = (2867321.4826392457 - 3035684.726689242) / 2867321.4826392457 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b9002422acc79f65a2f1b83f342ce52eb1ce2326
Compare 2bacbca96ddcf7ae809a0e32a0d5dbffd6354199
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[AbsBenchmark - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_VectorOf(Byte).AbsBenchmark.html>) 13.32 ns 14.45 ns 1.09 0.32 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_VectorOf<Byte>.AbsBenchmark ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 14.452039906060195 > 14.215019766143751. IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -8.474550521106528 (T) = (0 -14.575941273995385) / Math.Sqrt((0.2721421688418943 / (48)) + (0.1149210604076723 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.07734232164285737 = (13.52953558138168 - 14.575941273995385) / 13.52953558138168 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b9002422acc79f65a2f1b83f342ce52eb1ce2326
Compare 2bacbca96ddcf7ae809a0e32a0d5dbffd6354199
Diff Diff

Regressions in System.Numerics.Tests.Perf_Vector3

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ZeroBenchmark - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_Vector3.ZeroBenchmark.html>) 11.47 ns 13.09 ns 1.14 0.50 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_Vector3.ZeroBenchmark ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 13.089494764527748 > 11.772108646550445. IsChangePoint: Marked as a change because one of 2/15/2023 8:48:29 AM, 2/21/2023 10:35:27 PM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -7.642733411448667 (T) = (0 -12.854107108745835) / Math.Sqrt((0.4714487008874269 / (48)) + (0.5791284154014491 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.16726231160592445 = (11.012183791885732 - 12.854107108745835) / 11.012183791885732 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b9002422acc79f65a2f1b83f342ce52eb1ce2326
Compare 2bacbca96ddcf7ae809a0e32a0d5dbffd6354199
Diff Diff

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetPinnableReference - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Int32).GetPinnableReference(Size%3a%20512).html>) 48.01 ns 51.42 ns 1.07 0.09 False
[GetPinnableReference - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Int32).GetPinnableReference(Size%3a%204).html>) 48.08 ns 51.28 ns 1.07 0.05 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Span<Int32>.GetPinnableReference(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 51.4190578266497 > 50.40756758556486. IsChangePoint: Marked as a change because one of 1/10/2023 11:45:30 PM, 1/11/2023 4:46:06 PM, 2/17/2023 12:34:41 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -10.686263034368055 (T) = (0 -51.96178243286712) / Math.Sqrt((0.33164942634582284 / (48)) + (1.2632591639701611 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.07397693802904472 = (48.3825868069635 - 51.96178243286712) / 48.3825868069635 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Int32>.GetPinnableReference(Size: 4) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 51.282881743900475 > 50.313979244103265. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/18/2023 11:30:09 AM, 2/17/2023 12:34:41 AM, 3/10/2023 8:06:53 PM, 3/14/2023 10:43:28 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 10:43:28 AM. IsRegressionStdDev: Marked as regression because -11.433810740502958 (T) = (0 -52.025383735705255) / Math.Sqrt((0.24724885664087243 / (48)) + (1.1776162935537047 / (12))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (12) - 2, .025) and -0.07599896504352456 = (48.35077488536507 - 52.025383735705255) / 48.35077488536507 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

emscripten bump https://github.com/dotnet/runtime/pull/81215

kg commented 1 year ago

emscripten bump dotnet/runtime#81215

Some of these are ones that were previously regressed by jiterpreter, is it possible this is a late detection for a preceding jiterp change?

radekdoulik commented 1 year ago

it is from this range https://github.com/dotnet/runtime/compare/728fd85bc7ad04f5a0ea2ad0d4d8afe371ff9b64...3816ed14fd92f4c61323f49fbd63470bcb93b916 so I think it is either the emscripten bump (likely) or icu update

obrazek