search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/arm64: 18 Regressions on 4/8/2023 9:16:19 PM #16072

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 3ab663e128e233ad1f4a1a4a0df64b08a846bcd5
Compare 1411364699b5784040e86f76cb3db8200f6a2c8c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SubSegment - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/Microsoft.Extensions.Primitives.StringSegmentBenchmark.SubSegment.html>) 1.99 ns 3.44 ns 1.73 0.41 False
[LastIndexOf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/Microsoft.Extensions.Primitives.StringSegmentBenchmark.LastIndexOf.html>) 3.30 ns 4.66 ns 1.41 0.36 False
[Ctor_String - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/Microsoft.Extensions.Primitives.StringSegmentBenchmark.Ctor_String.html>) 0.57 ns 3.64 ns 6.37 0.67 True
[Trim - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/Microsoft.Extensions.Primitives.StringSegmentBenchmark.Trim.html>) 26.56 ns 28.63 ns 1.08 0.03 False

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'
### Payloads [Baseline]() [Compare]() ### Histogram #### Microsoft.Extensions.Primitives.StringSegmentBenchmark.SubSegment ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.4417018314841834 > 2.0539801417274473. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -11.208579859962859 (T) = (0 -3.2822296675912117) / Math.Sqrt((0.076273067725579 / (24)) + (0.19423665078690686 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.8507443272698755 = (1.7734646645833523 - 3.2822296675912117) / 1.7734646645833523 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### Microsoft.Extensions.Primitives.StringSegmentBenchmark.LastIndexOf ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.663869186092034 > 3.5437301811512545. IsChangePoint: Marked as a change because one of 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -39.260268189481046 (T) = (0 -4.637329605447481) / Math.Sqrt((0.008173970218649504 / (18)) + (0.009108507736353607 / (19))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (19) - 2, .025) and -0.3489241813322427 = (3.4377985580090193 - 4.637329605447481) / 3.4377985580090193 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### Microsoft.Extensions.Primitives.StringSegmentBenchmark.Ctor_String ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.6437981926499408 > 0.6347087083598971. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -127.14842954544808 (T) = (0 -3.6230801919227416) / Math.Sqrt((0.012198030766506489 / (24)) + (0.0008633906617676965 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -5.300420238782765 = (0.575053735244606 - 3.6230801919227416) / 0.575053735244606 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### Microsoft.Extensions.Primitives.StringSegmentBenchmark.Trim ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 28.626532479812415 > 27.909209111978434. IsChangePoint: Marked as a change because one of 3/17/2023 1:20:12 AM, 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -18.013444053856986 (T) = (0 -28.469189365236893) / Math.Sqrt((0.103116027958174 / (24)) + (0.07432617404777571 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.06759752901329816 = (26.66659353506454 - 28.469189365236893) / 26.66659353506454 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 3ab663e128e233ad1f4a1a4a0df64b08a846bcd5
Compare 1411364699b5784040e86f76cb3db8200f6a2c8c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Constructors<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ArrayAsMemory - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Memory.Constructors(Byte).ArrayAsMemory.html>) 2.17 ns 3.78 ns 1.74 0.53 False
[ArrayAsMemoryStartLength - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Memory.Constructors(Byte).ArrayAsMemoryStartLength.html>) 1.30 ns 2.82 ns 2.16 0.54 True
[MemoryFromArrayStartLength - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Memory.Constructors(Byte).MemoryFromArrayStartLength.html>) 1.32 ns 2.90 ns 2.20 0.47 True
[ReadOnlyMemoryFromArrayStartLength - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Memory.Constructors(Byte).ReadOnlyMemoryFromArrayStartLength.html>) 1.29 ns 2.81 ns 2.18 0.52 True
[MemoryFromArray - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Memory.Constructors(Byte).MemoryFromArray.html>) 2.33 ns 3.74 ns 1.61 0.53 False

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Constructors&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Constructors<Byte>.ArrayAsMemory ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.777594773090289 > 2.341286862205988. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -39.472228022359936 (T) = (0 -3.7001437056543827) / Math.Sqrt((0.0181228318141427 / (24)) + (0.0061641872908159675 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.5975055338845024 = (2.316200868899086 - 3.7001437056543827) / 2.316200868899086 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### System.Memory.Constructors<Byte>.ArrayAsMemoryStartLength ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.816630687651461 > 1.1982496874733375. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -13.805102260253163 (T) = (0 -2.6748164513960138) / Math.Sqrt((0.02583956212744784 / (24)) + (0.13691230003629748 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -1.2526183218862994 = (1.187425506313104 - 2.6748164513960138) / 1.187425506313104 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### System.Memory.Constructors<Byte>.MemoryFromArrayStartLength ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.8990760916283183 > 1.3363855730950829. IsChangePoint: Marked as a change because one of 4/9/2023 1:58:52 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -7.6680990663736655 (T) = (0 -2.612094745689136) / Math.Sqrt((0.15098603198313057 / (28)) + (0.214753438943181 / (9))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (9) - 2, .025) and -1.0084358685963166 = (1.3005616890892877 - 2.612094745689136) / 1.3005616890892877 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### System.Memory.Constructors<Byte>.ReadOnlyMemoryFromArrayStartLength ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.8149503186607965 > 1.382056659829317. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -16.52692117835125 (T) = (0 -2.7615514001184995) / Math.Sqrt((0.02358688659878311 / (24)) + (0.09914699346723448 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -1.2487042807149042 = (1.228063389126716 - 2.7615514001184995) / 1.228063389126716 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### System.Memory.Constructors<Byte>.MemoryFromArray ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.7374895693857457 > 2.361739842432836. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -6.56762889508124 (T) = (0 -3.4112249742947145) / Math.Sqrt((0.3132374245276564 / (24)) + (0.4900628159184795 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.7659147496298712 = (1.9317042201554147 - 3.4112249742947145) / 1.9317042201554147 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 3ab663e128e233ad1f4a1a4a0df64b08a846bcd5
Compare 1411364699b5784040e86f76cb3db8200f6a2c8c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Order00LinqQueryX - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/LinqBenchmarks.Order00LinqQueryX.html>) 104.44 ms 119.40 ms 1.14 0.05 False
[Order00LinqMethodX - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/LinqBenchmarks.Order00LinqMethodX.html>) 104.06 ms 121.25 ms 1.17 0.03 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'LinqBenchmarks*'
### Payloads [Baseline]() [Compare]() ### Histogram #### LinqBenchmarks.Order00LinqQueryX ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 119.3960963 > 109.37541867000002. IsChangePoint: Marked as a change because one of 1/13/2023 3:30:06 PM, 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -29.77940584480758 (T) = (0 -120037084.06831501) / Math.Sqrt((3150094354748.7793 / (24)) + (1747314061175.447 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.1466169746248437 = (104688040.3175519 - 120037084.06831501) / 104688040.3175519 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### LinqBenchmarks.Order00LinqMethodX ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 121.24636240000001 > 109.510000145. IsChangePoint: Marked as a change because one of 1/13/2023 3:30:06 PM, 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -60.160216498563564 (T) = (0 -119647953.52161172) / Math.Sqrt((145806647322.52533 / (24)) + (754901626987.067 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.14592914690752753 = (104411301.38324939 - 119647953.52161172) / 104411301.38324939 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 3ab663e128e233ad1f4a1a4a0df64b08a846bcd5
Compare 1411364699b5784040e86f76cb3db8200f6a2c8c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Decimal

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Round - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Tests.Perf_Decimal.Round.html>) 13.44 ns 16.00 ns 1.19 0.44 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Decimal*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Decimal.Round ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.99773028263984 > 13.577035226765789. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -19.36088063811028 (T) = (0 -16.108064146190518) / Math.Sqrt((0.41638721753805147 / (24)) + (0.12302844570143846 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.24504165285585303 = (12.93777128599846 - 16.108064146190518) / 12.93777128599846 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 3ab663e128e233ad1f4a1a4a0df64b08a846bcd5
Compare 1411364699b5784040e86f76cb3db8200f6a2c8c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Logging.LoggingOverhead

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[NoArguments - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/Microsoft.Extensions.Logging.LoggingOverhead.NoArguments.html>) 38.60 ns 44.37 ns 1.15 0.15 False
[NoArguments_FilteredByLevel - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/Microsoft.Extensions.Logging.LoggingOverhead.NoArguments_FilteredByLevel.html>) 23.30 ns 29.49 ns 1.27 0.18 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Logging.LoggingOverhead*'
### Payloads [Baseline]() [Compare]() ### Histogram #### Microsoft.Extensions.Logging.LoggingOverhead.NoArguments ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 44.36703552989892 > 39.68360449937283. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -15.256647756652166 (T) = (0 -43.85389006981203) / Math.Sqrt((0.9541790690142429 / (24)) + (0.9751820497129721 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.1336074411625154 = (38.685252475795174 - 43.85389006981203) / 38.685252475795174 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### Microsoft.Extensions.Logging.LoggingOverhead.NoArguments_FilteredByLevel ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 29.48659204722188 > 24.25518734713546. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -4.997472292289468 (T) = (0 -26.584757907615785) / Math.Sqrt((1.1815676299404345 / (24)) + (3.4768590514542694 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.11830104234730847 = (23.772452050848944 - 26.584757907615785) / 23.772452050848944 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 3ab663e128e233ad1f4a1a4a0df64b08a846bcd5
Compare 1411364699b5784040e86f76cb3db8200f6a2c8c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in FractalPerf.Launch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Test - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/FractalPerf.Launch.Test.html>) 255.33 ms 275.35 ms 1.08 0.00 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'FractalPerf.Launch*'
### Payloads [Baseline]() [Compare]() ### Histogram #### FractalPerf.Launch.Test ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 275.345981 > 268.0931092269231. IsChangePoint: Marked as a change because one of 1/11/2023 5:39:19 AM, 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -32.048941757175044 (T) = (0 -274622762.05054945) / Math.Sqrt((10951551194.407557 / (24)) + (4678743479144.491 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.07533355170750544 = (255383793.81400332 - 274622762.05054945) / 255383793.81400332 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 3ab663e128e233ad1f4a1a4a0df64b08a846bcd5
Compare 1411364699b5784040e86f76cb3db8200f6a2c8c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToStringHex - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Tests.Perf_Int32.ToStringHex(value%3a%202147483647).html>) 27.98 ns 30.16 ns 1.08 0.13 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Int32.ToStringHex(value: 2147483647) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 30.163471930089226 > 29.38403490152716. IsChangePoint: Marked as a change because one of 1/13/2023 9:23:19 AM, 3/16/2023 8:14:39 PM, 3/17/2023 1:30:52 PM, 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -9.370877207492645 (T) = (0 -30.495377305052898) / Math.Sqrt((0.519603967684155 / (24)) + (0.21321911430734478 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.06376441179009959 = (28.667416363116875 - 30.495377305052898) / 28.667416363116875 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 3ab663e128e233ad1f4a1a4a0df64b08a846bcd5
Compare 1411364699b5784040e86f76cb3db8200f6a2c8c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Constructors<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadOnlyMemoryFromArray - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Memory.Constructors(String).ReadOnlyMemoryFromArray.html>) 2.21 ns 3.84 ns 1.73 0.37 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Constructors&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Constructors<String>.ReadOnlyMemoryFromArray ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.838116805145135 > 2.376194391468509. IsChangePoint: Marked as a change because one of 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -9.932864259601223 (T) = (0 -3.7275472176726674) / Math.Sqrt((0.11398747449585167 / (24)) + (0.23631483900689843 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.6764108492049694 = (2.2235284503440433 - 3.7275472176726674) / 2.2235284503440433 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 3ab663e128e233ad1f4a1a4a0df64b08a846bcd5
Compare 1411364699b5784040e86f76cb3db8200f6a2c8c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateObject

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EnumerateProperties - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase%3a%20ObjectProperties).html>) 2.12 μs 2.24 μs 1.06 0.00 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: ObjectProperties) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.239641463915763 > 2.229286534162877. IsChangePoint: Marked as a change because one of 1/30/2023 10:02:08 AM, 3/16/2023 8:14:39 PM, 4/8/2023 3:10:58 PM, 4/12/2023 8:39:35 PM falls between 4/3/2023 11:26:06 PM and 4/12/2023 8:39:35 PM. IsRegressionStdDev: Marked as regression because -107.00215800564862 (T) = (0 -2241.1083700377962) / Math.Sqrt((20.061165894774653 / (24)) + (5.52312723273835 / (13))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (13) - 2, .025) and -0.0566463225812124 = (2120.963582746532 - 2241.1083700377962) / 2120.963582746532 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
DrewScoggins commented 1 year ago

https://github.com/dotnet/runtime/pull/84399