search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.17k stars 4.72k forks source link

[Perf] Windows/x64: 59 Regressions due to PGO update #87179

Closed performanceautofiler[bot] closed 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ICollection - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsTrue(Int32).ICollection(Size%3a%20512).html>) 13.10 μs 17.22 μs 1.32 0.04 False
[List - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsTrue(Int32).List(Size%3a%20512).html>) 12.83 μs 17.39 μs 1.35 0.08 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsTrue<Int32>.ICollection(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 17.221965465701697 > 13.807469303835454. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -12.59767594136789 (T) = (0 -17214.049295440363) / Math.Sqrt((2013151.2654274935 / (25)) + (98.19219233095977 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.2621305908275654 = (13638.881285773525 - 17214.049295440363) / 13638.881285773525 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.ContainsTrue<Int32>.List(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 17.388277422049 > 13.959162170242886. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -12.904286364505177 (T) = (0 -17409.06331095206) / Math.Sqrt((2056912.7223882207 / (25)) + (11076.39780713844 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.27331935161706755 = (13672.189375621447 - 17409.06331095206) / 13672.189375621447 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Remove_IntInt - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.Remove_IntInt(s%3a%20%22dzsdzsDDZSDZSDZSddsz%22%2c%20i1%3a%2010%2c%20i2%3a%201).html>) 11.45 ns 16.29 ns 1.42 0.06 False
[Replace_String - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.Replace_String(text%3a%20%22This%20is%20a%20very%20nice%20sentence.%20This%20is%20another%20very%20nice%20sentence.%22%2c%20oldValue%3a%20%22a%22%2c%20newValue%3a%20%22%22).html>) 38.91 ns 46.95 ns 1.21 0.03 False
[TrimStart - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.TrimStart(s%3a%20%22%20Test%22).html>) 11.08 ns 16.17 ns 1.46 0.07 False
[ToLowerInvariant - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.ToLowerInvariant(s%3a%20%22TeSt%22).html>) 18.06 ns 22.96 ns 1.27 0.03 False
[TrimEnd - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.TrimEnd(s%3a%20%22Test%20%22).html>) 11.57 ns 14.91 ns 1.29 0.09 False
[Trim_CharArr - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.Trim_CharArr(s%3a%20%22%20Te%20st%20%20%22%2c%20c%3a%20%5b%27%20%27%2c%20%27%e2%80%85%27%5d).html>) 16.40 ns 19.69 ns 1.20 0.10 False
[TrimStart_CharArr - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.TrimStart_CharArr(s%3a%20%22%20Test%22%2c%20c%3a%20%5b%27%20%27%2c%20%27%e2%80%85%27%5d).html>) 12.41 ns 14.03 ns 1.13 0.05 False
[ToUpperInvariant - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.ToUpperInvariant(s%3a%20%22test%22).html>) 18.13 ns 27.33 ns 1.51 0.05 False
[Trim_CharArr - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.Trim_CharArr(s%3a%20%22%20Test%22%2c%20c%3a%20%5b%27%20%27%2c%20%27%e2%80%85%27%5d).html>) 13.18 ns 16.50 ns 1.25 0.05 False
[Replace_String - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.Replace_String(text%3a%20%22This%20is%20a%20very%20nice%20sentence%22%2c%20oldValue%3a%20%22nice%22%2c%20newValue%3a%20%22bad%22).html>) 35.69 ns 50.18 ns 1.41 0.03 False
[Remove_IntInt - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.Remove_IntInt(s%3a%20%22dzsdzsDDZSDZSDZSddsz%22%2c%20i1%3a%200%2c%20i2%3a%208).html>) 11.93 ns 15.23 ns 1.28 0.05 False
[ToLowerInvariant - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.ToLowerInvariant(s%3a%20%22TEST%22).html>) 18.59 ns 23.14 ns 1.24 0.04 False
[Trim_CharArr - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.Trim_CharArr(s%3a%20%22Test%20%22%2c%20c%3a%20%5b%27%20%27%2c%20%27%e2%80%85%27%5d).html>) 14.50 ns 16.73 ns 1.15 0.08 False
[ToLowerInvariant - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.ToLowerInvariant(s%3a%20%22This%20is%20a%20much%20longer%20piece%20of%20text%20that%20might%20benefit%20more%20from%20vectorization.%22).html>) 31.45 ns 35.73 ns 1.14 0.03 False
[Remove_IntInt - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.Remove_IntInt(s%3a%20%22dzsdzsDDZSDZSDZSddsz%22%2c%20i1%3a%207%2c%20i2%3a%204).html>) 11.28 ns 15.25 ns 1.35 0.06 False

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 10, i2: 1) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.286023356741666 > 12.91177764480207. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/2/2023 3:18:41 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -13.550925850566289 (T) = (0 -16.143226907608938) / Math.Sqrt((1.2931853427437008 / (24)) + (0.28293719304310155 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.34729928512542557 = (11.981916034421483 - 16.143226907608938) / 11.981916034421483 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 46.948632398824834 > 45.132741645284355. IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 3/17/2023 10:06:19 AM, 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -9.467008971783427 (T) = (0 -46.62551707133518) / Math.Sqrt((8.53770693900059 / (25)) + (0.573687182384257 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.15224457929309637 = (40.46494807546849 - 46.62551707133518) / 40.46494807546849 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.TrimStart(s: " Test") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.165391021408933 > 11.30719128424789. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -48.63622296556723 (T) = (0 -16.35964596795198) / Math.Sqrt((0.17202241041931612 / (24)) + (0.030943304456261533 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.47065447176134534 = (11.124058221751213 - 16.35964596795198) / 11.124058221751213 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToLowerInvariant(s: "TeSt") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 22.962254251870444 > 19.454100154148634. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -35.500729363171196 (T) = (0 -23.347707883502615) / Math.Sqrt((0.37532936199380573 / (24)) + (0.04925291202905812 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.29695458966437155 = (18.001947076300166 - 23.347707883502615) / 18.001947076300166 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.TrimEnd(s: "Test ") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 14.913651208547238 > 12.72516506002495. IsChangePoint: Marked as a change because one of 3/21/2023 10:36:00 PM, 4/19/2023 8:38:27 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -8.274444649292139 (T) = (0 -15.648369454951709) / Math.Sqrt((0.14975776255986786 / (24)) + (1.2964555813686132 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.3010052373317946 = (12.02790658017998 - 15.648369454951709) / 12.02790658017998 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Trim_CharArr(s: " Te st ", c: [' ', ' ']) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 19.68856913041714 > 17.4880858480232. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 5/22/2023 7:40:07 AM, 5/23/2023 1:41:12 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -19.497367065447687 (T) = (0 -19.924239585484628) / Math.Sqrt((0.31606642421014763 / (24)) + (0.12677602164963053 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.20929676797304597 = (16.475889221866108 - 19.924239585484628) / 16.475889221866108 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.TrimStart_CharArr(s: " Test", c: [' ', ' ']) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 14.02842604331755 > 13.737467144653055. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -7.813505745388528 (T) = (0 -14.142283264510453) / Math.Sqrt((0.7491159584776212 / (24)) + (0.11758300126126087 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1377322553843227 = (12.43023848324774 - 14.142283264510453) / 12.43023848324774 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToUpperInvariant(s: "test") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.33346013334883 > 18.687162821963323. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -3.161744711592882 (T) = (0 -22.108599390788534) / Math.Sqrt((0.21718701830331613 / (24)) + (14.017759538709216 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2544395865715673 = (17.62428388537403 - 22.108599390788534) / 17.62428388537403 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Trim_CharArr(s: " Test", c: [' ', ' ']) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.50055467177426 > 15.102117059623264. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -14.258750931932699 (T) = (0 -16.487820179957204) / Math.Sqrt((0.9421479652021646 / (24)) + (0.002043369491451209 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2077020133146914 = (13.652225464710694 - 16.487820179957204) / 13.652225464710694 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 50.183800855262206 > 42.44772404786201. IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -14.937590072600948 (T) = (0 -50.36156535756361) / Math.Sqrt((19.54562471648434 / (25)) + (0.4426933315480499 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.3748792685114141 = (36.62980925742682 - 50.36156535756361) / 36.62980925742682 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.234881995647653 > 12.548569291285613. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -7.291263627626198 (T) = (0 -14.559666237283732) / Math.Sqrt((0.11878048749863417 / (24)) + (0.8419137856409523 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.21537965450867097 = (11.979521117761033 - 14.559666237283732) / 11.979521117761033 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToLowerInvariant(s: "TEST") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.135248297997048 > 19.596599138145027. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -34.88193701168203 (T) = (0 -23.20114159891524) / Math.Sqrt((0.31180615878346046 / (24)) + (0.06536239609790322 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2897602741505068 = (17.98872400081987 - 23.20114159891524) / 17.98872400081987 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Trim_CharArr(s: "Test ", c: [' ', ' ']) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.733750673324913 > 15.749647040257416. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 5/23/2023 1:41:12 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -17.592966750486983 (T) = (0 -16.873858487586865) / Math.Sqrt((0.32950062302134553 / (24)) + (0.0043234044063571436 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1427040498573515 = (14.766604257413194 - 16.873858487586865) / 14.766604257413194 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 35.72884961714696 > 34.557854647388986. IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -11.669014811508955 (T) = (0 -35.66501082277905) / Math.Sqrt((2.398002263904492 / (24)) + (0.061138336249441684 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.12088405020055965 = (31.818644235679432 - 35.66501082277905) / 31.818644235679432 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 7, i2: 4) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.253034117136021 > 12.635653580588492. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -8.303553574006205 (T) = (0 -15.15861929406102) / Math.Sqrt((0.3863057974234546 / (24)) + (1.1840634869368947 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.30849969383118686 = (11.584732778712194 - 15.15861929406102) / 11.584732778712194 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Sort<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Array_ComparerStruct - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Sort(String).Array_ComparerStruct(Size%3a%20512).html>) 240.68 μs 282.80 μs 1.18 0.15 False
[LinqOrderByExtension - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Sort(String).LinqOrderByExtension(Size%3a%20512).html>) 213.08 μs 253.26 μs 1.19 0.07 False
[LinqQuery - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Sort(String).LinqQuery(Size%3a%20512).html>) 214.01 μs 249.26 μs 1.16 0.05 False
[Array_Comparison - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Sort(String).Array_Comparison(Size%3a%20512).html>) 233.63 μs 281.74 μs 1.21 0.13 False
[List - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Sort(String).List(Size%3a%20512).html>) 230.80 μs 269.02 μs 1.17 0.14 False
[Array - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Sort(String).Array(Size%3a%20512).html>) 224.32 μs 271.32 μs 1.21 0.17 False
[Array_ComparerClass - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Sort(String).Array_ComparerClass(Size%3a%20512).html>) 235.83 μs 284.99 μs 1.21 0.17 False

graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Sort<String>.Array_ComparerStruct(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 282.80378666666655 > 262.63437830000004. IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/20/2023 7:45:57 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -27.5218931740511 (T) = (0 -285219.21895238094) / Math.Sqrt((35488010.78479053 / (24)) + (3308863.3287484846 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15579322644767585 = (246773.5685119047 - 285219.21895238094) / 246773.5685119047 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.LinqOrderByExtension(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 253.26065866666667 > 234.69183755000003. IsChangePoint: Marked as a change because one of 4/25/2023 3:46:39 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -21.665681347818406 (T) = (0 -250614.69681946628) / Math.Sqrt((49150568.40816188 / (24)) + (2214237.6471708487 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15330552502037817 = (217301.21930616614 - 250614.69681946628) / 217301.21930616614 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.LinqQuery(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 249.26142933333333 > 232.7877699. IsChangePoint: Marked as a change because one of 4/25/2023 9:59:52 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -18.766436696394923 (T) = (0 -251000.35188278387) / Math.Sqrt((58408438.660455175 / (24)) + (5685731.849366357 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15567254107877324 = (217189.85522359583 - 251000.35188278387) / 217189.85522359583 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.Array_Comparison(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 281.73827066666666 > 253.89817395000003. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -34.11144897021536 (T) = (0 -280901.71866666665) / Math.Sqrt((32505894.62266493 / (24)) + (1933736.6810228333 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.18352986804596233 = (237342.31492648553 - 280901.71866666665) / 237342.31492648553 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.List(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 269.024396 > 246.05755494230766. IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/19/2023 8:24:48 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -32.467794440344896 (T) = (0 -269152.1215238095) / Math.Sqrt((34983085.52278327 / (24)) + (272014.575374633 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17311381076405472 = (229433.93816880343 - 269152.1215238095) / 229433.93816880343 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.Array(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 271.31517199999996 > 245.20233394999997. IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/20/2023 3:57:43 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -29.192584994534666 (T) = (0 -269557.5234034537) / Math.Sqrt((39448744.262925684 / (24)) + (1897646.3343733747 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17627491207936258 = (229162.0102029914 - 269557.5234034537) / 229162.0102029914 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.Array_ComparerClass(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 284.9897853333334 > 253.62559705000004. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -32.70111773399092 (T) = (0 -283583.14293877553) / Math.Sqrt((33488824.27026515 / (24)) + (3985248.79086351 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1927945385846994 = (237746.848904304 - 283583.14293877553) / 237746.848904304 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToString - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Int32.ToString(value%3a%2012345).html>) 9.97 ns 12.19 ns 1.22 0.06 False
[ToString - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Int32.ToString(value%3a%202147483647).html>) 12.18 ns 15.37 ns 1.26 0.07 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Int32.ToString(value: 12345) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.194173766352954 > 10.42855260869867. IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 3/31/2023 8:05:26 PM, 4/25/2023 6:46:56 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -24.017672147410114 (T) = (0 -12.13573175075449) / Math.Sqrt((0.16339912761450404 / (24)) + (0.0051315193111998525 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.20753412309473165 = (10.05001143955452 - 12.13573175075449) / 10.05001143955452 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Int32.ToString(value: 2147483647) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.366940449177877 > 13.379995387957416. IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 3/31/2023 4:22:12 PM, 4/9/2023 11:04:55 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -58.56776692202065 (T) = (0 -15.382709670426832) / Math.Sqrt((0.06363330067880928 / (24)) + (0.00037871781104181795 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.24694206139038322 = (12.33634676921122 - 15.382709670426832) / 12.33634676921122 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ForEach_Array - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array.html>) 3.55 ns 4.85 ns 1.36 0.17 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark*'
### Payloads [Baseline]() [Compare]() ### Histogram #### Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.845897894674565 > 3.72914748446663. IsChangePoint: Marked as a change because one of 3/25/2023 12:18:49 AM, 4/9/2023 11:04:55 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -39.80117946438802 (T) = (0 -4.8432498896594165) / Math.Sqrt((0.024104688773064764 / (25)) + (8.835308936469366E-06 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.34290009337618577 = (3.606560095980781 - 4.8432498896594165) / 3.606560095980781 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(%2c%20None)).html>) 29.84 ns 35.78 ns 1.20 0.07 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(en-US%2c%20None)).html>) 30.59 ns 35.03 ns 1.14 0.06 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(%2c%20IgnoreCase)).html>) 31.05 ns 34.90 ns 1.12 0.06 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(en-US%2c%20IgnoreCase)).html>) 30.57 ns 34.85 ns 1.14 0.07 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(pl-PL%2c%20None)).html>) 29.24 ns 35.40 ns 1.21 0.05 False
[Compare_Same_Upper - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count%3a%201024%2c%20Options%3a%20(en-US%2c%20OrdinalIgnoreCase)).html>) 1.06 μs 1.22 μs 1.15 0.03 False

graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringEquality*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, None)) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 35.777936054757255 > 31.8225536374239. IsChangePoint: Marked as a change because one of 3/21/2023 10:36:00 PM, 3/22/2023 9:02:35 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -25.9944892803942 (T) = (0 -35.12692066255957) / Math.Sqrt((0.825578600028122 / (25)) + (0.10713989745344599 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.19429200836063254 = (29.412338370058425 - 35.12692066255957) / 29.412338370058425 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, None)) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 35.027398082722215 > 32.24310684667808. IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 3/17/2023 10:06:19 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -24.9215516446299 (T) = (0 -35.1391844099929) / Math.Sqrt((1.0049650611505123 / (25)) + (0.05394692354217035 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.18375393369134166 = (29.684534437336264 - 35.1391844099929) / 29.684534437336264 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, IgnoreCase)) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 34.89905207112815 > 32.55239759268578. IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 3/17/2023 10:06:19 AM, 5/21/2023 4:09:07 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -18.599874696735537 (T) = (0 -34.93385628453139) / Math.Sqrt((1.461578163747821 / (25)) + (0.008834045992968482 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.14958305788440154 = (30.388283860777136 - 34.93385628453139) / 30.388283860777136 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, IgnoreCase)) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 34.84770300810809 > 32.36832380707033. IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 3/17/2023 10:06:19 AM, 5/25/2023 10:38:38 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -16.282833598294843 (T) = (0 -35.02266932753604) / Math.Sqrt((1.7929051406585863 / (25)) + (0.06097231859316187 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.15187479328425865 = (30.404927281791103 - 35.02266932753604) / 30.404927281791103 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (pl-PL, None)) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 35.399874882569996 > 31.431500723585202. IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 3/17/2023 10:06:19 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -21.82108344371345 (T) = (0 -35.185459162233364) / Math.Sqrt((1.1945592332826311 / (24)) + (0.11179093079518872 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.18908246512094118 = (29.59042807737869 - 35.185459162233364) / 29.59042807737869 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase)) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.216070719470538 > 1.2139448913586368. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -6.711893969776785 (T) = (0 -1289.006398188405) / Math.Sqrt((11506.51789963786 / (24)) + (1765.8431492360558 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1639411709935116 = (1107.4497838135073 - 1289.006398188405) / 1107.4497838135073 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Reader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadReturnBytes - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact%3a%20False%2c%20TestCase%3a%20DeepTree).html>) 6.62 μs 9.25 μs 1.40 0.52 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Reader*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: DeepTree) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.247130770735524 > 7.127696773203326. IsChangePoint: Marked as a change because one of 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -8.994528545319696 (T) = (0 -8866.647675742965) / Math.Sqrt((168433.7183788124 / (24)) + (270592.0478579049 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2768086567869769 = (6944.3824872361265 - 8866.647675742965) / 6944.3824872361265 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToString - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_UInt32.ToString(value%3a%2012345).html>) 11.36 ns 13.81 ns 1.22 0.07 False
[ToString - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_UInt32.ToString(value%3a%204294967295).html>) 11.82 ns 16.83 ns 1.42 0.10 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt32*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_UInt32.ToString(value: 12345) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 13.805227304194526 > 11.907051987488424. IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -13.15791615034516 (T) = (0 -13.783123848207003) / Math.Sqrt((1.0947929629492876 / (24)) + (0.0017769354453144226 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2570052281164397 = (10.965048943241337 - 13.783123848207003) / 10.965048943241337 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_UInt32.ToString(value: 4294967295) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.828897958496626 > 12.518416905356082. IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 3/31/2023 8:05:26 PM, 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -19.308874539302327 (T) = (0 -17.241301055312608) / Math.Sqrt((0.0725864063081279 / (24)) + (0.5436750537920058 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.46655259782795455 = (11.756346878283075 - 17.241301055312608) / 11.756346878283075 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_OrderBy

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[OrderByString - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Linq.Tests.Perf_OrderBy.OrderByString(NumberOfPeople%3a%20512).html>) 229.73 μs 267.57 μs 1.16 0.05 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_OrderBy*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Linq.Tests.Perf_OrderBy.OrderByString(NumberOfPeople: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 267.56547366828084 > 249.77837025201143. IsChangePoint: Marked as a change because one of 3/10/2023 12:53:36 AM, 4/25/2023 6:46:56 PM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -15.730206654781929 (T) = (0 -260848.48772384392) / Math.Sqrt((33086084.44142996 / (24)) + (10881201.970933482 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.11517210717257442 = (233908.7267751015 - 260848.48772384392) / 233908.7267751015 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorFromCollectionNonGeneric<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SortedList - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.CtorFromCollectionNonGeneric(String).SortedList(Size%3a%20512).html>) 239.24 μs 261.47 μs 1.09 0.08 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CtorFromCollectionNonGeneric<String>.SortedList(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 261.4689342948718 > 252.28922403846158. IsChangePoint: Marked as a change because one of 5/10/2023 12:32:34 AM, 5/13/2023 6:35:24 AM, 5/23/2023 8:43:31 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -9.460652268457213 (T) = (0 -266712.15221685596) / Math.Sqrt((22962302.274779834 / (24)) + (72462445.52526964 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.13543957513952068 = (234897.70662968382 - 266712.15221685596) / 234897.70662968382 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Perf_Convert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToHexString - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Perf_Convert.ToHexString.html>) 35.61 ns 41.11 ns 1.15 0.01 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Perf_Convert*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Perf_Convert.ToHexString ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 41.108947363361395 > 37.8864965239414. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -67.86532446338427 (T) = (0 -41.16012128464307) / Math.Sqrt((0.13543207802879936 / (25)) + (0.0049278608384559055 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.14810644868855471 = (35.8504399410516 - 41.16012128464307) / 35.8504399410516 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EnumerateUsingIndexer - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase%3a%20ArrayOfNumbers).html>) 1.35 μs 1.43 μs 1.06 0.03 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfNumbers) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.4280076392795011 > 1.4129753762867647. IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 3/25/2023 12:18:49 AM, 4/6/2023 1:55:21 PM, 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -12.634445529896837 (T) = (0 -1428.070834754199) / Math.Sqrt((779.3334737013729 / (24)) + (0.19981601918857617 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.05311653235372219 = (1356.0425564324316 - 1428.070834754199) / 1356.0425564324316 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Email_IsMatch - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options%3a%20Compiled).html>) 104.09 ns 120.91 ns 1.16 0.12 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 120.9061087231152 > 108.18101259099014. IsChangePoint: Marked as a change because one of 3/25/2023 12:18:49 AM, 3/28/2023 8:13:28 AM, 4/19/2023 8:38:27 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -24.2530373590888 (T) = (0 -121.52991415282663) / Math.Sqrt((4.961668062545958 / (24)) + (2.3169952998706913 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17142852719271842 = (103.74505258470033 - 121.52991415282663) / 103.74505258470033 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[DeserializeFromUtf8Bytes - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.Json.Serialization.Tests.ReadJson(MyEventsListerViewModel).DeserializeFromUtf8Bytes(Mode%3a%20Reflection).html>) 218.31 μs 296.62 μs 1.36 0.09 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.DeserializeFromUtf8Bytes(Mode: Reflection) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 296.6232204861111 > 235.87053683035717. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -6.143205658751805 (T) = (0 -280767.39393674134) / Math.Sqrt((21901424.551616486 / (24)) + (549701872.2299329 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.24226181005016648 = (226013.0607455469 - 280767.39393674134) / 226013.0607455469 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyTrue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SortedList - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsKeyTrue(String%2c%20String).SortedList(Size%3a%20512).html>) 178.83 μs 214.91 μs 1.20 0.05 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue&lt;String, String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsKeyTrue<String, String>.SortedList(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 214.90989155251143 > 196.96154535520128. IsChangePoint: Marked as a change because one of 4/4/2023 4:23:54 PM, 5/13/2023 1:49:08 AM, 5/23/2023 8:43:31 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -25.043793111382673 (T) = (0 -216468.02785893442) / Math.Sqrt((22470899.648432195 / (24)) + (7303704.540361785 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.19443031901101895 = (181231.18980952245 - 216468.02785893442) / 181231.18980952245 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrueComparer<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SortedSet - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsTrueComparer(String).SortedSet(Size%3a%20512).html>) 217.29 μs 258.24 μs 1.19 0.08 False
[ImmutableSortedSet - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsTrueComparer(String).ImmutableSortedSet(Size%3a%20512).html>) 218.41 μs 258.39 μs 1.18 0.09 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrueComparer&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsTrueComparer<String>.SortedSet(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 258.24418715846997 > 232.50905164930558. IsChangePoint: Marked as a change because one of 4/4/2023 4:23:54 PM, 5/13/2023 2:42:33 PM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -48.366713596759695 (T) = (0 -258794.92201104906) / Math.Sqrt((14381339.742568674 / (25)) + (686664.9440006126 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.18113892478834256 = (219106.25124595274 - 258794.92201104906) / 219106.25124595274 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.ContainsTrueComparer<String>.ImmutableSortedSet(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 258.3912773224044 > 241.90896277573538. IsChangePoint: Marked as a change because one of 3/10/2023 12:53:36 AM, 4/4/2023 4:23:54 PM, 4/9/2023 11:04:55 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -27.511509267728616 (T) = (0 -258727.78465484554) / Math.Sqrt((36405436.53794992 / (25)) + (1207056.8860890076 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.1570081518638032 = (223617.94447002443 - 258727.78465484554) / 223617.94447002443 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueTrue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SortedDictionary - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.TryGetValueTrue(String%2c%20String).SortedDictionary(Size%3a%20512).html>) 198.46 μs 241.93 μs 1.22 0.09 False
[SortedList - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.TryGetValueTrue(String%2c%20String).SortedList(Size%3a%20512).html>) 177.71 μs 215.00 μs 1.21 0.06 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue&lt;String, String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.TryGetValueTrue<String, String>.SortedDictionary(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 241.9265576923077 > 213.25930154896741. IsChangePoint: Marked as a change because one of 5/8/2023 9:08:59 PM, 5/13/2023 1:49:08 AM, 5/21/2023 1:46:51 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -8.947397728207946 (T) = (0 -236757.29316406048) / Math.Sqrt((15501942.451120304 / (24)) + (117139536.39338888 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1870148464190742 = (199456.0505104867 - 236757.29316406048) / 199456.0505104867 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.TryGetValueTrue<String, String>.SortedList(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 215.00434360730597 > 195.3036030159884. IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/19/2023 3:32:16 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -33.163794837002506 (T) = (0 -214951.60721137514) / Math.Sqrt((17237805.88271307 / (24)) + (1721084.4632037359 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1785379970854021 = (182388.35552435633 - 214951.60721137514) / 182388.35552435633 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndRemove<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SortedList - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.CreateAddAndRemove(String).SortedList(Size%3a%20512).html>) 443.62 μs 516.32 μs 1.16 0.08 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CreateAddAndRemove<String>.SortedList(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 516.3165477667494 > 493.0252450284091. IsChangePoint: Marked as a change because one of 5/20/2023 7:45:57 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -18.77546180360394 (T) = (0 -516995.3329417478) / Math.Sqrt((159313991.32457975 / (24)) + (26255949.173059206 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.13257320563818242 = (456478.51314867655 - 516995.3329417478) / 456478.51314867655 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SortedList - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.IndexerSet(String).SortedList(Size%3a%20512).html>) 178.61 μs 215.41 μs 1.21 0.08 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.IndexerSet<String>.SortedList(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 215.40935502283102 > 194.33377322036304. IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 5/13/2023 1:49:08 AM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -38.14898230442877 (T) = (0 -214752.4881846482) / Math.Sqrt((16151506.236441173 / (25)) + (491334.4019260547 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.17694270276452084 = (182466.3916775354 - 214752.4881846482) / 182466.3916775354 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SortedSet - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsTrue(String).SortedSet(Size%3a%20512).html>) 175.73 μs 216.13 μs 1.23 0.07 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsTrue<String>.SortedSet(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 216.13018835616438 > 191.13948797686692. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -18.20021333022759 (T) = (0 -217710.74406468167) / Math.Sqrt((50278012.85131415 / (24)) + (13664011.670386543 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.20217449963520906 = (181097.4564264542 - 217710.74406468167) / 181097.4564264542 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_FromString<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SystemTextJsonReflection - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/MicroBenchmarks.Serializers.Json_FromString(MyEventsListerViewModel).SystemTextJsonReflection.html>) 229.98 μs 247.97 μs 1.08 0.08 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;MyEventsListerViewModel&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### MicroBenchmarks.Serializers.Json_FromString<MyEventsListerViewModel>.SystemTextJson_Reflection_ ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 247.9710018382353 > 242.90568001383429. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -3.023498789448895 (T) = (0 -288314.05399125116) / Math.Sqrt((69841315.17357163 / (25)) + (2421190311.3286247 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.24350214234159073 = (231856.49961835862 - 288314.05399125116) / 231856.49961835862 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToString_Format_Flags_Large - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Enum.ToString_Format_Flags_Large(value%3a%20All%2c%20format%3a%20%22d%22).html>) 16.11 ns 23.19 ns 1.44 0.12 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Enum.ToString_Format_Flags_Large(value: All, format: "d") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.189677808061752 > 17.991915259990463. IsChangePoint: Marked as a change because one of 4/11/2023 3:40:00 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -17.5478289766965 (T) = (0 -22.07517190443852) / Math.Sqrt((0.3551127890941325 / (24)) + (0.5883041066815936 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.3331750046348609 = (16.558345174259113 - 22.07517190443852) / 16.558345174259113 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Get

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetDecimal - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.Json.Tests.Perf_Get.GetDecimal.html>) 3.61 μs 4.29 μs 1.19 0.02 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Tests.Perf_Get.GetDecimal ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.287676877528575 > 4.158444303203006. IsChangePoint: Marked as a change because one of 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -9.925445162083971 (T) = (0 -4311.1949156006995) / Math.Sqrt((74811.6585864201 / (24)) + (864.9813857869368 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15082864716180078 = (3746.1657964746223 - 4311.1949156006995) / 3746.1657964746223 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToString_MultipleSegments - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length%3a%20100).html>) 23.87 ns 31.37 ns 1.31 0.07 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length: 100) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 31.365683172667303 > 25.52096050313922. IsChangePoint: Marked as a change because one of 3/17/2023 10:06:19 AM, 5/10/2023 6:18:38 PM, 5/11/2023 5:02:01 PM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -8.67313164952705 (T) = (0 -29.365274089750752) / Math.Sqrt((0.9370196294302815 / (25)) + (2.578172603950685 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.23174736909741914 = (23.84033838957463 - 29.365274089750752) / 23.84033838957463 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Asinh - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.MathBenchmarks.Single.Asinh.html>) 89.01 μs 94.61 μs 1.06 0.01 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.MathBenchmarks.Single.Asinh ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 94.61360943775098 > 94.44397185304918. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -32.28112493260026 (T) = (0 -94597.97863746982) / Math.Sqrt((630456.5711809898 / (24)) + (1055.9468095413195 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.05872389948211555 = (89350.94285086346 - 94597.97863746982) / 89350.94285086346 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Add_Remove_SteadyState<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ConcurrentStack - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Tests.Add_Remove_SteadyState(String).ConcurrentStack(Count%3a%20512).html>) 14.60 ns 16.98 ns 1.16 0.05 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Add_Remove_SteadyState&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Tests.Add_Remove_SteadyState<String>.ConcurrentStack(Count: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.98158993663234 > 15.703474968011061. IsChangePoint: Marked as a change because one of 3/8/2023 9:36:36 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -28.532097498797846 (T) = (0 -17.041236485949234) / Math.Sqrt((0.07120845961853098 / (24)) + (0.02820371434351972 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.16284877589279093 = (14.65473141412186 - 17.041236485949234) / 14.65473141412186 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryParseDecimal - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Buffers.Text.Tests.Utf8ParserTests.TryParseDecimal(value%3a%20123456.789).html>) 34.24 ns 39.61 ns 1.16 0.02 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Text.Tests.Utf8ParserTests.TryParseDecimal(value: 123456.789) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 39.60591479240837 > 39.12622222666717. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -7.705811960885725 (T) = (0 -39.21589997080073) / Math.Sqrt((5.469400193594025 / (24)) + (0.30444971130660026 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.11403775020011185 = (35.20158985973004 - 39.21589997080073) / 35.20158985973004 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern%3a%20%22%5b%5c%5cw%5d%2b%3a%2f%2f%5b%5e%2f%5c%5cs%3f%23%5d%2b%5b%5e%5c%5cs%3f%23%5d%2b(%3f%3a%5c%5c%3f%5b%5e%5c%5cs%23%5d)%3f(%3f%3a%23%5b%5e%5c%5cs%5d)%3f%22%2c%20Options%3a%20NonBacktracking).html>) 3.10 ms 4.35 ms 1.40 0.05 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\\w]+://[^/\\s?#]+[^\\s?#]+(?:\\?[^\\s#]*)?(?:#[^\\s]*)?", Options: NonBacktracking) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.354508076923076 > 3.066873263653233. IsChangePoint: Marked as a change because one of 5/23/2023 8:43:31 PM, 5/25/2023 6:27:07 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -10.63618348495044 (T) = (0 -4332207.307692307) / Math.Sqrt((170888892553.52792 / (25)) + (29159411450.345142 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.34682189849605943 = (3216614.8415985103 - 4332207.307692307) / 3216614.8415985103 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 0214a047483067a6a813a34be5b6b1f757600961
Compare 2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EncodeUtf8 - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments%3a%20UnsafeRelaxed%2cno%20(escaping%20%2f)%20required%2c16).html>) 13.05 ns 15.54 ns 1.19 0.06 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: UnsafeRelaxed,no <escaping /> required,16) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.538433351486187 > 13.968079092640599. IsChangePoint: Marked as a change because one of 5/1/2023 4:40:42 AM, 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -10.7443895509019 (T) = (0 -15.314357393532148) / Math.Sqrt((0.18450410126129144 / (25)) + (0.22707380932608648 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.16279140889120783 = (13.170339302846516 - 15.314357393532148) / 13.170339302846516 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
kunalspathak commented 1 year ago

Similar regression: https://github.com/dotnet/perf-autofiling-issues/issues/18510

kunalspathak commented 1 year ago

https://github.com/dotnet/runtime/pull/86698

kunalspathak commented 1 year ago

Similar issue https://github.com/dotnet/perf-autofiling-issues/issues/18511

AndyAyersMS commented 1 year ago

With Dynamic PGO enabled we would not expect a static PGO update to have much impact on benchmarks. Will have to investigate.

AndyAyersMS commented 1 year ago

Using the autofiling collator, there is very little overlap between the reports. Given this is all x64 and so using identical PGO data it seems hard to understand why we'd see this kind of pattern.

Notes Recent Score Orig Score x64-win-intel x64-win-amd Benchmark
noise 2.05 1.15 2.05
1.15		 System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
still analyzing 1.46 1.19 1.46
1.19		 System.Text.Json.Tests.Perf_Get.GetDecimal
noise 1.43 1.44 1.43
1.44		 System.Collections.CtorFromCollection(Int32).SortedSet(Size: 512)
noise 1.41 1.36 1.41
1.36		 Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array
newly noisy
#87324		 1.28 1.32 1.28
1.32		 System.Collections.ContainsTrue(Int32).ICollection(Size: 512)
1.16 1.16 1.16
1.16		 System.Buffers.Text.Tests.Utf8ParserTests.TryParseDecimal(value: 123456.789)
1.13 1.21 1.13
1.21		 System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "")
1.12 1.15 1.12
1.15		 System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreNonSpace, False))
1.12 1.15 1.12
1.15		 System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, None, False))
1.12 1.15 1.12
1.15		 System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, False))
1.10 1.10 1.10
1.10		 Span.IndexerBench.WriteViaIndexer1(length: 1024)
1.09 1.42 1.09
1.42		 System.Tests.Perf_UInt32.ToString(value: 4294967295)
1.09 1.41 1.09
1.41		 System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")
1.06 1.40 1.06
1.40		 System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: DeepTree)
1.05 1.26 1.05
1.26		 System.Tests.Perf_Int32.ToString(value: 2147483647)
1.03 1.11 1.03
1.11		 System.Collections.ContainsTrue(Int32).Stack(Size: 512)
1.02 1.20 1.06
1.27		 0.99
1.14		 System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, IgnoreCase))
1.02 1.36 1.02
1.36		 System.Text.Json.Serialization.Tests.ReadJson(MyEventsListerViewModel).DeserializeFromUtf8Bytes(Mode: Reflection)
1.02 1.19 1.02
1.19		 System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: UnsafeRelaxed,no (escaping /) required,16)
1.02 1.17 1.02
1.17		 System.Reflection.Invoke.StaticMethod4_arrayNotCached_int_string_struct_class
1.01 1.35 1.01
1.35		 System.Collections.ContainsTrue(Int32).List(Size: 512)
1.01 1.14 0.99
1.13		 1.03
1.16		 System.Linq.Tests.Perf_OrderBy.OrderByString(NumberOfPeople: 512)
1.00 1.46 1.00
1.46		 MicroBenchmarks.Serializers.Json_FromStream(MyEventsListerViewModel).SystemTextJsonSourceGen
1.00 1.34 1.00
1.34		 System.Text.Json.Serialization.Tests.ReadJson(MyEventsListerViewModel).DeserializeFromString(Mode: SourceGen)
1.00 1.06 1.00
1.06		 System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfNumbers)
1.00 1.06 1.00
1.06		 System.MathBenchmarks.Single.Asinh
0.99 1.10 0.99
1.10		 System.Collections.AddGivenSize(String).HashSet(Size: 512)
0.99 1.15 0.99
1.15		 System.Perf_Convert.ToHexString
0.99 1.21 1.01
1.23		 0.98
1.20		 System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, None))
0.99 1.17 0.99
1.17		 System.Collections.Sort(String).List(Size: 512)
0.99 1.16 0.99
1.16		 System.Collections.Tests.Add_Remove_SteadyState(String).ConcurrentStack(Count: 512)
0.99 1.26 0.99
1.31		 0.99
1.21		 System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (pl-PL, None))
0.99 1.13 1.00
1.17		 0.98
1.09		 System.Collections.CtorFromCollectionNonGeneric(String).SortedList(Size: 512)
0.99 1.31 0.99
1.31		 System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, IgnoreNonSpace))
0.98 1.20 0.94
1.20		 1.02
1.21		 System.Collections.Sort(String).Array_ComparerClass(Size: 512)
0.98 1.17 0.99
1.19		 0.97
1.16		 System.Collections.Sort(String).LinqQuery(Size: 512)
0.98 1.17 0.98
1.16		 0.97
1.19		 System.Collections.Sort(String).LinqOrderByExtension(Size: 512)
0.98 1.13 0.97
1.19		 0.99
1.08		 MicroBenchmarks.Serializers.Json_FromString(MyEventsListerViewModel).SystemTextJsonReflection
0.98 1.16 0.98
1.16		 System.Collections.CreateAddAndRemove(String).SortedList(Size: 512)
0.97 1.16 0.97
1.16		 System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: Compiled)
0.97 1.21 0.99
1.21		 0.95
1.21		 System.Collections.TryGetValueTrue(String, String).SortedList(Size: 512)
0.96 1.18 0.96
1.18		 System.Collections.ContainsTrueComparer(String).ImmutableSortedSet(Size: 512)
0.96 1.18 0.96
1.18		 System.Collections.Sort(String).Array_ComparerStruct(Size: 512)
0.96 1.17 1.00
1.21		 0.92
1.14		 System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, None))
0.96 1.21 0.96
1.22		 0.96
1.21		 System.Collections.Sort(String).Array(Size: 512)
0.96 1.23 0.96
1.23		 System.Collections.ContainsKeyTrue(String, String).ImmutableSortedDictionary(Size: 512)
0.96 1.21 0.96
1.21		 0.96
1.21		 System.Collections.IndexerSet(String).SortedList(Size: 512)
0.95 1.20 0.95
1.20		 System.Text.Tests.Perf_StringBuilder.Insert_Strings
0.95 1.19 0.95
1.19		 System.Collections.ContainsTrueComparer(String).SortedSet(Size: 512)
0.95 1.31 0.95
1.31		 System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length: 100)
0.94 1.16 0.99
1.21		 0.90
1.12		 System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, IgnoreCase))
0.94 1.14 0.94
1.14		 System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
0.94 1.20 0.91
1.21		 0.96
1.20		 System.Collections.ContainsKeyTrue(String, String).SortedList(Size: 512)
0.93 1.22 0.93
1.22		 System.Tests.Perf_Int32.ToString(value: 12345)
0.93 1.26 0.93
1.26		 System.Collections.TryGetValueTrue(String, String).ImmutableSortedDictionary(Size: 512)
0.92 1.23 0.92
1.23		 System.Collections.ContainsTrue(String).SortedSet(Size: 512)
0.91 1.22 0.91
1.22		 System.Collections.ContainsKeyTrue(String, String).SortedDictionary(Size: 512)
0.91 1.22 0.91
1.22		 System.Collections.TryGetValueTrue(String, String).SortedDictionary(Size: 512)
0.90 1.40 0.90
1.40		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w]+://[^/\s?#]+[^\s?#]+(?:\?[^\s#])?(?:#[^\s])?", Options: NonBacktracking)
0.90 1.16 0.87
1.12		 0.93
1.21		 System.Collections.Sort(String).Array_Comparison(Size: 512)
0.90 1.44 0.90
1.44		 System.Tests.Perf_Enum.ToString_Format_Flags_Large(value: All, format: "d")
0.90 1.15 0.90
1.15		 System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)
0.89 1.18 0.98
1.24		 0.81
1.13		 System.Tests.Perf_String.TrimStart_CharArr(s: " Test", c: [' ', ' '])
0.86 1.29 0.86
1.29		 System.Tests.Perf_String.TrimEnd(s: "Test ")
0.85 1.14 0.85
1.14		 System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture)
0.84 1.27 0.84
1.27		 System.Tests.Perf_String.ToLowerInvariant(s: "TeSt")
0.83 1.44 1.01
1.42		 0.69
1.46		 System.Tests.Perf_String.TrimStart(s: " Test")
0.82 1.51 0.82
1.51		 System.Tests.Perf_String.ToUpperInvariant(s: "test")
0.81 1.24 0.81
1.24		 System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, None, False))
0.80 1.18 0.80
1.18		 System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture)
0.80 1.42 0.80
1.42		 System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 10, i2: 1)
0.79 1.35 0.79
1.35		 System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 7, i2: 4)
0.79 1.24 0.79
1.24		 System.Tests.Perf_String.ToLowerInvariant(s: "TEST")
0.77 1.22 0.77
1.22		 System.Tests.Perf_UInt32.ToString(value: 12345)
0.75 1.20 0.75
1.20		 System.Tests.Perf_String.Trim_CharArr(s: " Te st ", c: [' ', ' '])
0.74 1.28 0.74
1.28		 System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8)
0.73 1.25 0.73
1.25		 System.Tests.Perf_String.Trim_CharArr(s: " Test", c: [' ', ' '])
0.70 1.15 0.70
1.15		 System.Tests.Perf_String.Trim_CharArr(s: "Test ", c: [' ', ' '])

Also most of these are already mitigated, eg newplot - 2023-06-07T085156 118

It looks like we took another Static PGO update on June 5, so perhaps it was static PGO both times?

Looking at the worst-case for intel, this one has persisted. Note the benchmark became noisier once dynamic PGO was introduced -- I had hopes that we'd see an overall reduction in noise with dynamic PGO, but am now starting to wonder if that was just wishful thinking.

The jit is still fairly sensitive to small details in the PGO data, and there is an inherent degree of randomness in live PGO data, both from nondeterminisim in the test, and deliberate nondeterminism in the collection.

newplot - 2023-06-07T085558 850

So, two tasks for follow-up:

AndyAyersMS commented 1 year ago

Looking at the example just above, it spends ~15% of benchmark intervals in corelib (byref write barrier, memcpy ,etc)

00.89%   4.6E+05     ?        Unknown
18.36%   9.53E+06    Tier-1   [System.Text.Json]System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1[System.__Canon].OnTryRead(value class System.Text.Json.Utf8JsonReader&,class System.Type,class System.Text.Json.JsonSerializerOptions,value class System.Text.Json.ReadStack&,!0&)
14.81%   7.69E+06    native   coreclr.dll
09.24%   4.8E+06     Tier-1   [System.Text.Json]System.Text.Json.Serialization.JsonConverter`1[System.__Canon].TryRead(value class System.Text.Json.Utf8JsonReader&,class System.Type,class System.Text.Json.JsonSerializerOptions,value class System.Text.Json.ReadStack&,!0&,bool&)
08.13%   4.22E+06    Tier-1   [System.Text.Json]JsonHelpers.TryParseDateTimeOffset(value class System.ReadOnlySpan`1<unsigned int8>,value class DateTimeParseData&)
07.24%   3.76E+06    Tier-1   [System.Text.Json]Utf8JsonReader.ReadSingleSegment()
06.97%   3.62E+06    Tier-1   [System.Private.CoreLib]IndexOfAnyAsciiSearcher.IndexOfAnyVectorized(unsigned int8&,int32,value class System.Runtime.Intrinsics.Vector256`1<unsigned int8>&)
05.60%   2.91E+06    Tier-1   [System.Text.Json]Utf8JsonReader.ConsumeNextToken(unsigned int8)
04.89%   2.54E+06    Tier-1   [System.Text.Json]Utf8JsonReader.TryGetDateTimeOffsetCore(value class System.DateTimeOffset&)
03.51%   1.82E+06    Tier-1   [System.Text.Json]Utf8JsonReader.ConsumeValue(unsigned int8)
02.70%   1.4E+06     Tier-1   [System.Text.Json]System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1[System.Nullable`1[System.DateTimeOffset]].ReadJsonAndSetMember(class System.Object,value class System.Text.Json.ReadStack&,value class System.Text.Json.Utf8JsonReader&)
02.35%   1.22E+06    Tier-1   [System.Text.Json]JsonHelpers.TryCreateDateTimeOffset(value class System.DateTime,value class DateTimeParseData&,value class System.DateTimeOffset&)
02.10%   1.09E+06    Tier-1   [System.Text.Json]System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1[System.__Canon].ReadJsonAndSetMember(class System.Object,value class System.Text.Json.ReadStack&,value class System.Text.Json.Utf8JsonReader&)
01.56%   8.1E+05     Tier-1   [System.Text.Json]System.Text.Json.Serialization.JsonCollectionConverter`2[System.__Canon,System.__Canon].OnTryRead(value class System.Text.Json.Utf8JsonReader&,class System.Type,class System.Text.Json.JsonSerializerOptions,value class System.Text.Json.ReadStack&,!0&)
01.17%   6.1E+05     Tier-1   [System.Text.Json]System.Text.Json.Serialization.Converters.ListOfTConverter`2[System.__Canon,System.__Canon].Add(required_modifier System.Runtime.InteropServices.InAttribute !1&,value class System.Text.Json.ReadStack&)
01.10%   5.7E+05     Tier-1   [System.Private.CoreLib]Utf8Utility.TranscodeToUtf16(unsigned int8*,int32,wchar*,int32,unsigned int8*&,wchar*&)
01.04%   5.4E+05     Tier-1   [System.Text.Json]JsonReaderHelper.TranscodeHelper(value class System.ReadOnlySpan`1<unsigned int8>)

My guess is that https://github.com/dotnet/runtime/pull/87076 invalidated native PGO for important parts of corelib, and this destablized perf for a time, and then https://github.com/dotnet/runtime/pull/87090 produced matching PGO data and things went back to "normal."

danmoseley commented 1 year ago

Naive q, could we imagine collecting data with and without new PGO on otherwise identical bits every time PGO data updates? That would immediately show how much of any change was the PGO update.

AndyAyersMS commented 1 year ago
Locally I consistently see SortedSet as faster with the "diff" compiler (which is at 0071d924c2537c2372101fa9fe64a2b37b9e2ca8) Method Job Toolchain Size Mean Error StdDev Median Min Max Ratio RatioSD Gen0 Gen1 Allocated Alloc Ratio
SortedSet Job-VCLHWJ \base-rel\corerun.exe 512 7.859 us 0.5589 us 0.6437 us 7.936 us 7.071 us 8.937 us 1.00 0.00 3.5734 0.4810 22.07 KB 1.00
SortedSet Job-QWFOGA \diff-rel\corerun.exe 512 7.316 us 0.1808 us 0.2082 us 7.280 us 7.074 us 7.779 us 0.94 0.09 3.5734 0.4810 22.07 KB 1.00

Since this benchmark is pretty noisy I think we can write this off (for now) as noise.

AndyAyersMS commented 1 year ago

As for the (implied) hypothesis that the benchmarks above all spend significant time in native code: System.Text.Json.Tests.Perf_Get.GetDecimal does not, and is still regressed, at least for x86 and amd64 (but not intel):

image 
38.51%   1.858E+07   Tier-1   [System.Private.CoreLib]Utf8Parser.TryParseNumber(value class System.ReadOnlySpan`1<unsigned int8>,value class NumberBuffer&,int32&,value class ParseNumberOptions,bool&)
22.26%   1.074E+07   Tier-1   [System.Private.CoreLib]Number.TryNumberToDecimal(value class NumberBuffer&,value class System.Decimal&)
22.22%   1.072E+07   Tier-1   [System.Text.Json]Utf8JsonReader.TryGetDecimalCore(value class System.Decimal&,value class System.ReadOnlySpan`1<unsigned int8>)
07.73%   3.73E+06    Tier-1   [System.Private.CoreLib]Decimal+DecCalc.DecAddSub(value class DecCalc&,value class DecCalc&,bool)
07.46%   3.6E+06     Tier-1   [MicroBenchmarks]Perf_Get.GetDecimal()
00.41%   2E+05       Tier-1   [System.Text.Json]Utf8JsonReader.TryGetNumber(value class System.ReadOnlySpan`1<unsigned int8>,int32&)
00.35%   1.7E+05     native   clrjit.dll
00.35%   1.7E+05     native   coreclr.dll
00.29%   1.4E+05     Tier-1   [System.Text.Json]Utf8JsonReader..ctor(value class System.ReadOnlySpan`1<unsigned int8>,bool,value class System.Text.Json.JsonReaderState)
00.25%   1.2E+05     native   ntoskrnl.exe
00.06%   3E+04       Tier-1   [System.Private.CoreLib]SpanHelpers.NonPackedIndexOfValueType(!!0&,!!0,int32)

Finally had time to drill in. Looking now with base f4ef5c8af72304f679663ba134035206baf84095 and diff 0071d924c2537c2372101fa9fe64a2b37b9e2ca8.

I can repro a small regression locally:

Method Job Toolchain Mean Error StdDev Median Min Max Ratio RatioSD Allocated Alloc Ratio
GetDecimal Job-TEKZDW \base-rel\corerun.exe 3.681 us 0.0491 us 0.0459 us 3.698 us 3.606 us 3.735 us 1.00 0.00 - NA
GetDecimal Job-NKOPKC \diff-rel\corerun.exe 3.878 us 0.0607 us 0.0568 us 3.895 us 3.748 us 3.947 us 1.05 0.02 - NA

There are a few surprises:

For the stale PGO, looks like the optimization repo is running ~2 week old builds? EG the update in https://github.com/dotnet/runtime/pull/86698 on June 2nd brought in data from the May 23 build 20230523.4, and that build used the runtime bits from dotnet-sdk-8.0.100-preview.5.23273.15 (also May 23).

So the recent PGO updates are:

The diff sync point is 02-Jun so PGO data should have been compatible once more by 06-Jun.

image

Thus if the regression was related to static PGO we'd expect it to be fixed by now. But that does not seem to be the case.

Looking at profiles it appears there is significant time spent in Memmove that does not show up in the baseline run. But I am not sure this makes sense as the apparent caller is TryParseNumber and the codegen for that method in base/diff is quite similar (save for some block ordering diffs caused by the stale PGO). Perhaps the call comes from a callee that has tail called into Memmove so will check for that next.

07.10%   3.84E+06    Tier-1   [System.Private.CoreLib]Buffer.Memmove(unsigned int8&,unsigned int8&,unsigned int)

Poked at this for quite a while and still have no idea why the base/diff profiles diverge. Did a second profile run with similar results; both show the diff run is invoking Memmove while the base run doesn't seem to be doing this. This is with --apples mode in BDN so the two sets of runs should be doing the same amount of work. Memmove tails calls to something but don't see unexpected samples piling up anywhere else or attributed to nothing.

Here's what PerfView shows for the first BDN interval of each run

BASE -- 236.986ms

image

DIFF -- 260.216ms

image

and PerfView's regression report

image

(not sure what it does when there is a novel entry on one side, perhaps it attributes it to the common parent).

At any rate the only difference in codegen for TryParseNumber is layout, and the layout difference comes from static PGO, so perhaps that is the culprit.

ghost commented 1 year ago

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [0214a047483067a6a813a34be5b6b1f757600961](https://github.com/dotnet/runtime/commit/0214a047483067a6a813a34be5b6b1f757600961) Compare | [2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc](https://github.com/dotnet/runtime/commit/2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc) Diff | [Diff](https://github.com/dotnet/runtime/compare/0214a047483067a6a813a34be5b6b1f757600961...2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Collections.ContainsTrue<Int32> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ICollection - Duration of single invocation]() | 13.10 μs | 17.22 μs | 1.32 | 0.04 | False | | | [List - Duration of single invocation]() | 12.83 μs | 17.39 μs | 1.35 | 0.08 | False | | | ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue<Int32>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsTrue<Int32>.ICollection(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 17.221965465701697 > 13.807469303835454. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -12.59767594136789 (T) = (0 -17214.049295440363) / Math.Sqrt((2013151.2654274935 / (25)) + (98.19219233095977 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.2621305908275654 = (13638.881285773525 - 17214.049295440363) / 13638.881285773525 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.ContainsTrue<Int32>.List(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 17.388277422049 > 13.959162170242886. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -12.904286364505177 (T) = (0 -17409.06331095206) / Math.Sqrt((2056912.7223882207 / (25)) + (11076.39780713844 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.27331935161706755 = (13672.189375621447 - 17409.06331095206) / 13672.189375621447 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [0214a047483067a6a813a34be5b6b1f757600961](https://github.com/dotnet/runtime/commit/0214a047483067a6a813a34be5b6b1f757600961) Compare | [2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc](https://github.com/dotnet/runtime/commit/2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc) Diff | [Diff](https://github.com/dotnet/runtime/compare/0214a047483067a6a813a34be5b6b1f757600961...2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Tests.Perf_String Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Remove_IntInt - Duration of single invocation]() | 11.45 ns | 16.29 ns | 1.42 | 0.06 | False | | | [Replace_String - Duration of single invocation]() | 38.91 ns | 46.95 ns | 1.21 | 0.03 | False | | | [TrimStart - Duration of single invocation]() | 11.08 ns | 16.17 ns | 1.46 | 0.07 | False | | | [ToLowerInvariant - Duration of single invocation]() | 18.06 ns | 22.96 ns | 1.27 | 0.03 | False | | | [TrimEnd - Duration of single invocation]() | 11.57 ns | 14.91 ns | 1.29 | 0.09 | False | | | [Trim_CharArr - Duration of single invocation]() | 16.40 ns | 19.69 ns | 1.20 | 0.10 | False | | | [TrimStart_CharArr - Duration of single invocation]() | 12.41 ns | 14.03 ns | 1.13 | 0.05 | False | | | [ToUpperInvariant - Duration of single invocation]() | 18.13 ns | 27.33 ns | 1.51 | 0.05 | False | | | [Trim_CharArr - Duration of single invocation]() | 13.18 ns | 16.50 ns | 1.25 | 0.05 | False | | | [Replace_String - Duration of single invocation]() | 35.69 ns | 50.18 ns | 1.41 | 0.03 | False | | | [Remove_IntInt - Duration of single invocation]() | 11.93 ns | 15.23 ns | 1.28 | 0.05 | False | | | [ToLowerInvariant - Duration of single invocation]() | 18.59 ns | 23.14 ns | 1.24 | 0.04 | False | | | [Trim_CharArr - Duration of single invocation]() | 14.50 ns | 16.73 ns | 1.15 | 0.08 | False | | | [ToLowerInvariant - Duration of single invocation]() | 31.45 ns | 35.73 ns | 1.14 | 0.03 | False | | | [Remove_IntInt - Duration of single invocation]() | 11.28 ns | 15.25 ns | 1.35 | 0.06 | False | | | ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 10, i2: 1) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.286023356741666 > 12.91177764480207. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/2/2023 3:18:41 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -13.550925850566289 (T) = (0 -16.143226907608938) / Math.Sqrt((1.2931853427437008 / (24)) + (0.28293719304310155 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.34729928512542557 = (11.981916034421483 - 16.143226907608938) / 11.981916034421483 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 46.948632398824834 > 45.132741645284355. IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 3/17/2023 10:06:19 AM, 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -9.467008971783427 (T) = (0 -46.62551707133518) / Math.Sqrt((8.53770693900059 / (25)) + (0.573687182384257 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.15224457929309637 = (40.46494807546849 - 46.62551707133518) / 40.46494807546849 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.TrimStart(s: " Test") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.165391021408933 > 11.30719128424789. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -48.63622296556723 (T) = (0 -16.35964596795198) / Math.Sqrt((0.17202241041931612 / (24)) + (0.030943304456261533 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.47065447176134534 = (11.124058221751213 - 16.35964596795198) / 11.124058221751213 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToLowerInvariant(s: "TeSt") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 22.962254251870444 > 19.454100154148634. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -35.500729363171196 (T) = (0 -23.347707883502615) / Math.Sqrt((0.37532936199380573 / (24)) + (0.04925291202905812 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.29695458966437155 = (18.001947076300166 - 23.347707883502615) / 18.001947076300166 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.TrimEnd(s: "Test ") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 14.913651208547238 > 12.72516506002495. IsChangePoint: Marked as a change because one of 3/21/2023 10:36:00 PM, 4/19/2023 8:38:27 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -8.274444649292139 (T) = (0 -15.648369454951709) / Math.Sqrt((0.14975776255986786 / (24)) + (1.2964555813686132 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.3010052373317946 = (12.02790658017998 - 15.648369454951709) / 12.02790658017998 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Trim_CharArr(s: " Te st ", c: [' ', ' ']) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 19.68856913041714 > 17.4880858480232. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 5/22/2023 7:40:07 AM, 5/23/2023 1:41:12 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -19.497367065447687 (T) = (0 -19.924239585484628) / Math.Sqrt((0.31606642421014763 / (24)) + (0.12677602164963053 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.20929676797304597 = (16.475889221866108 - 19.924239585484628) / 16.475889221866108 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.TrimStart_CharArr(s: " Test", c: [' ', ' ']) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 14.02842604331755 > 13.737467144653055. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -7.813505745388528 (T) = (0 -14.142283264510453) / Math.Sqrt((0.7491159584776212 / (24)) + (0.11758300126126087 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1377322553843227 = (12.43023848324774 - 14.142283264510453) / 12.43023848324774 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToUpperInvariant(s: "test") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.33346013334883 > 18.687162821963323. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -3.161744711592882 (T) = (0 -22.108599390788534) / Math.Sqrt((0.21718701830331613 / (24)) + (14.017759538709216 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2544395865715673 = (17.62428388537403 - 22.108599390788534) / 17.62428388537403 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Trim_CharArr(s: " Test", c: [' ', ' ']) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.50055467177426 > 15.102117059623264. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -14.258750931932699 (T) = (0 -16.487820179957204) / Math.Sqrt((0.9421479652021646 / (24)) + (0.002043369491451209 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2077020133146914 = (13.652225464710694 - 16.487820179957204) / 13.652225464710694 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 50.183800855262206 > 42.44772404786201. IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -14.937590072600948 (T) = (0 -50.36156535756361) / Math.Sqrt((19.54562471648434 / (25)) + (0.4426933315480499 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.3748792685114141 = (36.62980925742682 - 50.36156535756361) / 36.62980925742682 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.234881995647653 > 12.548569291285613. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -7.291263627626198 (T) = (0 -14.559666237283732) / Math.Sqrt((0.11878048749863417 / (24)) + (0.8419137856409523 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.21537965450867097 = (11.979521117761033 - 14.559666237283732) / 11.979521117761033 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToLowerInvariant(s: "TEST") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.135248297997048 > 19.596599138145027. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -34.88193701168203 (T) = (0 -23.20114159891524) / Math.Sqrt((0.31180615878346046 / (24)) + (0.06536239609790322 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2897602741505068 = (17.98872400081987 - 23.20114159891524) / 17.98872400081987 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Trim_CharArr(s: "Test ", c: [' ', ' ']) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.733750673324913 > 15.749647040257416. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 5/23/2023 1:41:12 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -17.592966750486983 (T) = (0 -16.873858487586865) / Math.Sqrt((0.32950062302134553 / (24)) + (0.0043234044063571436 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1427040498573515 = (14.766604257413194 - 16.873858487586865) / 14.766604257413194 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 35.72884961714696 > 34.557854647388986. IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -11.669014811508955 (T) = (0 -35.66501082277905) / Math.Sqrt((2.398002263904492 / (24)) + (0.061138336249441684 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.12088405020055965 = (31.818644235679432 - 35.66501082277905) / 31.818644235679432 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 7, i2: 4) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.253034117136021 > 12.635653580588492. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -8.303553574006205 (T) = (0 -15.15861929406102) / Math.Sqrt((0.3863057974234546 / (24)) + (1.1840634869368947 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.30849969383118686 = (11.584732778712194 - 15.15861929406102) / 11.584732778712194 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [0214a047483067a6a813a34be5b6b1f757600961](https://github.com/dotnet/runtime/commit/0214a047483067a6a813a34be5b6b1f757600961) Compare | [2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc](https://github.com/dotnet/runtime/commit/2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc) Diff | [Diff](https://github.com/dotnet/runtime/compare/0214a047483067a6a813a34be5b6b1f757600961...2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Collections.Sort<String> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Array_ComparerStruct - Duration of single invocation]() | 240.68 μs | 282.80 μs | 1.18 | 0.15 | False | | | [LinqOrderByExtension - Duration of single invocation]() | 213.08 μs | 253.26 μs | 1.19 | 0.07 | False | | | [LinqQuery - Duration of single invocation]() | 214.01 μs | 249.26 μs | 1.16 | 0.05 | False | | | [Array_Comparison - Duration of single invocation]() | 233.63 μs | 281.74 μs | 1.21 | 0.13 | False | | | [List - Duration of single invocation]() | 230.80 μs | 269.02 μs | 1.17 | 0.14 | False | | | [Array - Duration of single invocation]() | 224.32 μs | 271.32 μs | 1.21 | 0.17 | False | | | [Array_ComparerClass - Duration of single invocation]() | 235.83 μs | 284.99 μs | 1.21 | 0.17 | False | | | ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort<String>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Sort<String>.Array_ComparerStruct(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 282.80378666666655 > 262.63437830000004. IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/20/2023 7:45:57 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -27.5218931740511 (T) = (0 -285219.21895238094) / Math.Sqrt((35488010.78479053 / (24)) + (3308863.3287484846 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15579322644767585 = (246773.5685119047 - 285219.21895238094) / 246773.5685119047 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.LinqOrderByExtension(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 253.26065866666667 > 234.69183755000003. IsChangePoint: Marked as a change because one of 4/25/2023 3:46:39 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -21.665681347818406 (T) = (0 -250614.69681946628) / Math.Sqrt((49150568.40816188 / (24)) + (2214237.6471708487 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15330552502037817 = (217301.21930616614 - 250614.69681946628) / 217301.21930616614 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.LinqQuery(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 249.26142933333333 > 232.7877699. IsChangePoint: Marked as a change because one of 4/25/2023 9:59:52 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -18.766436696394923 (T) = (0 -251000.35188278387) / Math.Sqrt((58408438.660455175 / (24)) + (5685731.849366357 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15567254107877324 = (217189.85522359583 - 251000.35188278387) / 217189.85522359583 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.Array_Comparison(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 281.73827066666666 > 253.89817395000003. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -34.11144897021536 (T) = (0 -280901.71866666665) / Math.Sqrt((32505894.62266493 / (24)) + (1933736.6810228333 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.18352986804596233 = (237342.31492648553 - 280901.71866666665) / 237342.31492648553 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.List(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 269.024396 > 246.05755494230766. IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/19/2023 8:24:48 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -32.467794440344896 (T) = (0 -269152.1215238095) / Math.Sqrt((34983085.52278327 / (24)) + (272014.575374633 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17311381076405472 = (229433.93816880343 - 269152.1215238095) / 229433.93816880343 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.Array(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 271.31517199999996 > 245.20233394999997. IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/20/2023 3:57:43 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -29.192584994534666 (T) = (0 -269557.5234034537) / Math.Sqrt((39448744.262925684 / (24)) + (1897646.3343733747 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17627491207936258 = (229162.0102029914 - 269557.5234034537) / 229162.0102029914 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Sort<String>.Array_ComparerClass(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 284.9897853333334 > 253.62559705000004. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM. IsRegressionStdDev: Marked as regression because -32.70111773399092 (T) = (0 -283583.14293877553) / Math.Sqrt((33488824.27026515 / (24)) + (3985248.79086351 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1927945385846994 = (237746.848904304 - 283583.14293877553) / 237746.848904304 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: AndyAyersMS
Labels: `os-windows`, `arch-x64`, `area-CodeGen-coreclr`, `runtime-coreclr`, `PGO`, `needs-area-label`
Milestone: 8.0.0
AndyAyersMS commented 1 year ago

Naive q, could we imagine collecting data with and without new PGO on otherwise identical bits every time PGO data updates? That would immediately show how much of any change was the PGO update.

In general it would be nice if the benchmark system could anticipate impactful changes and run them in isolation. I don't know how challenging that would be to arrange as the whole system is capacity limited. As is the benchmark system seems to end up batching 10 or so commits per run, though sometimes it is less and sometimes much more.

We expect to see less impact from managed static PGO updates going forward as most benchmarks should now be using dynamic PGO data instead. PGO updates as a whole may still be impactful because they also update native PGO, and there are a number of benchmarks (like some of the ones here) that spend significant time in native code.

AndyAyersMS commented 1 year ago

System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Seems to have calmed down recently

newplot - 2023-06-21T082756 333

AndyAyersMS commented 1 year ago

Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array

Also looks like noise newplot - 2023-06-21T084012 165

AndyAyersMS commented 1 year ago

Not seeing anything actionable here, and most tests are either noisy or are still improved over the long run.