[Perf] Regressions in System.IO.Tests.Perf_File

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture	arm64
OS	ubuntu 18.04
Baseline	64807b3ccdc22da1175aa1bc1777d3abb1d9306f
Compare	9342649430c19af4bc9fe0d870609778f244a721
Diff	Diff

Regressions in System.IO.Tests.Perf_File

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
[Exists - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.IO.Tests.Perf_File.Exists.html>)	1.77 μs	2.38 μs	1.35	0.73	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_File*'

### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_File.Exists ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.3802197181343807 > 1.8330029791171227. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -11.098757048081785 (T) = (0 -2345.191633152322) / Math.Sqrt((28032.81587906434 / (14)) + (21047.306146025316 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.33264316155044416 = (1759.8046504991203 - 2345.191633152322) / 1759.8046504991203 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture	arm64
OS	ubuntu 18.04
Baseline	64807b3ccdc22da1175aa1bc1777d3abb1d9306f
Compare	9342649430c19af4bc9fe0d870609778f244a721
Diff	Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
[GetDirectoryName - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.IO.Tests.Perf_Path.GetDirectoryName.html>)	54.30 ns	72.20 ns	1.33	0.39	True
[ChangeExtension - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.IO.Tests.Perf_Path.ChangeExtension.html>)	39.29 ns	45.04 ns	1.15	0.31	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*'

### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_Path.GetDirectoryName ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 72.19606910795592 > 56.08286076934895. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -33.878088661687855 (T) = (0 -76.55617786846257) / Math.Sqrt((1.3664567865764794 / (14)) + (10.178336253236514 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.4385485158979128 = (53.217654477699526 - 76.55617786846257) / 53.217654477699526 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ```#### System.IO.Tests.Perf_Path.ChangeExtension ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 45.0361177715001 > 41.651022806176584. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -10.987450104898775 (T) = (0 -46.41335687040927) / Math.Sqrt((2.9168816128207897 / (14)) + (1.6851544558106764 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.14048768049944257 = (40.69606157436433 - 46.41335687040927) / 40.69606157436433 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture	arm64
OS	ubuntu 18.04
Baseline	64807b3ccdc22da1175aa1bc1777d3abb1d9306f
Compare	9342649430c19af4bc9fe0d870609778f244a721
Diff	Diff

Regressions in System.IO.Tests.Perf_FileInfo

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
[ctor_str - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.IO.Tests.Perf_FileInfo.ctor_str.html>)	226.37 ns	352.09 ns	1.56	0.37	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_FileInfo*'

### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_FileInfo.ctor_str ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 352.09365039076334 > 232.1363772047483. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -34.27215866084171 (T) = (0 -348.7170090836399) / Math.Sqrt((106.54393892364753 / (14)) + (167.49647202474404 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.5754975887401986 = (221.33769773807234 - 348.7170090836399) / 221.33769773807234 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture	arm64
OS	ubuntu 18.04
Baseline	64807b3ccdc22da1175aa1bc1777d3abb1d9306f
Compare	9342649430c19af4bc9fe0d870609778f244a721
Diff	Diff

Regressions in System.IO.Tests.Perf_Directory

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
[Exists - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.IO.Tests.Perf_Directory.Exists.html>)	2.00 μs	2.33 μs	1.16	0.71	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Directory*'

### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_Directory.Exists ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.328385926985556 > 1.797063406535883. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -11.621415429923713 (T) = (0 -2407.3153347219395) / Math.Sqrt((17943.416546602777 / (14)) + (49224.520896553804 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.36797838008953715 = (1759.7612431304474 - 2407.3153347219395) / 1759.7612431304474 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

DrewScoggins commented 2 years ago

This is the commit range, https://github.com/dotnet/runtime/compare/a077f271c3b5a8487f94111dc00a404a01fdc86b...6af4abc5a0db0560e719539a869233d15d28dc09. We could not easily understand where the issue was coming from.

adityamandaleeka commented 2 years ago

@jeffschwMSFT Did you see something that made you tag this as pipelines? The tests affected don't seem to be pipeline related.

jeffschwMSFT commented 2 years ago

In this case, I choose a label for one of the PRs that is in the range above. Closest to IO was pipelines. (nothing deeper than that)

adityamandaleeka commented 2 years ago

This is the commit range, a077f27...6af4abc. We could not easily understand where the issue was coming from.

@DrewScoggins The commit range included in the first post in this issue is a different (bigger) range: https://github.com/dotnet/runtime/compare/64807b3ccdc22da1175aa1bc1777d3abb1d9306f...9342649430c19af4bc9fe0d870609778f244a721

That range includes some arm64 changes that might be more likely to cause this (for instance, I see a peephole opt was removed on arm64).

I downloaded the attached artifacts and checked the commit hashes on libcoreclr and they match what was reported by the bot. Is the bot incorrect here about the versions used during the test? Or did you run additional tests to narrow down when the regression occurred (and if so, can you narrow it down even further? :smile:)?

DrewScoggins commented 2 years ago

Sorry for the delayed response, I had a filter that was a little too aggressive.

The way that we create the diff range is through the use of a changepoint algorithm. When that algorithm finds a change point, it does not always find it exactly. It can sometimes be a point off. This is a limitation of the algorithm. So to counter act this the diff link that is included in the issue goes one commit back and one commit forward to make sure that link has a higher chance of including the regression. The range that was found and included was obtained by using a feature in the reports that allows you to click a point and set it as a baseline, and click another point after it to set a compare, and then get a link to the commit range between those two points. Since that is done manually we can get as tight a range as possible.

adityamandaleeka commented 2 years ago

Thanks for that explanation @DrewScoggins. In this case, I think the commit range is likely wrong given the tests affected and the fact that it's arm64-specific.

@AndyAyersMS Any chance this arm64 regression is related to https://github.com/dotnet/runtime/pull/68631 ?

AndyAyersMS commented 2 years ago

@AndyAyersMS Any chance this arm64 regression is related to #68631 ?

Extremely unlikely.

Based on multiple charts the implicated commit range is indeed https://github.com/dotnet/runtime/compare/a077f271c3b5a8487f94111dc00a404a01fdc86b...6af4abc5a0db0560e719539a869233d15d28dc09

newplot - 2022-05-12T122749 858

newplot - 2022-05-12T123050 156

These regressions have persisted so they look real.

Ubuntu x64 shows a regression at the same spot (sorry my x axis span keeps shifting; @DrewScoggins it sure would be nice if we had an easy way of getting multiple charts over the same time range) so perhaps this is unix specific? Windows results are noisy but don't show a regression here.

newplot - 2022-05-12T123623 922

(full history) https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_ubuntu%2018.04%2fSystem.IO.Tests.Perf_File.Exists.html

The only possibilities I see in that commit range are https://github.com/dotnet/runtime/pull/68210 or https://github.com/dotnet/runtime/pull/68457 and it's not obvious either of them is involved.

So somebody needs to grab some before and after builds and drill into these tests to see what is going on.

adityamandaleeka commented 2 years ago

Thanks @AndyAyersMS

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-memory See info in area-owners.md if you want to be subscribed.

Issue Details

### Run Information Architecture | arm64 -- | -- OS | ubuntu 18.04 Baseline | [64807b3ccdc22da1175aa1bc1777d3abb1d9306f](https://github.com/dotnet/runtime/commit/64807b3ccdc22da1175aa1bc1777d3abb1d9306f) Compare | [9342649430c19af4bc9fe0d870609778f244a721](https://github.com/dotnet/runtime/commit/9342649430c19af4bc9fe0d870609778f244a721) Diff | [Diff](https://github.com/dotnet/runtime/compare/64807b3ccdc22da1175aa1bc1777d3abb1d9306f...9342649430c19af4bc9fe0d870609778f244a721) ### Regressions in System.IO.Tests.Perf_File Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Exists - Duration of single invocation]() | 1.77 μs | 2.38 μs | 1.35 | 0.73 | False | | | [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_File*' ```

### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_File.Exists ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.3802197181343807 > 1.8330029791171227. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -11.098757048081785 (T) = (0 -2345.191633152322) / Math.Sqrt((28032.81587906434 / (14)) + (21047.306146025316 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.33264316155044416 = (1759.8046504991203 - 2345.191633152322) / 1759.8046504991203 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

### Run Information Architecture | arm64 -- | -- OS | ubuntu 18.04 Baseline | [64807b3ccdc22da1175aa1bc1777d3abb1d9306f](https://github.com/dotnet/runtime/commit/64807b3ccdc22da1175aa1bc1777d3abb1d9306f) Compare | [9342649430c19af4bc9fe0d870609778f244a721](https://github.com/dotnet/runtime/commit/9342649430c19af4bc9fe0d870609778f244a721) Diff | [Diff](https://github.com/dotnet/runtime/compare/64807b3ccdc22da1175aa1bc1777d3abb1d9306f...9342649430c19af4bc9fe0d870609778f244a721) ### Regressions in System.IO.Tests.Perf_Path Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [GetDirectoryName - Duration of single invocation]() | 54.30 ns | 72.20 ns | 1.33 | 0.39 | True | | | [ChangeExtension - Duration of single invocation]() | 39.29 ns | 45.04 ns | 1.15 | 0.31 | False | | | [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*' ```

### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_Path.GetDirectoryName ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 72.19606910795592 > 56.08286076934895. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -33.878088661687855 (T) = (0 -76.55617786846257) / Math.Sqrt((1.3664567865764794 / (14)) + (10.178336253236514 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.4385485158979128 = (53.217654477699526 - 76.55617786846257) / 53.217654477699526 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ```#### System.IO.Tests.Perf_Path.ChangeExtension ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 45.0361177715001 > 41.651022806176584. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -10.987450104898775 (T) = (0 -46.41335687040927) / Math.Sqrt((2.9168816128207897 / (14)) + (1.6851544558106764 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.14048768049944257 = (40.69606157436433 - 46.41335687040927) / 40.69606157436433 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

### Run Information Architecture | arm64 -- | -- OS | ubuntu 18.04 Baseline | [64807b3ccdc22da1175aa1bc1777d3abb1d9306f](https://github.com/dotnet/runtime/commit/64807b3ccdc22da1175aa1bc1777d3abb1d9306f) Compare | [9342649430c19af4bc9fe0d870609778f244a721](https://github.com/dotnet/runtime/commit/9342649430c19af4bc9fe0d870609778f244a721) Diff | [Diff](https://github.com/dotnet/runtime/compare/64807b3ccdc22da1175aa1bc1777d3abb1d9306f...9342649430c19af4bc9fe0d870609778f244a721) ### Regressions in System.IO.Tests.Perf_FileInfo Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ctor_str - Duration of single invocation]() | 226.37 ns | 352.09 ns | 1.56 | 0.37 | True | | | [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_FileInfo*' ```

### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_FileInfo.ctor_str ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 352.09365039076334 > 232.1363772047483. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -34.27215866084171 (T) = (0 -348.7170090836399) / Math.Sqrt((106.54393892364753 / (14)) + (167.49647202474404 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.5754975887401986 = (221.33769773807234 - 348.7170090836399) / 221.33769773807234 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

### Run Information Architecture | arm64 -- | -- OS | ubuntu 18.04 Baseline | [64807b3ccdc22da1175aa1bc1777d3abb1d9306f](https://github.com/dotnet/runtime/commit/64807b3ccdc22da1175aa1bc1777d3abb1d9306f) Compare | [9342649430c19af4bc9fe0d870609778f244a721](https://github.com/dotnet/runtime/commit/9342649430c19af4bc9fe0d870609778f244a721) Diff | [Diff](https://github.com/dotnet/runtime/compare/64807b3ccdc22da1175aa1bc1777d3abb1d9306f...9342649430c19af4bc9fe0d870609778f244a721) ### Regressions in System.IO.Tests.Perf_Directory Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Exists - Duration of single invocation]() | 2.00 μs | 2.33 μs | 1.16 | 0.71 | False | | | [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Directory*' ```

### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_Directory.Exists ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.328385926985556 > 1.797063406535883. IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/5/2022 2:36:12 AM falls between 4/26/2022 10:28:11 AM and 5/5/2022 2:36:12 AM. IsRegressionStdDev: Marked as regression because -11.621415429923713 (T) = (0 -2407.3153347219395) / Math.Sqrt((17943.416546602777 / (14)) + (49224.520896553804 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.36797838008953715 = (1759.7612431304474 - 2407.3153347219395) / 1759.7612431304474 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Author:	performanceautofiler[bot]
Assignees:	DrewScoggins
Labels:	`area-System.Memory`, `tenet-performance`, `tenet-performance-benchmarks`, `untriaged`, `refs/heads/main`, `ubuntu 18.04`, `RunKind=micro`, `Regression`, `CoreClr`, `arm64`
Milestone:	-

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.