search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.03k stars 4.68k forks source link

Regressions in Microsoft.Extensions.DependencyInjection.ScopeValidation #89579

Closed performanceautofiler[bot] closed 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 2ee61bbe7bb04e97159f017c4f9eaf486dc46c1f
Compare a4a8e8e712eff30c9223ebdeeeb4c80fb798e972
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.Perf_RandomAccess

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
5.42 ms 5.89 ms 1.09 0.18 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_RandomAccess*'
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_RandomAccess.Write(fileSize: 1048576, bufferSize: 4096, options: Asynchronous) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 5/23/2023 8:43:31 PM, 7/20/2023 8:15:02 AM, 7/26/2023 10:32:59 PM falls between 7/18/2023 9:59:03 AM and 7/26/2023 10:32:59 PM. IsRegressionStdDev: Marked as regression because -5.966407543366567 (T) = (0 -5839819.7549007805) / Math.Sqrt((1717212977.202498 / (6)) + (75014545037.80371 / (15))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (15) - 2, .025) and -0.08025088202255358 = (5405984.713446275 - 5839819.7549007805) / 5405984.713446275 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 2ee61bbe7bb04e97159f017c4f9eaf486dc46c1f
Compare a4a8e8e712eff30c9223ebdeeeb4c80fb798e972
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.ScopeValidation

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
24.44 ns 49.53 ns 2.03 0.29 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.ScopeValidation*'
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.DependencyInjection.ScopeValidation.TransientWithScopeValidation #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 7/20/2023 8:15:02 AM, 7/26/2023 10:32:59 PM falls between 7/18/2023 9:59:03 AM and 7/26/2023 10:32:59 PM. IsRegressionStdDev: Marked as regression because -54.75286757222362 (T) = (0 -49.38343542733087) / Math.Sqrt((0.5241935187568165 / (6)) + (1.527196223135816 / (14))) is less than -2.100922040224263 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (14) - 2, .025) and -0.9662585539660059 = (25.115433231160225 - 49.38343542733087) / 25.115433231160225 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
EgorBo commented 1 year ago

Regressions in Microsoft.Extensions.DependencyInjection.ScopeValidation (all platforms) caused by https://github.com/dotnet/runtime/pull/89185 cc @mapogolions

EgorBo commented 1 year ago

Also,

mapogolions commented 1 year ago

I do not know if this is correct, but according to my results, there are 2 parts that affect the final value of the regression.

In my local repo, I have changes that isolated from Keyed Services Support feature. I ran benchmarks and got the following

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-extensions-dependencyinjection See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Name | Value -- | -- Architecture | arm64 OS | Windows 10.0.19041 Queue | SurfaceWindows Baseline | [2ee61bbe7bb04e97159f017c4f9eaf486dc46c1f](https://github.com/dotnet/runtime/commit/2ee61bbe7bb04e97159f017c4f9eaf486dc46c1f) Compare | [a4a8e8e712eff30c9223ebdeeeb4c80fb798e972](https://github.com/dotnet/runtime/commit/a4a8e8e712eff30c9223ebdeeeb4c80fb798e972) Diff | [Diff](https://github.com/dotnet/runtime/compare/2ee61bbe7bb04e97159f017c4f9eaf486dc46c1f...a4a8e8e712eff30c9223ebdeeeb4c80fb798e972) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.IO.Tests.Perf_RandomAccess Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [Write - Duration of single invocation]()
  • 📝 - [Benchmark Source]()
  • [ADX - Test Multi Config Graph]()
| 5.42 ms | 5.89 ms | 1.09 | 0.18 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_RandomAccess*' ```
### Payloads [Baseline]() [Compare]() ### System.IO.Tests.Perf_RandomAccess.Write(fileSize: 1048576, bufferSize: 4096, options: Asynchronous) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 5/23/2023 8:43:31 PM, 7/20/2023 8:15:02 AM, 7/26/2023 10:32:59 PM falls between 7/18/2023 9:59:03 AM and 7/26/2023 10:32:59 PM. IsRegressionStdDev: Marked as regression because -5.966407543366567 (T) = (0 -5839819.7549007805) / Math.Sqrt((1717212977.202498 / (6)) + (75014545037.80371 / (15))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (15) - 2, .025) and -0.08025088202255358 = (5405984.713446275 - 5839819.7549007805) / 5405984.713446275 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | arm64 OS | Windows 10.0.19041 Queue | SurfaceWindows Baseline | [2ee61bbe7bb04e97159f017c4f9eaf486dc46c1f](https://github.com/dotnet/runtime/commit/2ee61bbe7bb04e97159f017c4f9eaf486dc46c1f) Compare | [a4a8e8e712eff30c9223ebdeeeb4c80fb798e972](https://github.com/dotnet/runtime/commit/a4a8e8e712eff30c9223ebdeeeb4c80fb798e972) Diff | [Diff](https://github.com/dotnet/runtime/compare/2ee61bbe7bb04e97159f017c4f9eaf486dc46c1f...a4a8e8e712eff30c9223ebdeeeb4c80fb798e972) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in Microsoft.Extensions.DependencyInjection.ScopeValidation Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [TransientWithScopeValidation - Duration of single invocation]()
  • 📝 - [Benchmark Source]()
  • [ADX - Test Multi Config Graph]()
| 24.44 ns | 49.53 ns | 2.03 | 0.29 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.ScopeValidation*' ```
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.DependencyInjection.ScopeValidation.TransientWithScopeValidation #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 7/20/2023 8:15:02 AM, 7/26/2023 10:32:59 PM falls between 7/18/2023 9:59:03 AM and 7/26/2023 10:32:59 PM. IsRegressionStdDev: Marked as regression because -54.75286757222362 (T) = (0 -49.38343542733087) / Math.Sqrt((0.5241935187568165 / (6)) + (1.527196223135816 / (14))) is less than -2.100922040224263 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (14) - 2, .025) and -0.9662585539660059 = (25.115433231160225 - 49.38343542733087) / 25.115433231160225 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: -
Labels: `arch-arm64`, `os-windows`, `tenet-performance`, `tenet-performance-benchmarks`, `untriaged`, `runtime-coreclr`, `area-Extensions-DependencyInjection`, `needs-area-label`
Milestone: -
steveharter commented 1 year ago

Need to check if this is a subset of https://github.com/dotnet/runtime/issues/89104. Update: closing as duplicate.