ghost
commented
2 years ago Tagging subscribers to this area: @JulieLeeMSFT See info in area-owners.md if you want to be subscribed.
Issue Details
### Run Information Architecture | arm64 -- | -- OS | ubuntu 18.04 Baseline | [2c3e1c7fee778b1e58c0678eb9bac273fd700216](https://github.com/dotnet/runtime/commit/2c3e1c7fee778b1e58c0678eb9bac273fd700216) Compare | [d81ac24b2ba08751fe21689b5182806fea6b8735](https://github.com/dotnet/runtime/commit/d81ac24b2ba08751fe21689b5182806fea6b8735) Diff | [Diff](https://github.com/dotnet/runtime/compare/2c3e1c7fee778b1e58c0678eb9bac273fd700216...d81ac24b2ba08751fe21689b5182806fea6b8735) ### Regressions in System.Collections.ContainsFalse<Int32> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ImmutableSortedSet - Duration of single invocation](
### Payloads
[Baseline]()
[Compare]()
### Histogram
#### System.Collections.ContainsFalse<Int32>.ImmutableSortedSet(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.97612073031136 > 40.968449278946196.
IsChangePoint: Marked as a change because one of 1/7/2022 4:24:34 PM, 3/22/2022 12:26:23 PM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -31.750718634657993 (T) = (0 -42956.59086093196) / Math.Sqrt((430473.58727225725 / (34)) + (1469.7221348553392 / (7))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (7) - 2, .025) and -0.09153028452612605 = (39354.46544168128 - 42956.59086093196) / 39354.46544168128 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```
### Docs
[Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md)
[Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information
Architecture | arm64
-- | --
OS | ubuntu 18.04
Baseline | [2c3e1c7fee778b1e58c0678eb9bac273fd700216](https://github.com/dotnet/runtime/commit/2c3e1c7fee778b1e58c0678eb9bac273fd700216)
Compare | [d81ac24b2ba08751fe21689b5182806fea6b8735](https://github.com/dotnet/runtime/commit/d81ac24b2ba08751fe21689b5182806fea6b8735)
Diff | [Diff](https://github.com/dotnet/runtime/compare/2c3e1c7fee778b1e58c0678eb9bac273fd700216...d81ac24b2ba08751fe21689b5182806fea6b8735)
### Regressions in System.Text.Perf_Utf8Encoding
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
[GetByteCount - Duration of single invocation](
### Payloads
[Baseline]()
[Compare]()
### Histogram
#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 57.55368987438726 > 45.359647407631215.
IsChangePoint: Marked as a change because one of 1/7/2022 4:24:34 PM, 2/23/2022 2:01:04 AM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -91.73236010708656 (T) = (0 -57550.17854516807) / Math.Sqrt((760051.9807085766 / (34)) + (74.63348954126008 / (7))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (7) - 2, .025) and -0.31298297947779785 = (43831.6257291142 - 57550.17854516807) / 43831.6257291142 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishMostlyAscii)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 126.50942006048385 > 99.47541709848485.
IsChangePoint: Marked as a change because one of 2/23/2022 2:01:04 AM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -83.80075313871093 (T) = (0 -126951.90332976988) / Math.Sqrt((2853991.9724543467 / (34)) + (374636.2011947571 / (7))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (7) - 2, .025) and -0.32403998559926195 = (95882.22765969663 - 126951.90332976988) / 95882.22765969663 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: Greek)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 62.89850403399311 > 49.58612193211858.
IsChangePoint: Marked as a change because one of 1/7/2022 4:24:34 PM, 2/23/2022 2:01:04 AM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -91.63171970002061 (T) = (0 -62909.027349448814) / Math.Sqrt((887786.6067112109 / (33)) + (64.99075446288354 / (7))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (7) - 2, .025) and -0.31397229406357396 = (47876.98160278339 - 62909.027349448814) / 47876.98160278339 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishMostlyAscii)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 274.7420202432266 > 253.95162019782788.
IsChangePoint: Marked as a change because one of 1/20/2022 9:28:17 PM, 2/22/2022 8:48:40 PM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -22.598811332621885 (T) = (0 -275170.4657499596) / Math.Sqrt((26557947.92640713 / (34)) + (10888849.200348983 / (7))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (7) - 2, .025) and -0.1435628743132779 = (240625.56762801736 - 275170.4657499596) / 240625.56762801736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.93451808534916 > 39.376826979743136.
IsChangePoint: Marked as a change because one of 1/7/2022 4:24:34 PM, 2/23/2022 2:01:04 AM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -8481.364447931946 (T) = (0 -49934.1444471577) / Math.Sqrt((8.318095246945353 / (34)) + (13.326755622261294 / (7))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (7) - 2, .025) and -0.3314904552625797 = (37502.442657247826 - 49934.1444471577) / 37502.442657247826 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetBytes(Input: Cyrillic)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 219.01502002314814 > 192.96708831532146.
IsChangePoint: Marked as a change because one of 2/22/2022 8:48:40 PM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -122.55527834589785 (T) = (0 -218875.21674137466) / Math.Sqrt((2201979.7565429383 / (34)) + (162724.6295646819 / (7))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (7) - 2, .025) and -0.1992027101980438 = (182517.27992278157 - 218875.21674137466) / 182517.27992278157 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetBytes(Input: Greek)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 316.0359689999999 > 278.0157029962924.
IsChangePoint: Marked as a change because one of 2/22/2022 8:48:40 PM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -60.13173971568646 (T) = (0 -313438.849855305) / Math.Sqrt((5785687.545973331 / (34)) + (3523085.341259236 / (7))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (7) - 2, .025) and -0.18685574527222645 = (264091.78293475945 - 313438.849855305) / 264091.78293475945 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetBytes(Input: Chinese)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 301.1389536057692 > 246.46762090718286.
IsChangePoint: Marked as a change because one of 2/22/2022 8:48:40 PM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -99.96235018366377 (T) = (0 -301315.6104602944) / Math.Sqrt((11239783.433485491 / (34)) + (632285.9207757254 / (7))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (7) - 2, .025) and -0.2742633916159605 = (236462.5810039008 - 301315.6104602944) / 236462.5810039008 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```
### Docs
[Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md)
[Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information
Architecture | arm64
-- | --
OS | ubuntu 18.04
Baseline | [2c3e1c7fee778b1e58c0678eb9bac273fd700216](https://github.com/dotnet/runtime/commit/2c3e1c7fee778b1e58c0678eb9bac273fd700216)
Compare | [d81ac24b2ba08751fe21689b5182806fea6b8735](https://github.com/dotnet/runtime/commit/d81ac24b2ba08751fe21689b5182806fea6b8735)
Diff | [Diff](https://github.com/dotnet/runtime/compare/2c3e1c7fee778b1e58c0678eb9bac273fd700216...d81ac24b2ba08751fe21689b5182806fea6b8735)
### Regressions in System.Collections.IndexerSetReverse<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
[IList - Duration of single invocation](
### Payloads
[Baseline]()
[Compare]()
### Histogram
#### System.Collections.IndexerSetReverse<Int32>.IList(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.370236299618321 > 2.078712465202614.
IsChangePoint: Marked as a change because one of 2/11/2022 12:34:36 AM, 4/16/2022 8:46:33 PM, 4/19/2022 4:53:57 AM falls between 4/10/2022 3:30:35 PM and 4/19/2022 4:53:57 AM.
IsRegressionStdDev: Marked as regression because -28.743781996752993 (T) = (0 -2370.4424267491386) / Math.Sqrt((5139.192095809534 / (33)) + (6.141634210601772 / (7))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (7) - 2, .025) and -0.17889588314060603 = (2010.7309395586528 - 2370.4424267491386) / 2010.7309395586528 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```
### Docs
[Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md)
[Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
| Author: | performanceautofiler[bot] |
|---|---|
| Assignees: | - |
| Labels: | `area-CodeGen-coreclr`, `untriaged`, `refs/heads/main`, `ubuntu 18.04`, `RunKind=micro`, `PGO`, `Regression`, `CoreClr`, `arm64` |
| Milestone: | - |
AndyAyersMS
dakersnar
lewing
jozkee
EgorBo
Run Information
Regressions in System.Collections.ContainsFalse<Int32>
Repro
Run Information
Regressions in System.Text.Perf_Utf8Encoding
Repro
Run Information
Regressions in System.Collections.IndexerSetReverse<Int32>
Repro
category:performance theme:benchmarks