performanceautofiler[bot]
commented
2 years ago Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | cb3bd225a88dd75ad804a3701a2ac9b2553a4973 |
| Compare | f9e9d86e77210bee1779970273d124ceeae0df7e |
| Diff | Diff |
Improvements in System.Collections.CreateAddAndRemove<String>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| [SortedList - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_LLVM=false_MonoAOT=false_MonoInterpreter=true/System.Collections.CreateAddAndRemove(String).SortedList(Size%3a%20512).html>) | 3.34 ms | 2.81 ms | 0.84 | 0.04 | False | |||||
| [Dictionary - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_LLVM=false_MonoAOT=false_MonoInterpreter=true/System.Collections.CreateAddAndRemove(String).Dictionary(Size%3a%20512).html>) | 416.26 μs | 278.85 μs | 0.67 | 0.01 | False | |||||
| [SortedDictionary - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_LLVM=false_MonoAOT=false_MonoInterpreter=true/System.Collections.CreateAddAndRemove(String).SortedDictionary(Size%3a%20512).html>) | 3.62 ms | 2.99 ms | 0.83 | 0.02 | False | |||||
| [Queue - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_LLVM=false_MonoAOT=false_MonoInterpreter=true/System.Collections.CreateAddAndRemove(String).Queue(Size%3a%20512).html>) | 70.58 μs | 53.45 μs | 0.76 | 0.09 | False | |||||
| [SortedSet - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_LLVM=false_MonoAOT=false_MonoInterpreter=true/System.Collections.CreateAddAndRemove(String).SortedSet(Size%3a%20512).html>) | 3.42 ms | 2.76 ms | 0.81 | 0.03 | False | |||||
| [LinkedList - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_LLVM=false_MonoAOT=false_MonoInterpreter=true/System.Collections.CreateAddAndRemove(String).LinkedList(Size%3a%20512).html>) | 122.09 μs | 100.52 μs | 0.82 | 0.06 | False | |||||
| [List - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_LLVM=false_MonoAOT=false_MonoInterpreter=true/System.Collections.CreateAddAndRemove(String).List(Size%3a%20512).html>) | 207.47 μs | 170.36 μs | 0.82 | 0.02 | False | |||||
| [HashSet - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_LLVM=false_MonoAOT=false_MonoInterpreter=true/System.Collections.CreateAddAndRemove(String).HashSet(Size%3a%20512).html>) | 398.39 μs | 262.38 μs | 0.66 | 0.01 | False | |||||
| [Stack - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_LLVM=false_MonoAOT=false_MonoInterpreter=true/System.Collections.CreateAddAndRemove(String).Stack(Size%3a%20512).html>) | 45.20 μs | 33.63 μs | 0.74 | 0.08 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CreateAddAndRemove<String>*'
### Payloads
[Baseline]()
[Compare]()
### Histogram
#### System.Collections.CreateAddAndRemove<String>.SortedList(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.8141002604166667 < 3.1919764716666665.
IsChangePoint: Marked as a change because one of 5/25/2022 7:30:00 PM, 6/5/2022 6:02:14 PM, 6/7/2022 6:20:49 PM falls between 5/30/2022 3:57:24 AM and 6/7/2022 6:20:49 PM.
IsImprovementStdDev: Marked as improvement because 15.620027069332744 (T) = (0 -2844033.214003357) / Math.Sqrt((33605582431.633415 / (299)) + (1362399956.2294073 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.07488040089720584 = (3074233.014587062 - 2844033.214003357) / 3074233.014587062 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.Dictionary(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 278.85224589688335 < 397.17812624778793.
IsChangePoint: Marked as a change because one of 5/25/2022 7:30:00 PM, 6/5/2022 6:02:14 PM, 6/7/2022 6:20:49 PM falls between 5/30/2022 3:57:24 AM and 6/7/2022 6:20:49 PM.
IsImprovementStdDev: Marked as improvement because 13.326974428091539 (T) = (0 -275198.52867697884) / Math.Sqrt((3061494590.4178367 / (299)) + (5324094.264324239 / (12))) is greater than 1.9676708853560316 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (12) - 2, .975) and 0.13665135715642862 = (318757.12200180243 - 275198.52867697884) / 318757.12200180243 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.SortedDictionary(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.9891565141369045 < 3.4574540818497024.
IsChangePoint: Marked as a change because one of 5/25/2022 7:30:00 PM, 6/5/2022 6:02:14 PM, 6/7/2022 6:20:49 PM falls between 5/30/2022 3:57:24 AM and 6/7/2022 6:20:49 PM.
IsImprovementStdDev: Marked as improvement because 20.61761195891767 (T) = (0 -2977992.4890529606) / Math.Sqrt((48853215045.34736 / (299)) + (226768664.771528 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.08517119566210546 = (3255245.653538726 - 2977992.4890529606) / 3255245.653538726 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.Queue(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 53.45062465753424 < 67.06528298533354.
IsChangePoint: Marked as a change because one of 5/25/2022 7:30:00 PM, 6/5/2022 6:02:14 PM, 6/7/2022 6:20:49 PM falls between 5/30/2022 3:57:24 AM and 6/7/2022 6:20:49 PM.
IsImprovementStdDev: Marked as improvement because 19.06040631355287 (T) = (0 -53495.46928318057) / Math.Sqrt((34459422.058368094 / (299)) + (223670.08865805273 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.11478776096226997 = (60432.3651707898 - 53495.46928318057) / 60432.3651707898 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.SortedSet(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.7605388197115386 < 3.2386285977905507.
IsChangePoint: Marked as a change because one of 5/25/2022 7:30:00 PM, 6/5/2022 6:02:14 PM, 6/7/2022 6:20:49 PM falls between 5/30/2022 3:57:24 AM and 6/7/2022 6:20:49 PM.
IsImprovementStdDev: Marked as improvement because 14.263504723371451 (T) = (0 -2789869.271218712) / Math.Sqrt((48015215373.25726 / (299)) + (1250058415.9387987 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.07571771008918828 = (3018416.8859147127 - 2789869.271218712) / 3018416.8859147127 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.LinkedList(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 100.5205868452381 < 115.98653645256441.
IsChangePoint: Marked as a change because one of 5/25/2022 7:30:00 PM, 6/5/2022 6:02:14 PM, 6/7/2022 6:20:49 PM falls between 5/30/2022 3:57:24 AM and 6/7/2022 6:20:49 PM.
IsImprovementStdDev: Marked as improvement because 11.587472716613673 (T) = (0 -101713.16241898987) / Math.Sqrt((55969119.15829647 / (299)) + (3450109.4003217635 / (12))) is greater than 1.9676708853560316 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (12) - 2, .975) and 0.07277851787268644 = (109696.7276746334 - 101713.16241898987) / 109696.7276746334 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.List(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 170.35728705197127 < 197.49590040865385.
IsChangePoint: Marked as a change because one of 5/25/2022 7:30:00 PM, 6/5/2022 6:02:14 PM, 6/7/2022 6:20:49 PM falls between 5/30/2022 3:57:24 AM and 6/7/2022 6:20:49 PM.
IsImprovementStdDev: Marked as improvement because 10.85262369797333 (T) = (0 -171695.7097742798) / Math.Sqrt((177481623.42933938 / (299)) + (11793546.341117142 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.07186922725267257 = (184990.8599259664 - 171695.7097742798) / 184990.8599259664 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.HashSet(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 262.3808057638889 < 379.2821242339538.
IsChangePoint: Marked as a change because one of 5/25/2022 7:30:00 PM, 6/5/2022 6:02:14 PM, 6/7/2022 6:20:49 PM falls between 5/30/2022 3:57:24 AM and 6/7/2022 6:20:49 PM.
IsImprovementStdDev: Marked as improvement because 12.315997799704254 (T) = (0 -262521.80549450184) / Math.Sqrt((2980918916.3666706 / (299)) + (9571312.874664718 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.13307513955960718 = (302819.5608107746 - 262521.80549450184) / 302819.5608107746 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.Stack(Size: 512)
```log
```
### Description of detection logic
```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 33.63103107078853 < 43.12200948998231.
IsChangePoint: Marked as a change because one of 5/25/2022 7:30:00 PM, 6/5/2022 6:02:14 PM, 6/7/2022 6:20:49 PM falls between 5/30/2022 3:57:24 AM and 6/7/2022 6:20:49 PM.
IsImprovementStdDev: Marked as improvement because 15.373623567739745 (T) = (0 -33807.45600178666) / Math.Sqrt((19377302.09677933 / (299)) + (105199.72632418195 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.10935308716443717 = (37958.314922075544 - 33807.45600178666) / 37958.314922075544 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```
### Docs
[Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md)
[Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Run Information
Improvements in System.Perf_Convert
Test Report
Repro