search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.56k stars 4.54k forks source link

Regressions in System.IO.Compression.Gzip #73389

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 54c717a4ed822f46a23893479b8d4398596c041d
Compare 1926f0939a855d6ef41593377c75ab4c31e9fcbb
Diff Diff

Regressions in System.IO.Compression.Gzip

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Compress - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.IO.Compression.Gzip.Compress(level%3a%20Fastest%2c%20file%3a%20%22TestDocument.pdf%22).html>) 2.85 ms 3.00 ms 1.05 0.00 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Compression.Gzip*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Compression.Gzip.Compress(level: Fastest, file: "TestDocument.pdf") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.9996544642857135 > 2.9892825729166663. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -68.96288776311692 (T) = (0 -2997694.554906898) / Math.Sqrt((117543328.83053406 / (41)) + (7034096.18717764 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.052052518626114985 = (2849377.2904242603 - 2997694.554906898) / 2849377.2904242603 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 54c717a4ed822f46a23893479b8d4398596c041d
Compare 1926f0939a855d6ef41593377c75ab4c31e9fcbb
Diff Diff

Regressions in System.Buffers.Tests.NonStandardArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[RentNoReturn - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Buffers.Tests.NonStandardArrayPoolTests(Object).RentNoReturn(RentalSize%3a%2064%2c%20UseSharedPool%3a%20True).html>) 51.08 ns 54.81 ns 1.07 0.01 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.NonStandardArrayPoolTests&lt;Object&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Tests.NonStandardArrayPoolTests<Object>.RentNoReturn(RentalSize: 64, UseSharedPool: True) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 54.809619047619044 > 53.66642539062501. IsChangePoint: Marked as a change because one of 5/7/2022 9:43:21 AM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -18.036614790113084 (T) = (0 -54.494937921626985) / Math.Sqrt((0.35170233208069396 / (41)) + (0.10175683495165423 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.06501357108349988 = (51.168303767421605 - 54.494937921626985) / 51.168303767421605 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 54c717a4ed822f46a23893479b8d4398596c041d
Compare 1926f0939a855d6ef41593377c75ab4c31e9fcbb
Diff Diff

Regressions in System.IO.Compression.ZLib

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Compress - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.IO.Compression.ZLib.Compress(level%3a%20Fastest%2c%20file%3a%20%22TestDocument.pdf%22).html>) 2.80 ms 2.99 ms 1.07 0.00 False
[Compress - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.IO.Compression.ZLib.Compress(level%3a%20Fastest%2c%20file%3a%20%22sum%22).html>) 440.94 μs 469.89 μs 1.07 0.00 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Compression.ZLib*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Compression.ZLib.Compress(level: Fastest, file: "TestDocument.pdf") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.989092638888889 > 2.9355226614583327. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -77.61854139659978 (T) = (0 -2990808.179372711) / Math.Sqrt((47550707.749999985 / (41)) + (19984158.008067783 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.06882161333651177 = (2798229.510008116 - 2990808.179372711) / 2798229.510008116 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Compression.ZLib.Compress(level: Fastest, file: "sum") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 469.8932116596639 > 461.38697743055553. IsChangePoint: Marked as a change because one of 6/15/2022 7:24:26 PM, 7/27/2022 2:14:42 PM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -15.066854202865196 (T) = (0 -469237.5212257106) / Math.Sqrt((34829965.80911427 / (41)) + (5512974.956863488 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.050337598102365544 = (446749.2376484259 - 469237.5212257106) / 446749.2376484259 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 54c717a4ed822f46a23893479b8d4398596c041d
Compare 1926f0939a855d6ef41593377c75ab4c31e9fcbb
Diff Diff

Regressions in System.IO.Compression.Deflate

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Compress - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.IO.Compression.Deflate.Compress(level%3a%20Fastest%2c%20file%3a%20%22TestDocument.pdf%22).html>) 2.75 ms 2.94 ms 1.07 0.00 False
[Compress - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.IO.Compression.Deflate.Compress(level%3a%20Fastest%2c%20file%3a%20%22sum%22).html>) 424.17 μs 458.61 μs 1.08 0.00 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Compression.Deflate*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Compression.Deflate.Compress(level: Fastest, file: "TestDocument.pdf") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.944779722222223 > 2.8891870000000006. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -91.72982329289043 (T) = (0 -2947826.1991567463) / Math.Sqrt((71882321.59658116 / (41)) + (11089664.504111752 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.07089152752181736 = (2752684.210676688 - 2947826.1991567463) / 2752684.210676688 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Compression.Deflate.Compress(level: Fastest, file: "sum") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 458.6060171568628 > 445.7794805743244. IsChangePoint: Marked as a change because one of 6/15/2022 7:24:26 PM, 7/27/2022 2:14:42 PM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -14.913874638375276 (T) = (0 -458959.1882002802) / Math.Sqrt((28626618.235419188 / (40)) + (10684463.222928915 / (4))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (4) - 2, .025) and -0.06360486089072237 = (431512.8720039151 - 458959.1882002802) / 431512.8720039151 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 54c717a4ed822f46a23893479b8d4398596c041d
Compare 1926f0939a855d6ef41593377c75ab4c31e9fcbb
Diff Diff

Regressions in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableArray - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Collections.CtorFromCollection(String).ImmutableArray(Size%3a%20512).html>) 468.98 ns 495.24 ns 1.06 0.00 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CtorFromCollection<String>.ImmutableArray(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 495.243094848904 > 487.6974805387518. IsChangePoint: Marked as a change because one of 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -18.059649380719744 (T) = (0 -498.1264953871379) / Math.Sqrt((27.368457010300922 / (41)) + (7.571560273277371 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.06158568436984263 = (469.22872333458963 - 498.1264953871379) / 469.22872333458963 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
dotnet-issue-labeler[bot] commented 1 year ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

EgorBo commented 1 year ago

Zlib update https://github.com/dotnet/runtime/pull/73082 cc @GrabYourPitchforks (win-arm64)

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-system-io-compression See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [54c717a4ed822f46a23893479b8d4398596c041d](https://github.com/dotnet/runtime/commit/54c717a4ed822f46a23893479b8d4398596c041d) Compare | [1926f0939a855d6ef41593377c75ab4c31e9fcbb](https://github.com/dotnet/runtime/commit/1926f0939a855d6ef41593377c75ab4c31e9fcbb) Diff | [Diff](https://github.com/dotnet/runtime/compare/54c717a4ed822f46a23893479b8d4398596c041d...1926f0939a855d6ef41593377c75ab4c31e9fcbb) ### Regressions in System.IO.Compression.Gzip Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Compress - Duration of single invocation]() | 2.85 ms | 3.00 ms | 1.05 | 0.00 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Compression.Gzip*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Compression.Gzip.Compress(level: Fastest, file: "TestDocument.pdf") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.9996544642857135 > 2.9892825729166663. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -68.96288776311692 (T) = (0 -2997694.554906898) / Math.Sqrt((117543328.83053406 / (41)) + (7034096.18717764 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.052052518626114985 = (2849377.2904242603 - 2997694.554906898) / 2849377.2904242603 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [54c717a4ed822f46a23893479b8d4398596c041d](https://github.com/dotnet/runtime/commit/54c717a4ed822f46a23893479b8d4398596c041d) Compare | [1926f0939a855d6ef41593377c75ab4c31e9fcbb](https://github.com/dotnet/runtime/commit/1926f0939a855d6ef41593377c75ab4c31e9fcbb) Diff | [Diff](https://github.com/dotnet/runtime/compare/54c717a4ed822f46a23893479b8d4398596c041d...1926f0939a855d6ef41593377c75ab4c31e9fcbb) ### Regressions in System.Buffers.Tests.NonStandardArrayPoolTests<Object> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [RentNoReturn - Duration of single invocation]() | 51.08 ns | 54.81 ns | 1.07 | 0.01 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.NonStandardArrayPoolTests<Object>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Tests.NonStandardArrayPoolTests<Object>.RentNoReturn(RentalSize: 64, UseSharedPool: True) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 54.809619047619044 > 53.66642539062501. IsChangePoint: Marked as a change because one of 5/7/2022 9:43:21 AM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -18.036614790113084 (T) = (0 -54.494937921626985) / Math.Sqrt((0.35170233208069396 / (41)) + (0.10175683495165423 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.06501357108349988 = (51.168303767421605 - 54.494937921626985) / 51.168303767421605 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [54c717a4ed822f46a23893479b8d4398596c041d](https://github.com/dotnet/runtime/commit/54c717a4ed822f46a23893479b8d4398596c041d) Compare | [1926f0939a855d6ef41593377c75ab4c31e9fcbb](https://github.com/dotnet/runtime/commit/1926f0939a855d6ef41593377c75ab4c31e9fcbb) Diff | [Diff](https://github.com/dotnet/runtime/compare/54c717a4ed822f46a23893479b8d4398596c041d...1926f0939a855d6ef41593377c75ab4c31e9fcbb) ### Regressions in System.IO.Compression.ZLib Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Compress - Duration of single invocation]() | 2.80 ms | 2.99 ms | 1.07 | 0.00 | False | | | [Compress - Duration of single invocation]() | 440.94 μs | 469.89 μs | 1.07 | 0.00 | False | | | ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Compression.ZLib*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Compression.ZLib.Compress(level: Fastest, file: "TestDocument.pdf") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.989092638888889 > 2.9355226614583327. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -77.61854139659978 (T) = (0 -2990808.179372711) / Math.Sqrt((47550707.749999985 / (41)) + (19984158.008067783 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.06882161333651177 = (2798229.510008116 - 2990808.179372711) / 2798229.510008116 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Compression.ZLib.Compress(level: Fastest, file: "sum") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 469.8932116596639 > 461.38697743055553. IsChangePoint: Marked as a change because one of 6/15/2022 7:24:26 PM, 7/27/2022 2:14:42 PM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -15.066854202865196 (T) = (0 -469237.5212257106) / Math.Sqrt((34829965.80911427 / (41)) + (5512974.956863488 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.050337598102365544 = (446749.2376484259 - 469237.5212257106) / 446749.2376484259 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [54c717a4ed822f46a23893479b8d4398596c041d](https://github.com/dotnet/runtime/commit/54c717a4ed822f46a23893479b8d4398596c041d) Compare | [1926f0939a855d6ef41593377c75ab4c31e9fcbb](https://github.com/dotnet/runtime/commit/1926f0939a855d6ef41593377c75ab4c31e9fcbb) Diff | [Diff](https://github.com/dotnet/runtime/compare/54c717a4ed822f46a23893479b8d4398596c041d...1926f0939a855d6ef41593377c75ab4c31e9fcbb) ### Regressions in System.IO.Compression.Deflate Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Compress - Duration of single invocation]() | 2.75 ms | 2.94 ms | 1.07 | 0.00 | False | | | [Compress - Duration of single invocation]() | 424.17 μs | 458.61 μs | 1.08 | 0.00 | False | | | ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Compression.Deflate*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Compression.Deflate.Compress(level: Fastest, file: "TestDocument.pdf") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.944779722222223 > 2.8891870000000006. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -91.72982329289043 (T) = (0 -2947826.1991567463) / Math.Sqrt((71882321.59658116 / (41)) + (11089664.504111752 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.07089152752181736 = (2752684.210676688 - 2947826.1991567463) / 2752684.210676688 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Compression.Deflate.Compress(level: Fastest, file: "sum") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 458.6060171568628 > 445.7794805743244. IsChangePoint: Marked as a change because one of 6/15/2022 7:24:26 PM, 7/27/2022 2:14:42 PM, 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -14.913874638375276 (T) = (0 -458959.1882002802) / Math.Sqrt((28626618.235419188 / (40)) + (10684463.222928915 / (4))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (4) - 2, .025) and -0.06360486089072237 = (431512.8720039151 - 458959.1882002802) / 431512.8720039151 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [54c717a4ed822f46a23893479b8d4398596c041d](https://github.com/dotnet/runtime/commit/54c717a4ed822f46a23893479b8d4398596c041d) Compare | [1926f0939a855d6ef41593377c75ab4c31e9fcbb](https://github.com/dotnet/runtime/commit/1926f0939a855d6ef41593377c75ab4c31e9fcbb) Diff | [Diff](https://github.com/dotnet/runtime/compare/54c717a4ed822f46a23893479b8d4398596c041d...1926f0939a855d6ef41593377c75ab4c31e9fcbb) ### Regressions in System.Collections.CtorFromCollection<String> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ImmutableArray - Duration of single invocation]() | 468.98 ns | 495.24 ns | 1.06 | 0.00 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorFromCollection<String>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CtorFromCollection<String>.ImmutableArray(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 495.243094848904 > 487.6974805387518. IsChangePoint: Marked as a change because one of 8/2/2022 9:32:02 PM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -18.059649380719744 (T) = (0 -498.1264953871379) / Math.Sqrt((27.368457010300922 / (41)) + (7.571560273277371 / (4))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (4) - 2, .025) and -0.06158568436984263 = (469.22872333458963 - 498.1264953871379) / 469.22872333458963 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: -
Labels: `arch-arm64`, `area-System.IO.Compression`, `tenet-performance`, `tenet-performance-benchmarks`, `untriaged`
Milestone: -
jeffhandley commented 1 year ago

The regression here is beyond our control at the moment. However, I'm going to put this into the 8.0.0 milestone as we are going to be considering zlib options next year.

DrewScoggins commented 1 year ago

Also regressed on win-x64: https://github.com/dotnet/perf-autofiling-issues/issues/7165 win-x86: https://github.com/dotnet/perf-autofiling-issues/issues/7180 win-amd64: https://github.com/dotnet/perf-autofiling-issues/issues/7203

carlossanlop commented 11 months ago

We're not changing our zlib implementation at the moment, only taking updates on the native version. Moving to future as we are still exploring our options.