search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 3 Regressions on 11/11/2022 10:59:41 PM #9819

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline ce0b82ea7a19957ad3796e7e9a92e393bf6e30cc
Compare 43272d63c80f1cf6247fec62c2e189aa763f63a7
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Replace_Char - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_String.Replace_Char(text%3a%20%22This%20is%20a%20very%20nice%20sentence%22%2c%20oldChar%3a%20%27i%27%2c%20newChar%3a%20%27I%27).html>) 91.04 ns 112.94 ns 1.24 0.13 False
[Replace_Char - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_String.Replace_Char(text%3a%20%22yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58lzfdql1fehvs91yzkt9xam7ahjbhvpd9edll13ab46i74ktwwgkgbi79.html>) 1.29 μs 1.98 μs 1.53 0.01 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'i', newChar: 'I') ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 112.93755741337394 > 97.51570763430595. IsChangePoint: Marked as a change because one of 9/23/2022 6:26:22 PM, 11/11/2022 5:53:56 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsRegressionStdDev: Marked as regression because -28.035210373642595 (T) = (0 -116.17811513716113) / Math.Sqrt((9.506606145012745 / (33)) + (3.798420926826738 / (12))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (12) - 2, .025) and -0.2309768777865854 = (94.3787956001745 - 116.17811513716113) / 94.3787956001745 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58lzfdql1fehvs91yzkt9xam7ahjbhvpd9edll13ab46i74ktwwgkgbi792e5gkuuzevo5qm8qt83edag7zovoe686gmtw730kms2i5xgji4xcp25287q68fvhwszd3mszht2uh7bchlgkj5qnq1x9m4lg7vwn8cq5l756akua6oyx9k71bmxbysnmhvxvlxde4k9maumfgxd8gxhxx4mwpph2ttyox9zilt3ylv1q9s4bopfuoa8qlrzodg2q67sh85wx4slcd6w7ufnendaxai633ove2ktbaxdt2sz6y6mo42473xd274gz833p6hj3mu77c4m4od9e5s8btxleh0efqnu9zj9rwtbk5758lio35b3q426j5fwwq1qyknfedrsmqyfw1m38mkkotdf7n0vr6p3erhy8dkzntr9fwjrslxjgrbegih0n6bpb5bfuy55bu65ce9kejcfifxwpcs05umrsb8kvd64q2iwugbbi7vd35g5ho0rff9rhombgzzaniyq7bbjbqr88jyw4ccgnoyl31of3a5thv0vg08gnrqzxas800hewtw8tnwgw5pav81ntdpdd62689x3iqpc317y82b3e2trbpdzieoxldaz009tz37gqmh4bdp1bv9lnl5s58udb11z0h7i2sdl5nbyhjyfzxwzezmp4qx0i3eyvsd3fg8sryq9jhlvkonnfcvb4snl4mcbimdzg49tzdhqjmfxfcq3p1st6b9x2xyevo17evpqp4yc4f2rm0f26ivr3t2f5m0boc44vituxaovcqy1jrkcs6im2kdu3jvcexx2k76egve63aon5a6nbxss4rcke90npmqp35qluf571ms160y2nhaqef835wah41qru8tauu362v0r8konl8", oldChar: 'b', newChar: '+') ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 1.9754337990969775 > 1.3576841988571637. IsChangePoint: Marked as a change because one of 9/23/2022 6:26:22 PM, 11/11/2022 5:53:56 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsRegressionStdDev: Marked as regression because -327.3165180771985 (T) = (0 -1981.0884349686369) / Math.Sqrt((23.90412682748299 / (36)) + (44.87922268233078 / (12))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (12) - 2, .025) and -0.530748054607372 = (1294.1962780914816 - 1981.0884349686369) / 1294.1962780914816 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline ce0b82ea7a19957ad3796e7e9a92e393bf6e30cc
Compare 43272d63c80f1cf6247fec62c2e189aa763f63a7
Diff Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Combine - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.IO.Tests.Perf_Path.Combine.html>) 22.77 ns 32.48 ns 1.43 0.65 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.IO.Tests.Perf_Path.Combine ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 11/11/2022 5:53:56 PM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsRegressionStdDev: Marked as regression because -4.573223417455437 (T) = (0 -25.288841481480265) / Math.Sqrt((4.318231289741045 / (35)) + (6.9481314865332875 / (12))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (12) - 2, .025) and -0.17863263062054527 = (21.456084639507896 - 25.288841481480265) / 21.456084639507896 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

Looks like https://github.com/dotnet/runtime/pull/76337. @gfoidl please take a look

gfoidl commented 1 year ago

Yep, seems to be https://github.com/dotnet/runtime/pull/76337#discussion_r982852584 I'll dig into this tomorrow.

Edit, some notes (for me): Improvements: https://github.com/dotnet/perf-autofiling-issues/issues/9788 Regression: https://github.com/dotnet/perf-autofiling-issues/issues/9762, https://github.com/dotnet/runtime/issues/78517

gfoidl commented 1 year ago

Just a question.

Configurations: AOT=true,CompilationMode=wasm,RunKind=micro

So it's wasm. What does this mean for Vector128 and Vector256, i.e. are they supported / accelerated or not?

radekdoulik commented 1 year ago

They are not accelerated in the default aot configuration.

gfoidl commented 1 year ago

I continued the investigation in https://github.com/dotnet/runtime/issues/78517 Please keep this issue open anyway, as with https://github.com/dotnet/perf-autofiling-issues/issues/9819#issuecomment-1317089868 there are other issues for the same problem.