search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.17k stars 4.72k forks source link

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed #65191

Closed adamsitnik closed 1 year ago

adamsitnik commented 2 years ago

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed, it might be related to https://github.com/dotnet/runtime/issues/59415

image

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 net7.0 --filter System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark

It seems that @performanceautofiler has missed the regression (cc @DrewScoggins ): https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_Windows%2010.0.18362%2fSystem.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark.html

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark

Result Base Diff Ratio Modality Operating System Bit Processor Name
Slower 4.67 8.10 0.58 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 3.72 7.00 0.53 Windows 11 X64 AMD Ryzen 9 5900X
Same 5.56 7.91 0.70 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 7.99 11.94 0.67 several? Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell)
Same 5.62 7.03 0.80 Windows 10 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 5.04 6.85 0.74 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 5.20 8.06 0.65 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Slower 7.80 10.54 0.74 Windows 11 X64 Unknown processor
Same 9.87 9.32 1.06 multimodal Windows 11 X64 Unknown processor
Slower 4.22 7.39 0.57 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 5.40 10.62 0.51 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 9.77 12.18 0.80 centos 7 X64 Intel Xeon CPU E5530 2.40GHz
Slower 7.62 11.41 0.67 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 6.06 8.61 0.70 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Slower 5.99 8.41 0.71 ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Slower 6.04 8.30 0.73 ubuntu 20.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 10.94 11.56 0.95 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Slower 7.16 11.15 0.64 bimodal Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 34.35 31.63 1.09 Windows 10 Arm Microsoft SQ1 3.0 GHz
Slower 7.07 11.71 0.60 macOS Big Sur 11.6.3 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Slower 6.35 10.57 0.60 macOS Big Sur 11.4 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)

cc @tannergooding

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-numerics See info in area-owners.md if you want to be subscribed.

Issue Details
`System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark` has regressed, it might be related to https://github.com/dotnet/runtime/issues/59415 ![image](https://user-images.githubusercontent.com/6011991/153592717-ac4571c3-9d78-40c3-baa1-a74994d221fe.png) Repro: ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 net7.0 --filter System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark ``` It seems that @performanceautofiler has missed the regression (cc @DrewScoggins ): https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_Windows%2010.0.18362%2fSystem.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark.html ## System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark | Result | Base | Diff | Ratio | Modality | Operating System | Bit | Processor Name | | ------ | -----:| -----:| -----:| ---------- | -------------------- | ----- | ----------------------------------------------- | | Slower | 4.67 | 8.10 | 0.58 | | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | Slower | 3.72 | 7.00 | 0.53 | | Windows 11 | X64 | AMD Ryzen 9 5900X | | Same | 5.56 | 7.91 | 0.70 | | Windows 10 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | Slower | 7.99 | 11.94 | 0.67 | several? | Windows 11 | X64 | Intel Core i5-4300U CPU 1.90GHz (Haswell) | | Same | 5.62 | 7.03 | 0.80 | | Windows 10 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | | Same | 5.04 | 6.85 | 0.74 | | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | Slower | 5.20 | 8.06 | 0.65 | | Windows 11 | X64 | Intel Core i9-9900T CPU 2.10GHz | | Slower | 7.80 | 10.54 | 0.74 | | Windows 11 | X64 | Unknown processor | | Same | 9.87 | 9.32 | 1.06 | multimodal | Windows 11 | X64 | Unknown processor | | Slower | 4.22 | 7.39 | 0.57 | | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | | Slower | 5.40 | 10.62 | 0.51 | | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | Slower | 9.77 | 12.18 | 0.80 | | centos 7 | X64 | Intel Xeon CPU E5530 2.40GHz | | Slower | 7.62 | 11.41 | 0.67 | | ubuntu 18.04 | X64 | Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge) | | Slower | 6.06 | 8.61 | 0.70 | | alpine 3.13 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | | Slower | 5.99 | 8.41 | 0.71 | | ubuntu 18.04 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | | Slower | 6.04 | 8.30 | 0.73 | | ubuntu 20.04 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | | Same | 10.94 | 11.56 | 0.95 | | Windows 10 | Arm64 | Microsoft SQ1 3.0 GHz | | Slower | 7.16 | 11.15 | 0.64 | bimodal | Windows 10 | X86 | Intel Xeon CPU E5-1650 v4 3.60GHz | | Same | 34.35 | 31.63 | 1.09 | | Windows 10 | Arm | Microsoft SQ1 3.0 GHz | | Slower | 7.07 | 11.71 | 0.60 | | macOS Big Sur 11.6.3 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | | Slower | 6.35 | 10.57 | 0.60 | | macOS Big Sur 11.4 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | cc @tannergooding
Author: adamsitnik
Assignees: -
Labels: `area-System.Numerics`, `tenet-performance`
Milestone: -
adamsitnik commented 2 years ago

Same goes for System.Numerics.Tests.Perf_Matrix4x4.Transpose

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_ubuntu%2018.04%2fSystem.Numerics.Tests.Perf_Matrix4x4.Transpose.html

image

| Result | Base | Diff | Ratio | Alloc Delta | Modality | Operating System | Bit | Processor Name | Base V | Diff V | | ------ | -----:| -----:| -----:| -----------:| -------- | -------------------- | ----- | ----------------------------------------------- | ------------- | ----------- | | Slower | 4.78 | 6.93 | 0.69 | +0 | | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | 6.0.121.56705 | 7.0.22.7608| | Slower | 3.73 | 6.99 | 0.53 | +0 | | Windows 11 | X64 | AMD Ryzen 9 5900X | 6.0.121.56705 | 7.0.22.7608| | Slower | 5.36 | 8.72 | 0.61 | +0 | | Windows 10 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | 6.0.121.56705 | 7.0.22.7608| | Slower | 8.56 | 11.67 | 0.73 | +0 | | Windows 11 | X64 | Intel Core i5-4300U CPU 1.90GHz (Haswell) | 6.0.121.56705 | 7.0.22.7608| | Same | 5.76 | 7.27 | 0.79 | +0 | | Windows 10 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | 6.0.121.56705 | 7.0.22.7608| | Same | 5.31 | 6.99 | 0.76 | +0 | | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | 6.0.121.56705 | 7.0.22.7608| | Slower | 5.32 | 7.61 | 0.70 | +0 | | Windows 11 | X64 | Intel Core i9-9900T CPU 2.10GHz | 6.0.121.56705 | 7.0.22.7608| | Slower | 7.89 | 10.23 | 0.77 | +0 | | Windows 11 | X64 | Unknown processor | 6.0.121.56705 | 7.0.22.7608| | Same | 7.19 | 9.15 | 0.78 | +0 | | Windows 11 | X64 | Unknown processor | 6.0.121.56705 | 7.0.22.7608| | Slower | 4.68 | 7.52 | 0.62 | +0 | | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | 6.0.121.56705 | 7.0.22.7608| | Slower | 5.86 | 10.62 | 0.55 | +0 | | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | 6.0.121.56705 | 7.0.22.7608| | Slower | 10.83 | 13.40 | 0.81 | +0 | | centos 7 | X64 | Intel Xeon CPU E5530 2.40GHz | 6.0.121.56705 | 7.0.22.7608| | Same | 8.18 | 7.92 | 1.03 | +0 | | ubuntu 18.04 | X64 | Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge) | 6.0.121.56705 | 7.0.22.7608| | Slower | 5.90 | 8.69 | 0.68 | +0 | | alpine 3.13 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | 6.0.121.56705 | 7.0.22.7608| | Slower | 5.88 | 8.02 | 0.73 | +0 | | ubuntu 18.04 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | 6.0.121.56705 | 7.0.22.7608| | Slower | 5.93 | 9.20 | 0.64 | +0 | bimodal | ubuntu 20.04 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | 6.0.121.56705 | 7.0.22.7608| | Same | 12.77 | 12.40 | 1.03 | +0 | | Windows 10 | Arm64 | Microsoft SQ1 3.0 GHz | 6.0.121.56705 | 7.0.22.7608| | Slower | 6.47 | 10.07 | 0.64 | +0 | | Windows 10 | X86 | Intel Xeon CPU E5-1650 v4 3.60GHz | 6.0.121.56705 | 7.0.22.7608| | Same | 30.33 | 33.24 | 0.91 | +0 | | Windows 10 | Arm | Microsoft SQ1 3.0 GHz | 6.0.121.56705 | 7.0.22.7608| | Slower | 7.54 | 11.13 | 0.68 | +0 | | macOS Big Sur 11.6.3 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | 6.0.121.56705 | 7.0.22.7608| | Slower | 6.42 | 12.87 | 0.50 | +0 | | macOS Big Sur 11.4 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | 6.0.121.56705 | 7.0.22.7608|
adamsitnik commented 2 years ago

And System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark:

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_Windows%2010.0.18362%2fSystem.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark.html

image

adamsitnik commented 2 years ago

There are more, I won't list them all here.

tannergooding commented 2 years ago

I'll have to look at the codegen for these. They haven't been changed in a long time and so I expect this is something that's changed in CSE or similar.

Notably, they could/should likely be simplified to use the xplat helper intrinsics now, rather than individual AdvSimd/Sse code paths

jozkee commented 2 years ago

Some benchmarks discussed here appeared in the 6.0 vs 7.0-rc2 perf report. Are they still under investigation? The historical data only goes back until october 2021 so we are unable to determine if this is noise nor to see what might've caused it.

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark

| Result | Ratio | Alloc Delta | Operating System | Bit | Processor Name | Modality| | ------ | ----- | -----------:| ------------------- | ----- | --------------------------------------------- | --------:| | Same | 1.06 | +0 | ubuntu 18.04 | Arm64 | Unknown processor | | | Noise | - | +0 | Windows 11 | Arm64 | Unknown processor | | | Same | 1.05 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Same | 1.03 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Same | 0.98 | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 | | | Same | 1.00 | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 Max | | | Slower | 0.70 | +0 | Windows 10 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Slower | 0.59 | +0 | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | | Slower | 0.50 | +0 | Windows 11 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.46 | +0 | Windows 11 | X64 | AMD Ryzen 9 7950X | | | Slower | 0.62 | +0 | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Faster | 1.84 | +0 | debian 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.59 | +0 | ubuntu 18.04 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.25 | +0 | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | | Slower | 0.55 | +0 | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.77 | +0 | ubuntu 20.04 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Noise | - | +0 | ubuntu 20.04 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.67 | +0 | macOS Big Sur 11.7 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | | | Slower | 0.65 | +0 | macOS Monterey 12.6 | X64 | Intel Core i7-4870HQ CPU 2.50GHz (Haswell) | |

System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark

| Result | Ratio | Alloc Delta | Operating System | Bit | Processor Name | Modality| | ------ | ----- | -----------:| ------------------- | ----- | --------------------------------------------- | -------- | | Faster | 1.15 | +0 | ubuntu 18.04 | Arm64 | Unknown processor | | | Slower | 0.67 | +0 | Windows 11 | Arm64 | Unknown processor | | | Faster | 1.23 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Faster | 1.17 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Noise | - | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 | | | Noise | - | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 Max | | | Same | 0.96 | +0 | Windows 10 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Slower | 0.73 | +0 | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | | Slower | 0.63 | +0 | Windows 11 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.67 | +0 | Windows 11 | X64 | AMD Ryzen 9 7950X | | | Same | 0.91 | +0 | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | several?| | Same | 0.92 | +0 | debian 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.29 | +0 | ubuntu 18.04 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.31 | +0 | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | | Slower | 0.67 | +0 | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | | | Same | 0.92 | +0 | ubuntu 20.04 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Same | 1.06 | +0 | ubuntu 20.04 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Same | 0.92 | +0 | macOS Big Sur 11.7 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | | | Noise | - | +0 | macOS Monterey 12.6 | X64 | Intel Core i7-4870HQ CPU 2.50GHz (Haswell) | |

System.Numerics.Tests.Perf_Matrix4x4.NegationOperatorBenchmark

| Result | Ratio | Alloc Delta | Operating System | Bit | Processor Name | Modality| | ------ | ----- | -----------:| ------------------- | ----- | --------------------------------------------- | --------:| | Same | 1.05 | +0 | ubuntu 18.04 | Arm64 | Unknown processor | | | Same | 0.91 | +0 | Windows 11 | Arm64 | Unknown processor | | | Same | 1.04 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Faster | 1.59 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Same | 0.89 | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 | | | Same | 0.99 | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 Max | | | Noise | - | +0 | Windows 10 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Slower | 0.59 | +0 | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | | Slower | 0.36 | +0 | Windows 11 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.43 | +0 | Windows 11 | X64 | AMD Ryzen 9 7950X | | | Slower | 0.67 | +0 | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.65 | +0 | debian 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.50 | +0 | ubuntu 18.04 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.21 | +0 | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | | Slower | 0.53 | +0 | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.77 | +0 | ubuntu 20.04 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Noise | - | +0 | ubuntu 20.04 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.64 | +0 | macOS Big Sur 11.7 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | | | Slower | 0.66 | +0 | macOS Monterey 12.6 | X64 | Intel Core i7-4870HQ CPU 2.50GHz (Haswell) | |

System.Numerics.Tests.Perf_Matrix4x4.SubtractOperatorBenchmark

| Result | Ratio | Alloc Delta | Operating System | Bit | Processor Name | Modality| | ------ | ----- | -----------:| ------------------- | ----- | --------------------------------------------- | --------:| | Same | 1.00 | +0 | ubuntu 18.04 | Arm64 | Unknown processor | | | Same | 0.96 | +0 | Windows 11 | Arm64 | Unknown processor | | | Same | 0.99 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Same | 1.06 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Same | 1.00 | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 | | | Same | 1.00 | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 Max | | | Slower | 0.46 | +0 | Windows 10 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Slower | 0.61 | +0 | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | | Slower | 0.68 | +0 | Windows 11 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.65 | +0 | Windows 11 | X64 | AMD Ryzen 9 7950X | | | Same | 0.91 | +0 | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Noise | - | +0 | debian 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.73 | +0 | ubuntu 18.04 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.25 | +0 | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | | Slower | 0.70 | +0 | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | | | Same | 0.96 | +0 | ubuntu 20.04 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Same | 0.97 | +0 | ubuntu 20.04 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.83 | +0 | macOS Big Sur 11.7 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | | | Slower | 0.78 | +0 | macOS Monterey 12.6 | X64 | Intel Core i7-4870HQ CPU 2.50GHz (Haswell) | |

System.Numerics.Tests.Perf_Matrix4x4.Transpose

| Result | Ratio | Alloc Delta | Operating System | Bit | Processor Name | Modality| | ------ | -----:| -----------:| ------------------- | ----- | --------------------------------------------- | -------- | | Same | 0.97 | +0 | ubuntu 18.04 | Arm64 | Unknown processor | | | Same | 0.92 | +0 | Windows 11 | Arm64 | Unknown processor | | | Same | 0.98 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Same | 1.03 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Same | 0.92 | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 | | | Same | 0.93 | +0 | macOS Monterey 12.6 | Arm64 | Apple M1 Max | | | Slower | 0.76 | +0 | Windows 10 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Slower | 0.64 | +0 | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | | Slower | 0.45 | +0 | Windows 11 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.49 | +0 | Windows 11 | X64 | AMD Ryzen 9 7950X | | | Slower | 0.66 | +0 | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.71 | +0 | debian 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.19 | +0 | ubuntu 18.04 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.24 | +0 | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | | Slower | 0.56 | +0 | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | | | Slower | 0.68 | +0 | ubuntu 20.04 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Same | 0.89 | +0 | ubuntu 20.04 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | | Slower | 0.73 | +0 | macOS Big Sur 11.7 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | bimodal | | Slower | 0.66 | +0 | macOS Monterey 12.6 | X64 | Intel Core i7-4870HQ CPU 2.50GHz (Haswell) | |
dakersnar commented 1 year ago

The .NET 7 GA report is also showing potential regression in the following. I ran into the same problem where the historical data does not go back far enough to help triage this. @DrewScoggins is this something we can change?

System.Numerics.Tests.Perf_Matrix4x4.AddOperatorBenchmark

| Result | Ratio | Alloc Delta | Operating System | Bit | Processor Name | Modality| | ------ | ----- | -----------:| ------------------- | ----- | --------------------------------------------- | --------:| | Slower | 0.76 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | | Same | 1.01 | +0 | macOS 13.0 | Arm64 | Apple M1 | | | Slower | 0.69 | +0 | Windows 10 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | | Noise | - | +0 | Windows 10 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | | | Same | 1.08 | +0 | Windows 10 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | | Slower | 0.63 | +0 | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | | Same | 0.93 | +0 | Windows 11 | X64 | 11th Gen Intel Core i9-11900 2.50GHz | | | Slower | 0.16 | +0 | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | | Slower | 0.88 | +0 | raspbian 11 | Arm | ARMv7 Processor rev 3 (v7l) | | | Slower | 0.69 | +0 | macOS Monterey 12.6 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | |
DrewScoggins commented 1 year ago

We should figure out something that we can do. Naively, we could just extend the window of that history page from 12 to 18 months. However, that would not be feasible with the way those pages are currently generated, they would take too long to retrieve all of the data.

For these individual tests that we need more data for we can go ahead and generate longer trends, and just do them on a case by case basis, just get me a list of the tests that we need this for. However, I think the right way to fix this going forward is to update the generation method for the data on these pages so that we are not redoing work over and over (Currently, we regenerate the data every day, even though all we need to do is drop the earliest and grab the latest)

tannergooding commented 1 year ago

This was resolved with my rewrite of Matrix4x4 so be able to utilize promotion, field enregistration, and general SIMD/vectorization