Open Brunnis opened 11 months ago
popcornmix
commented
3 months ago We have been in contact with Micron, and there is some good news. They have said they actually test their 8GB sdram with the 4GB refresh rate timing (rather than the slower jedec timings), and so it will be safe to run the 8GB parts with 4GB timing.
This should remove the lower performance observed with 8GB Pi5 compared to 4GB.
Latest pi5 bootloader firmware contains this update (and can get got with rpi-update).
Brunnis
commented
3 months ago @popcornmix That's great. I ran some tests on my 4GB and 8GB boards, using the same image. On average, there were no particular performance discrepancies in Geekbench 5, Passmark or Google Octane v2. There is some variability in the scores between runs and typically the 4GB peaks a bit higher, though.
The only test were there's still a bit more of a difference appears to be Geekbench 6. It's not something I personally would worry about, but reporting it for completeness:
https://browser.geekbench.com/v6/cpu/compare/7160591?baseline=7159462
popcornmix
commented
3 months ago It's interesting that the single core GB6 results also shows the difference. In general there is 4x the sdram bandwidth for multi-core compared to single core, so if sdram is the bottleneck it tends to show more significantly in the multicore results.
The scores for tests with little sdram bandwidth tend to get 4x with multicore. Ray Tracer is a good example of this with 3.98x. The scores for tests that are limited by sdram bandwidth tend to get closer to 1x (or even less due to thrashing). File Compression is a good example of this with .89x.
HTML5 Browser is a surprising result. It is somewhat sdram limited with a multicore speedup of 1.64x. It shows a single core score 17% higher on 4GB vs 8GB. It shows a multi core score 1.7% higher on 4GB vs 8GB.
If it is sdram causing this difference (which seems a reasonable conclusion based on the sdram size being the main differentiator) it seem surprising the difference is much greater in the less loaded, single core sdram case.
It would be interesting if you are able to run this a couple more times and confirm if the 17% vs 1.7% numbers are consistent across runs.
popcornmix
commented
3 months ago Thinking a bit more, the sdram is dual rank, with the top address bit choosing the rank. There is some benefit to accessing both ranks (in terms of keeping sdram pages open).
So a test that spans ranks (i.e. top and bottom halves of memory) may be beneficial.
We know that GB6 fails to run on a 2GB Pi with out-of-memory, and does run on a 4GB Pi. That suggests it (at least in some of its component tests) may utilise both ranks on a 4GB Pi but not an 8GB Pi.
We have actually been working on a scheme to try to avoid this type of inefficient behaviour due to precise locations of buffers. See https://github.com/raspberrypi/linux/pull/6273.
If you are feeling bold, then
sudo rpi-update pulls/6273should get you the numa feature which should mitigate some of these effects (and produce better benchmark scores, and performance of the system in general). Perhaps give it a go. Backing up first, or testing on a fresh install may be advisable.
by
commented
3 months ago Just on a side note: with this new firmware, my Pi5 now runs completely stable at 3 GHz; before, 2.9 GHz was the maximum for this very board of mine.
Brunnis
commented
3 months ago I've tested the emulated NUMA feature now and it's highly effective. The performance of both the 4GB and 8GB models is improved, but the 8GB significantly more so. With the NUMA feature enabled, both boards perform in a very consistent manner and there is no measureable performance difference between them.
The GB6 results are shown below. I did 5 runs on each configuration. The non-NUMA results are with the latest firmware that uses the same memory timings for the 4GB and 8GB. force_turbo=1 was used. The results match with what you've seen, i.e. +6% in single and +18% in multi composite scores.
Below is a comparison between non-NUMA and NUMA on the 8GB board. As can be seen, several sub-tests increase by ~30%. There are no regressions.
https://browser.geekbench.com/v6/cpu/compare/7189117?baseline=7173406
Out of interest, are you able to give any more background to this? It sounds like this would be something that could affect other platforms as well. Would this be beneficial for non Pi hardware or is it dealing with some peculiarity in your SoCs? And is this not needed or already mitigated on the x86 side?
EDIT: Are there any known negatives of applying this emulated NUMA feature?
EDIT 2: Running some other stuff as well, performance seems to be improved overall. Geekbench 5 is also much faster. Google Octane v2 web browser test is now at over 31k, where previously it was 29k. I have some older results for the WebGL Aquarium test, so not sure if something else has changed, but it now runs at 60 FPS with 1000 fish at 1080p fullscreen. Previously it was ~38 FPS.
popcornmix
commented
3 months ago Good to hear you are seeing the benefit. Every test I've run performs better with NUMA (the extent of the gain depends on how sdram bandwidth constrained the use case was to begin with).
Pi4 and Pi5 show similar benefits.
There is currently a minor issue with NUMA that means the CMA size is limited to a single NUMA region. e.g. CMA>=512M will fail on a 4GB Pi5 with 8 numa regions. That doesn't effect default RPiOS settings, but may if you've manually increased it.
Pi5 doesn't really need CMA, as it has MMUs in front of all hardware blocks (e.g. hevc or hvs), so can use system memory. Pi4 doesn't have the MMUs, so does still need CMA (and 512M may be needed for 4K hevc playback).
A solution for CMA + NUMA is being investigated (and once found will hopefully lead to PR being merged).
I'd expect it to affect some other platforms. I think it will help any "obvious" implementations of sdram controller. Some sdram controllers have some cleverness in, where they scramble the address lines coming in to break up the structure (note, it needs more that just reordering address lines), and they are effectively doing a similar thing to the NUMA interleave in hardware. I'd expect most x86 sdram controllers to be pretty clever.
Brunnis
commented
3 months ago Thanks for the additional info.
Also, I retested the WebGL Aquarium sample. Turns out it runs equally well without NUMA. Maybe some other SW optimization since I last tested? However, I can confirm that compilation runs faster. Compiling dosbox-staging is 10.5 % faster on the 8GB board with NUMA enabled.
Describe the bug The Raspberry Pi 5 4GB performs slightly better (0-10%) than the 8GB version at default 2.4 GHz and the gap widens to >100% for certain workloads when overclocked. These workloads will see a dramatic reduction in performance when overclocking the 8GB board. This reduction in performance is not present at all on the 4GB board.
It's unclear to me whether the small performance difference at default clock frequency has the same root cause as the more dramatic one that emerges as the ARM core frequency is increased. For the time being I'm treating them as related and reporting on both in this issue.
To reproduce I have found two workloads in particular that expose the issue: Geekbench 5 "Text Rendering" multi-core sub-test and stress-ng "numa" stressor. To reproduce this issue, I suggest benchmarking the 4GB and 8GB boards at both 2.4 GHz and 2.8 GHz.
Geekbench 5 is available here: https://www.geekbench.com/preview/
To run stress-ng with profiling info:
sudo apt install stress-ng linux-perf linux-perf-dbgsymsudo sh -c 'echo 1 >/proc/sys/kernel/perf_event_paranoid'stress-ng --numa 4 --numa-ops 1000 --metrics --perfExpected behaviour Ideally the 4GB and 8GB boards would perform close to the same. The smaller difference at stock frequencies could be deemed normal/expected (for example due to different RAM ICs being used), but the dramatically increasing gap in performance as ARM core frequency increases suggests something may be misbehaving.
Actual behaviour The 4GB board is anywhere from a few to more than 100% faster than the 8GB board, depending on clock frequency and workload. Below is a summary of tests I've run. As can be seen, the 4GB uses Samsung RAM and the 8GB uses Micron RAM.
These benchmark results are completely reproducible. I've also looked at other people's submission of Geekbench 5 results and can see the same reduction in "Text Rendering" scores on overclocked 8GB boards (but not on overclocked 4GB boards), so this is not limited to my specimen.
Below are the Geekbench 5 results at 2.4 and 2.8 GHz for the runs listed in the table above.
4GB (2400 MHz): https://browser.geekbench.com/v5/cpu/22028307 8GB (2400 MHz): https://browser.geekbench.com/v5/cpu/22028116 4GB (2800 MHz): https://browser.geekbench.com/v5/cpu/22028479 8GB (2800 MHz): https://browser.geekbench.com/v5/cpu/22028225
Below are "perf" tool output for the stress-ng runs at 2.4 and 2.8 GHz for both boards:
4GB (2.4 GHz):
8GB (2.4 GHz):
4GB (2.8 GHz):
8GB (2.8 GHz):
The 8GB 2.8 GHz result sticks out when compared to the same board running at 2.4 GHz, due to:
Finally, I should mention that RAM bandwidth and latency tests do not show any issues.
System raspinfo output: https://gist.github.com/Brunnis/4d8242cf757f28e1d5331b3f73b3a446