ThomasKaiser
commented
1 year ago A few thoughts on methodology with a new platform like RISC-V:
Crypto performance:
This is two times a JH7110 board (Star64 and StarFive VisionFive V2) running exactly same OS image but different kernel config / device-tree settings:
| Clockspeed | Kernel | Distro | 7-zip multi | 7-zip single | AES | memcpy | memset |
|---|---|---|---|---|---|---|---|
| 1750 MHz | 5.15 | Sid riscv64 | 4820 | 1396 | 28970 | 1170 | 1120 |
| 1500 MHz | 5.15 | Sid riscv64 | 4040 | 1182 | 6830 | 1150 | 860 |
The 1st is slightly overclocked which ends up with better 7-zip scores, better memset numbers and way better AES scores (openssl speed -elapsed -evp aes-${bytelength}-cbc). At 1750 MHz (17% higher cpufreq) the crypto score gain is an impressive 424%! Or is it the hardware Star64 vs. StarFive VisionFive V2 (both boards being almost the same)?
Is that even possible? Of course not, it's 'OpenSSL 3.0.5, built on 5 Jul 2022' vs. 'OpenSSL 1.1.1f, built on 31 Mar 2020' (and that's why sbc-bench -j will always print the OpenSSL version since it might explain differing numbers).
On every 'new' platform software needs to be optimized. Even if the platform isn't that new any more – see this refreshing bit about a few lines NEON optimizations needed on ARM to really outperform x86 where this level of optimization was standard for a decade already: link to cnx-software)
Same with crypto stuff. On x86 and ARM we have optimized assembler routines since ages or even AES-NI and ARMv8 Crypto Extensions.
While on RISC-V we had only generic C routines but "scalar crypto" extension ratified in the meantime (of course not part of JH7110's silicon since basing on old U74 21G1 cores that predate the ratification). So my assumption is OpenSSL 3.0.5 now making use of optimized (assembler) code and/or the SHA and AES hardware accelerators that these U74 cores provide.
Still being a RISC-V noob I'm trying to ask @brucehoult for clarification (since your review will get a massive target audience compared to other RISC-V reviews).
While your target audience wrt 'SBC performance' most probably just wants a single score and 'less is better' or 'more is better' as only information they deserve better. IMO it's important when reviewing stuff to explain that new hardware usually gets faster over time since software matures and gradually makes use of hardware features. That's why listing a crypto score of 6830 is a bit unfair when comparing with ARM or x86 since those platforms are way older and the software side of things already received much more love. Even if JH7110 will never compete with recent x86 or ARMv8/ARMv9 CPUs due to lack of real crypto extensions.
Benchmarking the benchmark
For whatever reason people blindly trust into the numbers spitten out by a specific benchmark tool. And consumers as usually don't care about individual scores but want one single score or at least one for single-threaded and another for multi-threaded (IMO a horrible choice since those combined scores hide too much and are pretty much worthless).
Let's check Geekbench vs. another benchmark. When measuring the JH7110 at 1.5 GHz with 7-zip then it achieves a ~4000 score. 7-zip is all about Integer performance and memory (latency). When choosing a quad-core ARM board with similar score we end up at a Khadas VIM1S for example (based on Amlogic S905Y4: quad A35 @ 2.0 GHz).
Geekbench when looking at total scores tells us JH7110 is only at 60% the S905Y4's performance: https://browser.geekbench.com/v5/cpu/compare/19322441?baseline=17155111
When looking at the individual results view, the board's 'Integer Score' difference is 96 vs. 140 (68%) and 'Crypto Score' is listed as 11 vs. 167 (6.5%). I'm fine with the crypto score but highly doubt the others especially when looking through the individual scores.
Do these benchmark scores represent real-world performance? Or does the rule 'software needs to mature on a new hardware platform' also apply to benchmark software? Most probably yes.
Maybe this is also a hint in the same direction: GB not running on a bunch of RISC-V CPUs due to a problematic instruction.
IMO your target audience deserves these explanations: software needs time and this also applies to benchmarks (maybe once the Primatelabs guy got a bit more familiar with RISC-V the benchmark scores magically improve?). And this fact makes maintaining a 'table of benchmark scores' so problematic since software optimization efforts happening in the meantime automagically outdate such a table...
geerlingguy
Icenowy
brucehoult
Opvolger
morphykuffour
igorpecovnik
github-actions[bot]
IOOI-SqAR
Headcrabed
platima
Basic information
Linux/system information
Benchmark results
CPU
Power
stress-ng --matrix 0): 5.3 Wtop500HPL benchmark: TODO WDisk
SanDisk Extreme 128GB microSD
KIOXIA XG6 1GB NVMe SSD
curl https://raw.githubusercontent.com/geerlingguy/pi-cluster/master/benchmarks/disk-benchmark.sh | sudo bashRun benchmark on any attached storage device (e.g. eMMC, microSD, NVMe, SATA) and add results under an additional heading. Download the script with
curl -o disk-benchmark.sh [URL_HERE]and runsudo DEVICE_UNDER_TEST=/dev/sda DEVICE_MOUNT_PATH=/mnt/sda1 ./disk-benchmark.sh(assuming the device issda).Also consider running PiBenchmarks.com script.
Network
iperf3results:iperf3 -c $SERVER_IP: 937 Mbpsiperf3 --reverse -c $SERVER_IP: 774 Mbpsiperf3 --bidir -c $SERVER_IP: 941 Mbps up / 262 Mbps downI tested both 1 Gbps interfaces, and they both worked with similar results, and I could connect to two different IPs at once.
GPU
Memory