search

yrzr / opnsense-tools

OPNsense images native builder for aarch64. Steps to build your own image: https://yrzr.github.io/tutorial-build-opnsense-arm64-on-a-rpi4/
https://yrzr.github.io/tags/opnsense/
BSD 2-Clause "Simplified" License
51 stars 3 forks source link

Nanopi R4S low bandwidth 460 mbps #25

Closed javierckr closed 1 month ago

javierckr commented 1 month ago

With the nanopi r4s the máximum throughput I've reached on opnsense or freebsd is 460 mbps, powerd enabled, the cpu usage is 20% Intr and 80% idle, on the same nanopi r4s with openwrt I'm getting 940 mbps. How could be debugged what's happening? why it's not using 100% cpu?

yrzr commented 1 month ago

Well, this is an old topic that many may be confused about. And somehow we have discussed the network throughput performance on RPi4 here.

The major reason leading to the poor performance from my perspective of view is that the FreeBSD kernel does not support running clusters.

Currently the kernel doesn't support running clusters (big.LITTLE) at different speeds, doing so will result in (silent) crashes. Running powerd will change frequency on demand and keep both clusters in sync.

Therefore big cores are treated equally with the little cores. The big cores are running at the same frequency as the little cores, and the kernel is unaware of that even under low frequency, the big cores are always more efficient.

Even worse, as far as I know, there is no such thing in FreeBSD that is equivalent to irqbalance in Linux. IRQs for different devices (especially NIC) and user programs may run on the same little core simultaneously, which leads to a performance disaster.

javierckr commented 1 month ago

Thank you, now I understand the problem, I found a command, cpuset https://man.freebsd.org/cgi/man.cgi?cpuset I'll try to bind nic's irqs to cpu0 and cpu1 and see if that improves something, I'll report back.

yrzr commented 1 month ago

Also, try to bind cpu4 and cpu5 as they are big cores for RK3399.

javierckr commented 1 month ago

Binding irqs to cpus 4 and 5 results on 780mbps still not 940, you could bind using this: https://forums.FreeBSD.org/threads/interrupts-affinity-on-arm-systems.89181/post-615309 because irqs on arm is also a little bit tricky.