search

ocochard / BSDRP

BSD Router Project
https://bsdrp.net
Other
177 stars 38 forks source link

honors numa-domain per card #30

Closed rururudy closed 4 years ago

rururudy commented 4 years ago

supports t5 and t6 cards, more verbose output, a sprinkle of comments

ocochard commented 4 years ago

the dev.cpu.X.%domain seems not always here: sysctl: unknown oid 'dev.cpu.0.%domain'

rururudy commented 4 years ago

OK, I have a new patch... hold on, testing.

rururudy commented 4 years ago

Last version works... I could add a 'sysctl vm.ndomains' to better detect whether to use NUMA code at the beginning of the script...

ocochard commented 4 years ago

Yes, but I need to troubleshoot why my NUMA system doesn't populate the cpu.X.%domain variable:

[root@r630]~# sysctl vm.ndomains
vm.ndomains: 2
[root@r630]~# sysctl dev.cpu.0.%domain
sysctl: unknown oid 'dev.cpu.0.%domain'
rururudy commented 4 years ago

I'm using FreeBSD 12.1 ... maybe in -HEAD the oid is different or without the % sign (I'm not sure what the % sign is doing in the variables, maybe that is for development?).

I closed this patch out and made a new one that works better (#30 had some issues, #31 assigns properly).

FYI, I still can't forward traffic from my T5 card to my T6 card without having packets start dropping around 2Gbps... I and am suspicious that having cards in different NUMA domains is the problem.

ocochard commented 4 years ago

I have a server with Chelsio T5 on domain 1 and Mellanox on domain 0: I will bench it by receiving PPS on the Chelsio and forwarding out using the Mellanox.