bad processor affinity for distributor threads when reuseport=yes

seanzhang422 commented 9 months ago

[x] This is not a support question, I have read about opensource and will send support questions to the IRC channel, Github Discussions or the mailing list.
[x] I have read and understood the 'out in the open' support policy

Program: Authoritative
Issue type: Bug report

Short description

When configured with repuseport=yes, receiver-threads=X and distributor-threads=Y only Y number of CPU cores will be used no matter what X is. Whilst when reuseport=no, X * Y CPU cores are used (if not exceeding the maximum CPU cores). Threads count are the same when checked with ps -o nlwp <pid> regardless of reuseport.

Y can be any value >=1.

Environment

Operating system: Ubuntu 22.04
Software version: 4.6.4
Software source: PowerDNS repository

Steps to reproduce

Install PowerDNS auth, bind and geoip backend.
Configure with:

pdns.conf

launch=geoip,bind
bind-config=/etc/powerdns/named.conf
local-address=0.0.0.0
enable-lua-records=yes
edns-subnet-processing=yes
geoip-database-files=mmdb:/usr/share/GeoIP/geoip.mmdb
server-id=disabled
loglevel=6
version-string=anonymous
disable-syslog=yes
max-tcp-connections=10000
max-queue-length=50000
overload-queue-length=10000
reuseport=yes
receiver-threads=2
distributor-threads=2

named.conf

options {
    directory "/etc/powerdns/bind";
};

zone "example.com" IN {
    type native;
    file "example.com.zone";
};

example.com.zone

$ORIGIN example.com.
$TTL 3600
@ IN SOA  ns1.example.com. dns.example.com. (
                        2020022900      ; serial
                        10800           ; refresh
                        3600            ; retry
                        604800          ; expire
                        300             ; minimum
                        )
@ IN NS ns1

ns1 IN A 1.1.1.1

demo IN LUA A (
    ";if(country('US')) then return {'1.1.1.1', '2.2.2.2'} "
    "else return '3.3.3.3' end"
)

Stress test it with large number of queries to demo.example.com with random ECS. And you can observe that only 2 CPU cores will be used.

Expected behaviour

X * Y CPUs should be used with

reuseport=yes
receiver-threads=X
distributor-threads=Y

Actual behaviour

Only Y CPUs are used

Other information

Habbie commented 9 months ago

Are your queries coming from a small set of IP/port sources?

seanzhang422 commented 9 months ago

Yes, the test packets were sent with the same source IP and port. I updated the test to randomize the source port and now all distributor threads are used. So it's because of the way kernel distributes the packets and not the processor affinity. I'm closing this issue now.

Habbie commented 9 months ago

Thank you for reporting back!

PowerDNS / pdns