Closed micron10 closed 1 year ago
Please add more information, including configuration, cpu usage (top -H -p [pid]), logs, and number of clients.
Hi @pymumu
Thanks for fast respons,
users is ~ 5600
this is first test config , on time of test i disable log nfset cache , but load is stay on 100% load
top - 12:14:58 up 1 day, 8:56, 1 user, load average: 1.36, 1.02, 0.81 Threads: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.7 sy, 0.0 ni, 66.0 id, 0.0 wa, 0.0 hi, 32.6 si, 0.0 st %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 71.2 id, 0.0 wa, 0.0 hi, 28.8 si, 0.0 st %Cpu2 : 0.7 us, 0.7 sy, 0.0 ni, 68.2 id, 0.0 wa, 0.0 hi, 30.4 si, 0.0 st %Cpu3 : 0.7 us, 0.7 sy, 0.0 ni, 63.7 id, 0.0 wa, 0.0 hi, 34.9 si, 0.0 st %Cpu4 : 10.7 us, 54.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 34.7 si, 0.0 st %Cpu5 : 0.7 us, 0.7 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 38.3 si, 0.0 st %Cpu6 : 0.0 us, 0.7 sy, 0.0 ni, 72.6 id, 0.0 wa, 0.0 hi, 26.7 si, 0.0 st %Cpu7 : 0.7 us, 0.7 sy, 0.0 ni, 73.8 id, 0.0 wa, 0.0 hi, 24.8 si, 0.0 st %Cpu8 : 0.7 us, 0.7 sy, 0.7 ni, 66.7 id, 0.0 wa, 0.0 hi, 31.3 si, 0.0 st %Cpu9 : 0.7 us, 0.0 sy, 0.7 ni, 62.1 id, 0.0 wa, 0.0 hi, 36.6 si, 0.0 st %Cpu10 : 0.0 us, 0.0 sy, 0.0 ni, 62.2 id, 0.0 wa, 0.0 hi, 37.8 si, 0.0 st %Cpu11 : 0.7 us, 0.0 sy, 0.0 ni, 66.2 id, 0.0 wa, 0.0 hi, 33.1 si, 0.0 st MiB Mem : 31955.6 total, 28996.4 free, 2539.1 used, 1433.6 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29416.5 avail Mem
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
1125943 root 20 0 315.5m 23.8m 99.9 0.1 0:23.52 R smartdns -c /etc/smartdns/smartdns.conf 1125947 root 20 0 315.5m 23.8m 4.0 0.1 0:01.15 S smartdns -c /etc/smartdns/smartdns.conf 1125944 root 20 0 315.5m 23.8m 1.3 0.1 0:00.20 S smartdns -c /etc/smartdns/smartdns.conf 1125945 root 15 -5 315.5m 23.8m 1.3 0.1 0:00.39 S smartdns -c /etc/smartdns/smartdns.conf 1125946 root 20 0 315.5m 23.8m 0.7 0.1 0:00.31 S smartdns -c /etc/smartdns/smartdns.conf
this is config:
user root
bind [::]:53 -nftset
server 8.8.8.8 server-tls 8.8.8.8 server 8.8.4.4 server-tls 8.8.4.4 server 1.1.1.1 server-tls 1.1.1.1 server 1.0.0.1 server-tls 1.0.0.1 server 9.9.9.9 server 149.112.112.112 server 2620:fe::fe server 2620:fe::9 server 2001:4860:4860::8888 server 2001:4860:4860::8844 server 2606:4700::1111 server-tls 2606:4700::1111 server 2606:4700:4700::1001 server-tls 2606:4700:4700::1001
cache-size 131072 cache-persist yes cache-file /var/run/smartdns.cache cache-checkpoint-time 86400 prefetch-domain yes serve-expired yes serve-expired-ttl 259200 serve-expired-prefetch-time 21600
tcp-idle-time 120
force-qtype-SOA 65
dualstack-ip-selection yes
log-level info
log-file /var/log/smartdns/smartdns.log log-size 256k log-num 2
nftset-timeout yes
nftset-debug no
this is logs :
smartdns.log smartdns.log-20230822-121509.gz smartdns.log-20230822-121511.gz
Try increasing the cache size, say 1000000.
You can also try to modify the 33 lines of code as follows
DECLARE_HASHTABLE(cache_hash, 20);
Oki change to 20 in dns_cache and change in config to 1000000
but same :
top - 02:11:46 up 1 day, 22:52, 2 users, load average: 0.58, 0.29, 0.21 Threads: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.0 sy, 0.0 ni, 91.0 id, 0.0 wa, 0.0 hi, 8.3 si, 0.0 st %Cpu1 : 0.7 us, 1.3 sy, 0.0 ni, 84.6 id, 0.0 wa, 0.0 hi, 13.4 si, 0.0 st %Cpu2 : 0.0 us, 0.7 sy, 0.0 ni, 85.8 id, 0.0 wa, 0.0 hi, 13.5 si, 0.0 st %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 91.3 id, 0.0 wa, 0.0 hi, 8.7 si, 0.0 st %Cpu4 : 0.7 us, 0.0 sy, 0.0 ni, 89.8 id, 0.0 wa, 0.0 hi, 9.5 si, 0.0 st %Cpu5 : 0.0 us, 0.7 sy, 0.0 ni, 89.2 id, 0.0 wa, 0.0 hi, 10.1 si, 0.0 st %Cpu6 : 0.7 us, 0.7 sy, 0.0 ni, 87.3 id, 0.0 wa, 0.0 hi, 11.3 si, 0.0 st %Cpu7 : 0.7 us, 0.7 sy, 0.0 ni, 87.2 id, 0.0 wa, 0.0 hi, 11.4 si, 0.0 st %Cpu8 : 15.3 us, 42.7 sy, 0.0 ni, 30.7 id, 0.0 wa, 0.0 hi, 11.3 si, 0.0 st %Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 90.5 id, 0.0 wa, 0.0 hi, 9.5 si, 0.0 st %Cpu10 : 0.7 us, 0.7 sy, 0.0 ni, 87.8 id, 0.0 wa, 0.0 hi, 10.8 si, 0.0 st %Cpu11 : 0.0 us, 0.7 sy, 0.0 ni, 93.0 id, 0.0 wa, 0.0 hi, 6.3 si, 0.0 st MiB Mem : 31955.6 total, 28725.1 free, 2519.3 used, 1717.1 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29436.3 avail Mem
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
1630082 root 20 0 322.9m 32.0m 70.0 0.1 0:31.08 R smartdns -c /etc/smartdns/smartdns.conf 1630086 root 20 0 322.9m 32.0m 1.3 0.1 0:00.97 S smartdns -c /etc/smartdns/smartdns.conf 1630085 root 20 0 322.9m 32.0m 0.7 0.1 0:00.23 S smartdns -c /etc/smartdns/smartdns.conf 1630083 root 20 0 322.9m 32.0m 0.0 0.1 0:00.21 S smartdns -c /etc/smartdns/smartdns.conf 1630084 root 15 -5 322.9m 32.0m 0.0 0.1 0:00.24 S smartdns -c /etc/smartdns/smartdns.conf
smartdns.log smartdns.log-20230823-021222.gz smartdns.log-20230823-021224.gz
its same when stop knot-resolve that work on 3-4 % and run smartdns process is go to 100%
m.
try to modify the 626 and 630 lines of code as follows
ttl = *now - dns_cache->info.replace_time;
if (ttl < ttl_inactive_pre || inactive_precallback == NULL) {
626-> break;
}
if (callback_num >= max_callback_num) {
630-> break;
}
Hi @pymumu
Change both line to break.
load is stay same on 100%
in config set : cache-size 1000000
see logs i run in debug for more info smartdns.log smartdns.log-20230823-085109.gz smartdns.log-20230823-085110.gz
run perf top -p [pid]
, collect log.
15.54% [kernel] [k] l3mdev_master_ifindex_rcu
12.02% [kernel] [k] __ipv6_dev_get_saddr
6.00% [kernel] [k] __snmp6_fill_stats64.constprop.0
3.91% [kernel] [k] ipv6_dev_get_saddr
3.70% [pppoe] [k] pppoe_rcv
2.50% [kernel] [k] rtnl_fill_ifinfo.constprop.0.isra.0
2.11% [kernel] [k] inet6_fill_ifla6_attrs
1.75% [kernel] [k] inet_dump_ifaddr
1.63% libc.so.6 [.] 0x0000000000137a10
1.46% libc.so.6 [.] 0x0000000000137cb8
1.39% libc.so.6 [.] 0x0000000000137ca1
1.33% libc.so.6 [.] 0x00000000001379f9
1.33% [kernel] [k] nla_put
1.12% libc.so.6 [.] 0x0000000000137a00
1.06% libc.so.6 [.] 0x0000000000137ca8
1.04% [kernel] [k] __dev_queue_xmit
0.95% libc.so.6 [.] 0x0000000000137a09
0.82% [nf_tables] [k] nft_do_chain
0.81% libc.so.6 [.] 0x0000000000137cb1
0.80% [kernel] [k] fib_table_lookup
0.76% [kernel] [k] __netif_receive_skb_core.constprop.0
0.75% libc.so.6 [.] 0x00000000001379f0
0.74% libc.so.6 [.] 0x00000000001379fc
0.72% [kernel] [k] memset_orig
0.63% [ixgbe] [k] ixgbe_clean_rx_irq
0.61% [kernel] [k] memcmp
0.58% libc.so.6 [.] 0x0000000000137c98
0.57% [nf_tables] [k] __nft_rbtree_lookup
0.57% libc.so.6 [.] 0x0000000000137aa8
0.57% [kernel] [k] strncpy
Use the following command to compile smartdns to the debug version.
make DEBUG=1 all -j8
Use the perf top -p [pid]
command to view function information.
[pid] is the pid of smartdns process.
yes i set smartdns pid
make with DEBUG=1
16.71% [kernel] [k] l3mdev_master_ifindex_rcu
12.39% [kernel] [k] __ipv6_dev_get_saddr
5.29% [kernel] [k] __snmp6_fill_stats64.constprop.0
4.36% [kernel] [k] ipv6_dev_get_saddr
3.11% [pppoe] [k] pppoe_rcv
2.40% [kernel] [k] rtnl_fill_ifinfo.constprop.0.isra.0
1.85% [kernel] [k] inet6_fill_ifla6_attrs
1.57% [kernel] [k] inet_dump_ifaddr
1.44% libc.so.6 [.] 0x0000000000137a10
1.36% libc.so.6 [.] 0x0000000000137cb8
1.25% [kernel] [k] __dev_queue_xmit
1.19% libc.so.6 [.] 0x00000000001379f9
1.18% [kernel] [k] nla_put
1.17% libc.so.6 [.] 0x0000000000137ca1
1.01% libc.so.6 [.] 0x0000000000137ca8
0.99% libc.so.6 [.] 0x0000000000137a00
0.89% [kernel] [k] fib_table_lookup
0.83% [nf_tables] [k] nft_do_chain
0.82% libc.so.6 [.] 0x0000000000137a09
0.80% [kernel] [k] __netif_receive_skb_core.constprop.0
0.67% [kernel] [k] memset_orig
0.66% libc.so.6 [.] 0x00000000001379f0
0.65% libc.so.6 [.] 0x0000000000137cb1
0.64% [kernel] [k] memcmp
0.64% [kernel] [k] strncpy
0.64% [kernel] [k] csum_partial_copy_generic
0.62% [nf_tables] [k] __nft_rbtree_lookup
0.61% [kernel] [k] dev_queue_xmit_nit
0.58% libc.so.6 [.] 0x00000000001379fc
0.56% libc.so.6 [.] 0x0000000000137c98
0.56% [ixgbe] [k] ixgbe_clean_rx_irq
0.56% [kernel] [k] kmem_cache_free
0.55% [kernel] [k] __copy_skb_header
0.54% libc.so.6 [.] 0x0000000000137aa8
0.53% [kernel] [k] skb_put
0.52% [kernel] [k] skb_clone
There is no smartdns function in the log, are you sure there is no mistake?
yes all is fine on system have two other dns server
both work ~ good with 1-2% load
i whan to change with smartdns for me look more better , fast and good .
if have options to debug and find what make this load
The above logs are all functions of the kernel or libc, and it is impossible to analyze where the CPU is occupied by smartdns.
Is the CPU usage normal when there is no load?
yes cpu is fine when no load :
this is perf when not run smartdns and top
12.31% [pppoe] [k] pppoe_rcv
3.62% [kernel] [k] __dev_queue_xmit
2.66% [nf_tables] [k] nft_do_chain
2.54% [kernel] [k] __netif_receive_skb_core.constprop.0
2.53% [kernel] [k] fib_table_lookup
2.49% [kernel] [k] memcmp
2.41% [ixgbe] [k] ixgbe_clean_rx_irq
2.30% [nf_tables] [k] __nft_rbtree_lookup
1.82% [kernel] [k] strncpy
1.65% [kernel] [k] csum_partial_copy_generic
1.60% [kernel] [k] dev_queue_xmit_nit
1.57% [kernel] [k] skb_clone
1.55% [kernel] [k] kmem_cache_free
1.48% [kernel] [k] __copy_skb_header
1.31% [kernel] [k] __skb_flow_dissect
1.28% [kernel] [k] skb_release_data
1.21% [nf_tables] [k] nft_rhash_lookup
1.15% [ixgbe] [k] ixgbe_tx_map
1.09% [nf_conntrack] [k] __nf_conntrack_find_get.isra.0
0.90% [kernel] [k] dev_hard_start_xmit
0.86% [kernel] [k] kmem_cache_alloc
0.85% [kernel] [k] __siphash_unaligned
0.84% [kernel] [k] kfree_skb_reason
0.81% [vlan_mon] [k] vlan_pt_recv
0.80% [kernel] [k] ip_route_input_slow
0.78% [kernel] [k] ip_finish_output2
0.71% [nf_tables] [k] nft_lookup_eval
0.70% [kernel] [k] nf_hook_slow
0.68% [kernel] [k] _raw_spin_lock_irqsave
0.66% [ixgbe] [k] ixgbe_xmit_frame_ring
0.65% [kernel] [k] netif_skb_features
0.60% [kernel] [k] vlan_do_receive
0.59% [kernel] [k] ip_rcv_core.constprop.0
0.58% [kernel] [k] dev_gro_receive
0.58% [kernel] [k] menu_select
0.58% [kernel] [k] ___slab_alloc.isra.0
0.56% [kernel] [k] get_rps_cpu
0.55% [kernel] [k] process_backlog
0.55% [nf_nat] [k] nf_nat_inet_fn
0.55% [kernel] [k] validate_xmit_skb.isra.0
0.54% [kernel] [k] page_frag_free
0.52% [kernel] [k] ip_forward
0.52% [kernel] [k] __rcu_read_unlock
0.52% [kernel] [k] sch_direct_xmit
0.51% [kernel] [k] __local_bh_enable_ip
Tasks: 250 total, 1 running, 249 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.0 sy, 0.0 ni, 71.3 id, 0.0 wa, 0.0 hi, 28.0 si, 0.0 st %Cpu1 : 0.7 us, 0.7 sy, 0.0 ni, 71.6 id, 0.0 wa, 0.0 hi, 27.0 si, 0.0 st %Cpu2 : 0.7 us, 0.0 sy, 0.7 ni, 74.0 id, 0.0 wa, 0.0 hi, 24.7 si, 0.0 st %Cpu3 : 0.7 us, 0.7 sy, 0.0 ni, 72.5 id, 0.0 wa, 0.0 hi, 26.2 si, 0.0 st %Cpu4 : 1.4 us, 0.7 sy, 0.0 ni, 73.6 id, 0.0 wa, 0.0 hi, 24.3 si, 0.0 st %Cpu5 : 1.4 us, 0.0 sy, 0.0 ni, 72.3 id, 0.0 wa, 0.0 hi, 26.4 si, 0.0 st %Cpu6 : 0.7 us, 0.7 sy, 0.0 ni, 68.5 id, 0.0 wa, 0.0 hi, 30.1 si, 0.0 st %Cpu7 : 1.4 us, 0.7 sy, 0.0 ni, 74.7 id, 0.0 wa, 0.0 hi, 23.3 si, 0.0 st %Cpu8 : 0.7 us, 0.7 sy, 0.0 ni, 70.7 id, 0.0 wa, 0.0 hi, 27.9 si, 0.0 st %Cpu9 : 0.7 us, 0.0 sy, 0.0 ni, 69.6 id, 0.0 wa, 0.0 hi, 29.7 si, 0.0 st %Cpu10 : 1.4 us, 0.0 sy, 0.0 ni, 71.2 id, 0.0 wa, 0.0 hi, 27.4 si, 0.0 st %Cpu11 : 0.0 us, 0.0 sy, 0.0 ni, 74.5 id, 0.0 wa, 0.0 hi, 25.5 si, 0.0 st MiB Mem : 31955.6 total, 26487.0 free, 4621.1 used, 3905.0 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 27334.5 avail Mem
try remove some upstream servers,keep one upstream server.
try remove all only two stay 8.8.8.8 and 8.8.4.4
still same
one note with 1 user all is fine may be when have to much users is not fine
Very strange, if smartdns occupies the CPU, perf tool should be able to see which function of smartdns is occupying it.
The performance of smartdns has been tested, it is about 40000qps. If there are only 5000 users, and all the caches are hit, the CPU usage should be <10%
Only 5 processes when using top query? Are you using docker? What is the operating system? Are you using a precompiled version of release43 for the first time? In addition to the high CPU usage, are the functions normal in actual use?
Hi yes in top is show only 5 process, machine have 6 core + 6HT Debian base os with cudtom kernel. Yes first time use and compile Smartdns.
I think yes dns function normal
Hi try and get binary from here same stay on 100%
and one more i try on time when run service to make dig :
;; communications error to 127.0.0.1#53: timed out ;; communications error to 127.0.0.1#53: timed out ;; communications error to 127.0.0.1#53: connection refused
i think and not work
perf top -p PID after remove kernel item in this moment smartdns is on 100%
3.18% smartdns [.] 0x0000000000215b80
2.47% smartdns [.] 0x000000000021db90
2.37% smartdns [.] 0x000000000021dc6a
1.56% smartdns [.] 0x000000000021db98
1.40% smartdns [.] 0x00000000000176a8
1.39% smartdns [.] 0x000000000021abac
1.00% smartdns [.] 0x0000000000215beb
0.96% smartdns [.] 0x000000000021db9d
0.92% smartdns [.] 0x0000000000212ff2
0.89% smartdns [.] 0x0000000000215b89
0.84% smartdns [.] 0x0000000000212e5f
0.77% smartdns [.] 0x000000000021b839
0.75% smartdns [.] 0x0000000000215bde
0.69% smartdns [.] 0x0000000000215b4d
0.58% smartdns [.] 0x000000000021308f
0.57% smartdns [.] 0x0000000000213997
0.54% smartdns [.] 0x000000000021db73
0.52% smartdns [.] 0x0000000000215d18
0.52% smartdns [.] 0x000000000001e38c
0.51% smartdns [.] 0x000000000021ddcb
0.48% smartdns [.] 0x0000000000215b47
0.47% smartdns [.] 0x0000000000212e3b
0.47% smartdns [.] 0x0000000000215b44
0.45% smartdns [.] 0x000000000021c363
0.43% smartdns [.] 0x0000000000215ce7
0.42% smartdns [.] 0x000000000021353f
0.41% smartdns [.] 0x000000000021f55a
0.39% smartdns [.] 0x00000000002139b2
0.39% smartdns [.] 0x0000000000215bfa
0.36% smartdns [.] 0x000000000021b852
0.35% smartdns [.] 0x000000000021dd83
0.35% smartdns [.] 0x000000000021db7d
0.35% smartdns [.] 0x00000000000176b5
0.34% smartdns [.] 0x000000000021bb08
0.34% smartdns [.] 0x000000000021dcdc
0.33% smartdns [.] 0x000000000021f087
0.33% smartdns [.] 0x000000000021a388
0.30% smartdns [.] 0x000000000021f09a
0.29% smartdns [.] 0x0000000000213b44
0.29% smartdns [.] 0x0000000000215f25
0.28% smartdns [.] 0x0000000000212a77
0.25% smartdns [.] 0x000000000001e51b
0.25% smartdns [.] 0x000000000001e519
0.25% smartdns [.] 0x0000000000212a83
0.24% smartdns [.] 0x000000000021b888
0.24% smartdns [.] 0x000000000021b896
0.23% smartdns [.] 0x0000000000009914
0.23% smartdns [.] 0x000000000021c158
0.23% smartdns [.] 0x000000000002f4a0
0.22% smartdns [.] 0x000000000021a313
0.22% [vdso] [.] 0x00000000000006e5
0.22% smartdns [.] 0x000000000021b8ff
0.22% smartdns [.] 0x000000000021bb36
0.21% smartdns [.] 0x000000000021a38d
0.21% smartdns [.] 0x0000000000212efe
0.21% smartdns [.] 0x0000000000215c15
0.21% smartdns [.] 0x000000000021a30e
0.20% smartdns [.] 0x0000000000215b60
0.20% smartdns [.] 0x000000000021aba9
0.20% smartdns [.] 0x000000000000bcd0
0.19% smartdns [.] 0x000000000021a331
0.19% smartdns [.] 0x00000000000178c5
0.19% smartdns [.] 0x000000000021abf5
0.18% smartdns [.] 0x000000000021b826
0.18% smartdns [.] 0x000000000021fc1b
0.18% smartdns [.] 0x000000000021abf8
0.18% smartdns [.] 0x000000000021a385
0.18% smartdns [.] 0x000000000001e217
0.18% smartdns [.] 0x000000000021b834
0.18% [vdso] [.] 0x00000000000006d5
0.18% smartdns [.] 0x000000000021b7e7
0.17% smartdns [.] 0x000000000021a3bc
0.17% smartdns [.] 0x000000000021db89
0.17% smartdns [.] 0x000000000021bdbe
0.17% smartdns [.] 0x0000000000213c48
run perf top with debug version of smartdns
this get :
7.25% libc.so.6 [.] 0x0000000000137cb8
6.70% libc.so.6 [.] 0x0000000000137a10
5.96% libc.so.6 [.] 0x00000000001379f9
5.62% libc.so.6 [.] 0x0000000000137ca1
5.14% libc.so.6 [.] 0x0000000000137ca8
4.83% libc.so.6 [.] 0x0000000000137a00
4.43% libc.so.6 [.] 0x0000000000137a09
4.22% libc.so.6 [.] 0x0000000000137cb1
3.33% libc.so.6 [.] 0x00000000001379f0
3.04% libc.so.6 [.] 0x0000000000137c98
2.80% libc.so.6 [.] 0x00000000001379fc
2.74% libc.so.6 [.] 0x0000000000137aa8
2.50% libc.so.6 [.] 0x0000000000137ca4
1.76% libc.so.6 [.] 0x00000000001377da
1.31% libc.so.6 [.] 0x000000000016d5ca
0.70% libc.so.6 [.] 0x0000000000137cdf
0.61% smartdns [.] 0x0000000000015492
0.54% libc.so.6 [.] 0x0000000000137755
0.53% libc.so.6 [.] 0x00000000000a1952
0.39% libc.so.6 [.] 0x00000000000a0039
0.39% libc.so.6 [.] 0x000000000016cca0
0.31% libc.so.6 [.] 0x000000000016cca9
0.30% libc.so.6 [.] 0x0000000000137acf
0.29% libc.so.6 [.] 0x00000000001377bf
0.23% libc.so.6 [.] 0x000000000016cca4
0.23% libc.so.6 [.] 0x000000000016cb91
0.22% libc.so.6 [.] 0x000000000008f051
0.21% libc.so.6 [.] 0x0000000000137a93
0.20% libc.so.6 [.] 0x000000000016ccbb
0.20% libc.so.6 [.] 0x00000000001377d1
0.18% libc.so.6 [.] 0x000000000016ccb7
0.18% libc.so.6 [.] 0x0000000000137a9e
0.18% libc.so.6 [.] 0x000000000016ccc0
0.17% libc.so.6 [.] 0x000000000016ccae
0.17% libc.so.6 [.] 0x00000000000a2c16
0.17% libc.so.6 [.] 0x0000000000137ada
0.16% libc.so.6 [.] 0x000000000016ccc5
0.16% libc.so.6 [.] 0x000000000016cb09
0.15% libc.so.6 [.] 0x00000000001377ce
0.15% libc.so.6 [.] 0x00000000001374ba
0.15% libc.so.6 [.] 0x000000000009798a
0.15% libc.so.6 [.] 0x000000000016cb07
0.14% libc.so.6 [.] 0x0000000000095fd0
0.14% libc.so.6 [.] 0x00000000001377b4
0.14% libc.so.6 [.] 0x000000000008f0c0
0.13% libc.so.6 [.] 0x0000000000058949
0.13% libc.so.6 [.] 0x000000000005895a
0.13% libc.so.6 [.] 0x00000000000a2bf0
0.13% smartdns [.] 0x0000000000021d3d
0.12% libc.so.6 [.] 0x0000000000137a9b
0.12% smartdns [.] 0x00000000000320bd
0.11% libc.so.6 [.] 0x00000000001377d4
0.11% libc.so.6 [.] 0x0000000000137a98
0.11% libc.so.6 [.] 0x0000000000137ae1
0.11% smartdns [.] 0x0000000000021d38
0.11% smartdns [.] 0x000000000001548a
0.10% smartdns [.] 0x0000000000031621
0.10% libc.so.6 [.] 0x0000000000137ab9
0.10% libc.so.6 [.] 0x0000000000096018
0.10% libc.so.6 [.] 0x00000000001377c8
0.10% libc.so.6 [.] 0x00000000000a0cdc
0.09% libc.so.6 [.] 0x000000000016cb80
0.09% smartdns [.] 0x0000000000005650
0.09% smartdns [.] 0x0000000000031640
0.09% [vdso] [.] 0x00000000000006a0
0.09% libc.so.6 [.] 0x000000000008f0c9
0.09% libc.so.6 [.] 0x000000000016f047
0.09% libc.so.6 [.] 0x0000000000026290
0.09% libc.so.6 [.] 0x000000000016cbd6
0.08% smartdns [.] 0x0000000000041130
0.08% libc.so.6 [.] 0x0000000000137718
0.08% libc.so.6 [.] 0x000000000005a3ca
0.08% libc.so.6 [.] 0x000000000016ee5f
0.08% libc.so.6 [.] 0x00000000000984c0
0.08% libc.so.6 [.] 0x000000000008f0dd
0.08% smartdns [.] 0x000000000000fd8f
0.08% libc.so.6 [.] 0x000000000005a32b
0.08% libc.so.6 [.] 0x0000000000137aac
see this is knot-resolver :
2.01% liblmdb.so [.] 0x00000000000037e5
1.46% liblmdb.so [.] 0x000000000000700e
0.71% libc.so.6 [.] 0x000000000016c695
0.51% libkres.so.9 [.] 0x000000000002242c
0.50% libuv.so.1.0.0 [.] 0x00000000000090f0
0.38% liblmdb.so [.] 0x0000000000003eb4
0.32% libuv.so.1.0.0 [.] 0x000000000000c240
0.32% kresd [.] 0x0000000000028570
0.30% libkres.so.9 [.] 0x00000000000223f3
0.29% libc.so.6 [.] 0x000000000016cd25
0.28% liblmdb.so [.] 0x0000000000003ee6
0.27% [JIT] tid 2966215 [.] 0x00007f4992a4f0cc
0.26% liblmdb.so [.] 0x0000000000003ed1
0.25% libluajit.so [.] 0x000000000001c061
0.25% libc.so.6 [.] 0x0000000000027f70
0.25% libjemalloc.so.2 [.] 0x000000000001d5a6
0.25% libuv.so.1.0.0 [.] 0x000000000002be7b
0.24% libhogweed.so [.] 0x000000000001ccf7
0.24% libjemalloc.so.2 [.] 0x000000000002094e
0.23% libgmp.so.10.5.0 [.] 0x0000000000068c39
0.23% libhogweed.so [.] 0x000000000001ccb5
0.23% libluajit.so [.] 0x000000000001bfd0
0.23% libknot.so.13.0.0 [.] 0x000000000000dfb5
0.23% libc.so.6 [.] 0x0000000000095fd0
0.23% libc.so.6 [.] 0x000000000016c69d
0.22% libluajit.so [.] 0x000000000000a68d
0.22% libkres.so.9 [.] 0x0000000000025f3c
0.22% libgnutls.so.30.34.2 [.] 0x000000000004b700
0.21% libgmp.so.10.5.0 [.] 0x000000000006b6a7
0.21% libgnutls.so.30.34.2 [.] 0x00000000001340f0
0.20% libluajit.so [.] 0x0000000000012093
0.20% liblmdb.so [.] 0x0000000000003ef0
0.20% [JIT] tid 2966215 [.] 0x00007f4992a4ef78
0.20% kresd [.] 0x000000000001a3b6
0.20% libgnutls.so.30.34.2 [.] 0x00000000001340e8
0.19% libkres.so.9 [.] 0x0000000000007d40
0.19% libkres.so.9 [.] 0x000000000002345c
0.18% libc.so.6 [.] 0x00000000000de6e7
0.18% libkres.so.9 [.] 0x00000000000235f8
0.18% libgmp.so.10.5.0 [.] 0x000000000006710b
0.18% libgmp.so.10.5.0 [.] 0x000000000006b6a0
0.18% libjemalloc.so.2 [.] 0x00000000000240fa
0.18% libkres.so.9 [.] 0x00000000000247ba
0.18% libuv.so.1.0.0 [.] 0x00000000000137fe
0.18% kresd [.] 0x0000000000009b10
0.18% libknot.so.13.0.0 [.] 0x0000000000009d80
0.18% libjemalloc.so.2 [.] 0x000000000001e4c7
0.18% libhogweed.so [.] 0x000000000000a6f0
0.17% libkres.so.9 [.] 0x000000000001a16b
0.17% stats.so [.] 0x0000000000002250
0.17% libjemalloc.so.2 [.] 0x000000000001e4f0
0.17% libhogweed.so [.] 0x0000000000015574
0.17% libc.so.6 [.] 0x000000000016d5ca
0.17% libc.so.6 [.] 0x0000000000044a20
0.17% libkres.so.9 [.] 0x0000000000013e10
0.17% libgnutls.so.30.34.2 [.] 0x000000000013e0a0
0.17% libkres.so.9 [.] 0x000000000001a2cf
0.17% kresd [.] 0x0000000000028062
0.17% liblmdb.so [.] 0x0000000000003220
Threads: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.0 sy, 0.0 ni, 51.4 id, 0.0 wa, 0.0 hi, 47.8 si, 0.0 st %Cpu1 : 2.0 us, 1.3 sy, 0.0 ni, 59.1 id, 0.0 wa, 0.0 hi, 37.6 si, 0.0 st %Cpu2 : 1.4 us, 0.7 sy, 0.0 ni, 63.3 id, 0.0 wa, 0.0 hi, 34.7 si, 0.0 st %Cpu3 : 0.7 us, 0.7 sy, 0.0 ni, 62.2 id, 0.0 wa, 0.0 hi, 36.5 si, 0.0 st %Cpu4 : 1.4 us, 0.7 sy, 0.0 ni, 59.6 id, 0.0 wa, 0.0 hi, 38.4 si, 0.0 st %Cpu5 : 0.7 us, 0.7 sy, 0.0 ni, 59.9 id, 0.0 wa, 0.0 hi, 38.8 si, 0.0 st %Cpu6 : 1.4 us, 0.0 sy, 0.0 ni, 53.4 id, 0.0 wa, 0.0 hi, 45.3 si, 0.0 st %Cpu7 : 1.3 us, 0.7 sy, 0.0 ni, 53.0 id, 0.0 wa, 0.0 hi, 45.0 si, 0.0 st %Cpu8 : 1.4 us, 0.7 sy, 0.0 ni, 51.0 id, 0.0 wa, 0.0 hi, 46.9 si, 0.0 st %Cpu9 : 0.7 us, 1.4 sy, 0.0 ni, 58.1 id, 0.0 wa, 0.0 hi, 39.9 si, 0.0 st %Cpu10 : 0.0 us, 0.7 sy, 0.7 ni, 57.8 id, 0.0 wa, 0.0 hi, 40.8 si, 0.0 st %Cpu11 : 0.7 us, 0.0 sy, 0.0 ni, 54.9 id, 0.0 wa, 0.0 hi, 44.4 si, 0.0 st MiB Mem : 31955.6 total, 25799.1 free, 4671.3 used, 4546.1 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 27284.2 avail Mem
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
2966215 root 20 0 2364.7m 48.5m 4.0 0.2 0:42.58 S /usr/bin/kresd -c /etc/knot-resolver/kresd.conf --forks=6 2966225 root 20 0 2364.7m 48.5m 0.0 0.2 0:00.00 S /usr/bin/kresd -c /etc/knot-resolver/kresd.conf --forks=6
Hi @pymumu is there options to add more debugs to find from where is come this load
i make some test and service work fine with 1 to 100 users after that retest and try to find from where is problem but not success.
I suspect it has some problems with the compiler or the runtime environment, which is why I recommend using a precompiled (static) version. but 127.0.0.1?any proxy between smartdns and end users?
Hi @PikuZheng
as i write , try with download static file from here but is same
Hi @PikuZheng
as i write , try with download static file from here but is same
sorry to hear that :(
when running perf top
, there should be the following output, and you can see the function names of the process.
Samples: 91 of event 'cpu-clock', 4000 Hz, Event count (approx.): 1036339
Overhead Shared Object Symbol
20.09% libpthread-2.28.so [.] __pthread_mutex_lock
12.01% smartdns [.] dns_cache_invalidate
10.84% [kernel] [k] nft_do_chain
10.83% [kernel] [k] do_epoll_wait
10.83% libc-2.28.so [.] __strnlen_avx2
5.55% smartdns [.] _dns_server_process_cache_packet
5.00% [kernel] [k] wait_for_stable_page
4.87% libc-2.28.so [.] epoll_wait
1.91% [kernel] [k] __fget
1.91% [kernel] [k] __virt_addr_valid
1.91% libc-2.28.so [.] __memset_avx2_unaligned_erms
1.91% smartdns [.] _dns_encode_domain
1.58% [kernel] [k] finish_task_switch
1.28% [kernel] [k] sock_poll
1.28% [vdso] [.] time
1.00% smartdns [.] dns_server_run
0.86% [kernel] [k] aa_sk_perm
0.70% [kernel] [k] ep_poll
0.67% libc-2.28.so [.] _IO_vfprintf
0.67% libc-2.28.so [.] calloc
0.66% smartdns [.] _dns_get_domain_offset
0.66% [kernel] [k] nf_conntrack_tuple_taken
0.66% libc-2.28.so [.] clock_gettime
0.66% smartdns [.] New_Prefix2
0.38% libc-2.28.so [.] __libc_disable_asynccancel
0.34% libpthread-2.28.so [.] pthread_cond_signal@@GLIBC_2.3.2
0.20% [kernel] [k] crc32c_pcl_intel_update
0.20% libpthread-2.28.so [.] __errno_location
0.10% [kernel] [k] netif_skb_features
0.07% [kernel] [k] _raw_spin_unlock_irqrestore
0.06% smartdns [.] _dns_client_work
0.05% smartdns [.] _fast_ping_work
Only in this way I can locate where the CPU is being used.
In your log, there are only addresses, which cannot locate the problem. The possible reason is
or you can try callgrind
valgrind --tool=callgrind ./smartdns
by the way, you can run dnsperf to verify that the performance is ok or not.
make -C test test.bin -j8
test/test.bin --gtest_filter=Perf.*
Hi @pymumu
is there options to build one debug version in your system and to test that show all function name. if show to send this binary to test here
Huh
ok i try
in gdb not found debug :
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
in perf top -p PID same :
5.78% smartdns [.] 0x0000000000235bca
5.09% smartdns [.] 0x00000000000597ee
4.48% smartdns [.] 0x000000000022bad8
3.73% smartdns [.] 0x00000000001e4770
3.64% smartdns [.] 0x000000000014d0b2
3.61% smartdns [.] 0x0000000000010467
3.61% smartdns [.] 0x00000000000ae25e
3.53% smartdns [.] 0x000000000016b788
3.51% [vdso] [.] 0x00000000000006b3
3.46% smartdns [.] 0x00000000001ff817
3.42% smartdns [.] 0x0000000000235390
3.40% smartdns [.] 0x00000000000154d8
3.28% smartdns [.] 0x00000000001ff1d0
3.20% smartdns [.] 0x000000000023352a
3.16% smartdns [.] 0x0000000000042a4c
3.15% smartdns [.] 0x0000000000009bd9
3.14% smartdns [.] 0x000000000003032a
2.88% smartdns [.] 0x0000000000235af9
2.87% smartdns [.] 0x00000000001e61d6
2.87% smartdns [.] 0x0000000000133d14
2.84% smartdns [.] 0x0000000000035617
2.71% smartdns [.] 0x00000000001e4f89
2.55% smartdns [.] 0x000000000000d4e2
2.43% smartdns [.] 0x0000000000117363
2.38% smartdns [.] 0x00000000001e7962
2.35% smartdns [.] 0x0000000000079fd1
2.02% smartdns [.] 0x00000000001aa796
1.97% smartdns [.] 0x00000000000dbe61
1.75% smartdns [.] 0x000000000001aa1e
1.66% smartdns [.] 0x000000000008e596
1.10% smartdns [.] 0x0000000000234b82
0.95% smartdns [.] 0x000000000001a9f0
0.88% smartdns [.] 0x00000000002361fd
0.77% smartdns [.] 0x000000000002aeff
0.37% smartdns [.] 0x0000000000235b42
0.36% smartdns [.] 0x00000000002361f0
0.35% smartdns [.] 0x000000000002a6d0
0.35% smartdns [.] 0x0000000000233797
0.14% smartdns [.] 0x000000000001606e
0.12% smartdns [.] 0x00000000002374be
0.06% smartdns [.] 0x0000000000015d77
0.04% smartdns [.] 0x00000000000090cd
0.04% smartdns [.] 0x0000000000004358
with other process where have debug enable i see all function
in kernel is need to enable options to show this ? or other options to not display function name ?"
Hi @pymumu @PikuZheng
first I want to apologize to you. problem is in my perf tool not build with libelf and not show function name ....
After see function name i think know where is a problem :
netlink_msg_to_ifaddr i think problem is when smartdns run and have pppoe interface make this problem. i miss to write on this machine this 5000 users is over pppoe interface and this make problem .
is there options to fix this . i think smartdns try to run on all ppp interfacess..
11.77% smartdns [.] netlink_msg_to_ifaddr
8.30% smartdns [.] printf_core
5.25% smartdns [.] memcpy
3.39% smartdns [.] _dns_get_domain_from_packet
3.32% smartdns [.] memset
2.89% smartdns [.] _dns_server_process_local_ptr
2.79% smartdns [.] __fwritex
2.58% smartdns [.] __libc_free
2.38% smartdns [.] enframe
2.32% smartdns [.] alloc_slot
2.19% smartdns [.] pad
2.17% smartdns [.] fmt_u
1.91% smartdns [.] dict_hash
1.56% smartdns [.] get_meta
1.24% smartdns [.] out
1.16% smartdns [.] __libc_malloc_impl
0.87% smartdns [.] _dns_read_short
0.84% smartdns [.] _dns_left_len
0.83% smartdns [.] vfprintf
0.75% smartdns [.] _dns_encode_domain
0.73% smartdns [.] sn_write
0.72% [vdso] [.] __vdso_clock_gettime
0.72% smartdns [.] size_to_class
0.63% smartdns [.] _dns_get_domain_offset
0.60% smartdns [.] __stpncpy
0.60% smartdns [.] sccp
0.60% smartdns [.] strncmp
0.59% smartdns [.] _dns_server_process_answer_A
0.56% smartdns [.] __malloc_allzerop
0.55% smartdns [.] _dns_write_short
0.54% smartdns [.] __pthread_mutex_lock
0.54% smartdns [.] _dns_server_process_answer
0.50% smartdns [.] vsnprintf
0.48% smartdns [.] _dns_decode_an
0.47% smartdns [.] pop_arg
0.46% smartdns [.] tlog_vext
0.44% smartdns [.] jhash
0.43% smartdns [.] __clock_gettime
0.42% smartdns [.] _dns_client_send_packet
0.41% smartdns [.] _dns_rr_add_end
0.40% smartdns [.] freeifaddrs
0.38% smartdns [.] __lock
0.36% smartdns [.] _dns_client_recv
0.35% smartdns [.] tlog_ext
0.32% smartdns [.] calloc
0.32% smartdns [.] do_ssl3_write
0.31% smartdns [.] _dns_ip_address_check_add
0.30% smartdns [.] jhash
0.30% smartdns [.] dns_cache_lookup
0.29% smartdns [.] _dns_decode_body
0.28% smartdns [.] __netlink_enumerate
0.28% smartdns [.] copy_addr
0.28% smartdns [.] jhash
0.28% smartdns [.] strtox
0.28% smartdns [.] rol32
0.27% smartdns [.] _dns_client_work
0.27% smartdns [.] nontrivial_free
0.26% smartdns [.] _fast_ping_process_icmp
0.26% smartdns [.] _dns_init_context_by_rrs
0.26% smartdns [.] __unlock
0.25% smartdns [.] _dns_client_process_udp
0.25% smartdns [.] _dns_decode_raw
0.23% smartdns [.] getint
0.23% smartdns [.] ssl3_get_record
0.23% smartdns [.] _dns_server_reply_passthrough
0.22% smartdns [.] __pthread_mutex_unlock
0.22% smartdns [.] dns_server_resolve_callback
0.22% smartdns [.] _dns_decode_head
0.20% smartdns [.] __strerror_l
0.20% smartdns [.] ssl3_clear
0.20% smartdns [.] aes_gcm_ctrl
0.19% smartdns [.] ssl3_read_bytes
0.19% smartdns [.] hash_string_initval
0.19% smartdns [.] EVP_DecryptUpdate
0.19% smartdns [.] memchr
0.19% smartdns [.] CRYPTO_gcm128_decrypt_ctr32
0.19% smartdns [.] setsockopt
0.19% smartdns [.] _dns_add_qr_head
Are you meaning that the device run as a pppoe server and smartdns needs to handle 5000 interfaces? I never seen this situation. Perhaps you could bind it to one interface and use iptables to forward all ppp requests to this interface
how isc bind and knot-resolver make this ?
yes on machine have pppoe server that handle 5-6k users (pppXXXX ) interfaces
is it possible to add in code if interface is pppoe or ipoe to not server ?
I'm guessing they have different processing method. For example, unbound only binds the physical interface corresponding to the pppoe interface, and ignores the connection and disconnection of pppoe. I'm not sure about smartdns, but maybe the problem handles when the number of pppoe interfaces changes?
Same problem i see in snmpd and after team add options : include_ifmib_iface_prefix eth vlan lo bond team with this options enable to see info for only this device, if not set and have 5k pppoe or ipoe interface daemon stay same on 100% load or more.... idea is ppp and ipoe interface is virtual and no need to binds smartdns , only on this type need to bind : eth vlan lo bond team or to have options to set .
P.S. in time when run smartdns for test users stay on 5200-5250 no disconnect or connect (may be 1-2 users connect disconnect).
P.S.2 this is knot-resolver :
-- Network interface configuration net.listen('0.0.0.0', 53, { kind = 'dns' }) net.listen('0.0.0.0', 853, { kind = 'tls' })
i think bind same use default not binds to all interfacess
Edit:
after set in config :
bind [::]:53@bond0.0101 -nftset bind [::]:53@lo -nftset
all is fine but need to set specific interface where is run IP + Default GW.
If you have time to add options if device is pppoe or ipoe to not list i think will fix all problems...
One more now make test
set in conf file ip address on uplink interface but is same :
bind 103.21.32.1:53 -nftset
server 8.8.8.8 server-tls 8.8.8.8 server 8.8.4.4 server-tls 8.8.4.4 server 1.1.1.1 server-tls 1.1.1.1 server 1.0.0.1 server-tls 1.0.0.1 server 9.9.9.9 server 149.112.112.112 server 2620:fe::fe server 2620:fe::9 server 2001:4860:4860::8888 server 2001:4860:4860::8844 server 2606:4700::1111 server-tls 2606:4700::1111 server 2606:4700:4700::1001 server-tls 2606:4700:4700::1001
cache-size 1000000 cache-persist yes cache-file /var/run/smartdns.cache cache-checkpoint-time 86400 prefetch-domain yes serve-expired yes serve-expired-ttl 259200 serve-expired-prefetch-time 21600
tcp-idle-time 120
force-qtype-SOA 65
dualstack-ip-selection yes
log-level error
log-file /var/log/smartdns/smartdns.log log-size 256k log-num 2
11.43% smartdns [.] netlink_msg_to_ifaddr
7.35% smartdns [.] printf_core
5.24% smartdns [.] memcpy
3.77% smartdns [.] _dns_get_domain_from_packet
3.32% smartdns [.] __fwritex
2.98% smartdns [.] memset
2.53% smartdns [.] __libc_free
2.11% smartdns [.] alloc_slot
2.06% smartdns [.] _dns_server_process_local_ptr
1.98% smartdns [.] fmt_u
1.96% smartdns [.] dict_hash
1.71% smartdns [.] enframe
1.61% smartdns [.] pad
1.53% smartdns [.] get_meta
1.24% smartdns [.] vfprintf
1.10% smartdns [.] _dns_encode_domain
1.00% smartdns [.] _dns_left_len
0.94% smartdns [.] _dns_read_short
0.92% smartdns [.] __libc_malloc_impl
0.89% smartdns [.] __malloc_allzerop
0.84% smartdns [.] jhash
0.72% smartdns [.] _dns_decode_an
0.70% smartdns [.] vsnprintf
0.64% smartdns [.] _dns_get_domain_offset
0.64% smartdns [.] _dns_client_send_packet
0.62% [vdso] [.] __vdso_clock_gettime
0.57% smartdns [.] strncmp
0.55% smartdns [.] out
0.51% smartdns [.] __pthread_mutex_unlock
0.51% smartdns [.] tlog_vext
0.49% smartdns [.] freeifaddrs
0.46% smartdns [.] pop_arg
0.45% smartdns [.] getint
0.44% smartdns [.] _dns_decode_body
0.44% smartdns [.] size_to_class
0.44% smartdns [.] _dns_client_recv
0.44% smartdns [.] __netlink_enumerate
0.43% smartdns [.] _dns_server_process_answer
0.42% smartdns [.] _dns_decode_qr_head
0.38% smartdns [.] sn_write
0.37% smartdns [.] _dns_write_short
0.36% smartdns [.] sccp
0.36% smartdns [.] __pthread_rwlock_trywrlock
0.36% smartdns [.] memchr
0.34% smartdns [.] _dns_init_context_by_rrs
0.34% smartdns [.] __stpncpy
0.33% smartdns [.] _fast_ping_period_run
0.32% smartdns [.] __pthread_mutex_lock
0.32% smartdns [.] _dns_server_process_answer_A
0.30% smartdns [.] _dns_get_RAW
0.30% smartdns [.] jhash
0.29% smartdns [.] ntohs
0.29% smartdns [.] dns_get_rrs_start
0.29% smartdns [.] tlog_ext
0.29% smartdns [.] snprintf
0.28% smartdns [.] _dns_client_create_socket_udp
0.28% smartdns [.] _dns_rr_add_end
0.27% smartdns [.] __clock_gettime
0.27% smartdns [.] do_ssl3_write
0.27% smartdns [.] rol32
0.27% smartdns [.] _dns_server_get_conf_ttl
0.26% smartdns [.] __syscall_ret
0.23% smartdns [.] tls13_enc
0.22% smartdns [.] _dns_client_query_release
0.22% smartdns [.] dns_server_resolve_callback
0.22% smartdns [.] calloc
0.22% smartdns [.] get_stride
0.22% smartdns [.] _dns_read_int
0.22% smartdns [.] BIO_read
0.22% smartdns [.] _dns_server_process_cache_packet
0.22% smartdns [.] dns_cache_lookup
0.22% smartdns [.] copy_addr
0.21% smartdns [.] _dns_get_qr_head
0.21% smartdns [.] jhash
0.21% smartdns [.] __intscan
0.21% smartdns [.] aes_gcm_cipher
0.21% smartdns [.] gcm_ghash_avx
0.20% smartdns [.] _dns_add_RAW
could you try to bind interface like bind [::]:53@eth0.1
but i think after i bind to :
bind [::]:53@bond0.0101 -nftset bind [::]:53@lo -nftset
users that is on pppoe and ipoe not resolv like not see dns .
is there options to debug why service is load
add line 3825:
static int _dns_server_process_local_ptr(struct dns_request *request)
{
struct ifaddrs *ifaddr = NULL;
struct ifaddrs *ifa = NULL;
unsigned char *addr = NULL;
char reverse_addr[128] = {0};
int found = 0;
> return -1;
if (getifaddrs(&ifaddr) == -1) {
after add see down is info
try to make dig local from server dig cisco.com 127.0.0.1
;; communications error to 127.0.0.1#53: timed out
and in config set : bind [::]:53 -nftset
Threads: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.7 sy, 0.0 ni, 33.1 id, 0.0 wa, 0.0 hi, 65.4 si, 0.0 st %Cpu1 : 1.3 us, 1.3 sy, 0.0 ni, 47.7 id, 0.0 wa, 0.0 hi, 49.7 si, 0.0 st %Cpu2 : 2.0 us, 2.0 sy, 0.0 ni, 39.9 id, 0.0 wa, 0.0 hi, 56.1 si, 0.0 st %Cpu3 : 2.0 us, 46.7 sy, 0.0 ni, 8.7 id, 0.0 wa, 0.0 hi, 42.7 si, 0.0 st %Cpu4 : 2.8 us, 3.5 sy, 0.0 ni, 48.6 id, 0.0 wa, 0.0 hi, 45.1 si, 0.0 st %Cpu5 : 1.4 us, 0.7 sy, 0.0 ni, 42.9 id, 0.0 wa, 0.0 hi, 55.1 si, 0.0 st %Cpu6 : 2.0 us, 2.7 sy, 0.0 ni, 38.7 id, 0.0 wa, 0.0 hi, 56.7 si, 0.0 st %Cpu7 : 1.4 us, 2.0 sy, 0.0 ni, 49.0 id, 0.0 wa, 0.0 hi, 47.6 si, 0.0 st %Cpu8 : 2.1 us, 8.9 sy, 0.7 ni, 39.0 id, 0.0 wa, 0.0 hi, 49.3 si, 0.0 st %Cpu9 : 1.4 us, 1.4 sy, 0.0 ni, 49.7 id, 0.0 wa, 0.0 hi, 47.6 si, 0.0 st %Cpu10 : 2.0 us, 2.0 sy, 0.7 ni, 47.7 id, 0.0 wa, 0.0 hi, 47.7 si, 0.0 st %Cpu11 : 1.4 us, 0.7 sy, 0.0 ni, 44.9 id, 0.7 wa, 0.0 hi, 52.2 si, 0.0 st MiB Mem : 31955.6 total, 27079.2 free, 2736.0 used, 3190.4 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29219.6 avail Mem
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
429254 root 20 0 304.9m 16.8m 99.9 0.1 1:27.11 R smartdns -c smartdns.conf 429258 root 20 0 304.9m 16.8m 6.0 0.1 0:05.09 S smartdns -c smartdns.conf 429256 root 15 -5 304.9m 16.8m 2.6 0.1 0:01.76 S smartdns -c smartdns.conf 429255 root 20 0 304.9m 16.8m 2.0 0.1 0:01.34 S smartdns -c smartdns.conf 429257 root 20 0 304.9m 16.8m 2.0 0.1 0:01.28 S smartdns -c smartdns.conf
3.90% libc.so.6 [.] 0x000000000016d5ca
3.10% smartdns [.] _dns_get_domain_from_packet
1.77% smartdns [.] dict_hash
1.74% smartdns [.] _dns_encode_domain
1.63% libc.so.6 [.] malloc
1.33% smartdns [.] _dns_client_send_packet
1.32% libc.so.6 [.] pthread_mutex_lock
1.22% libc.so.6 [.] pthread_mutex_unlock
1.16% smartdns [.] dns_cache_lookup
1.06% smartdns [.] dns_cache_invalidate
0.88% smartdns [.] _dns_decode_an
0.86% smartdns [.] _dns_server_process_answer
0.82% libc.so.6 [.] cfree
0.80% libc.so.6 [.] clock_gettime
0.78% smartdns [.] _dns_get_domain_offset
0.60% libc.so.6 [.] connect
0.58% smartdns [.] dns_packet_init
0.58% smartdns [.] _dns_client_get_request
0.56% smartdns [.] tlog_vext
0.55% [vdso] [.] __vdso_clock_gettime
0.55% smartdns [.] _dns_read_short
0.54% libcrypto.so.3 [.] EVP_CIPHER_CTX_get_iv_length
0.50% smartdns [.] _dns_server_request_release_complete
0.48% libc.so.6 [.] pthread_cond_signal
0.47% libc.so.6 [.] pthread_cond_wait
0.42% smartdns [.] _dns_client_recv
0.41% smartdns [.] _dns_ip_address_check_add
0.41% smartdns [.] _dns_client_create_socket_udp
0.41% smartdns [.] jhash
0.41% smartdns [.] _dns_server_get_answer
0.40% smartdns [.] dns_decode
0.40% libc.so.6 [.] 0x000000000016ee5f
0.40% smartdns [.] get_tick_count
0.39% smartdns [.] tlog_ext
0.39% libcrypto.so.3 [.] 0x0000000000339a74
0.37% smartdns [.] _dns_client_query_release
0.37% [vdso] [.] __vdso_time
0.37% libc.so.6 [.] sprintf
0.36% libc.so.6 [.] 0x000000000008f051
0.35% [vdso] [.] __vdso_gettimeofday
0.35% smartdns [.] _dns_rr_add_end
0.33% smartdns [.] _dns_server_process_udp_one
0.33% smartdns [.] _dns_client_work
0.33% smartdns [.] _dns_decode_qr_head
0.32% smartdns [.] _dns_server_reply_passthrough
0.32% libc.so.6 [.] __errno_location
0.31% smartdns [.] _dns_add_RAW
0.31% smartdns [.] _dns_server_get_conf_ttl
0.31% smartdns [.] fast_ping_start
0.31% libcrypto.so.3 [.] BIO_write
0.29% smartdns [.] dns_get_rrs_start
0.29% smartdns [.] dns_cache_update
0.29% libc.so.6 [.] pthread_rwlock_unlock
0.28% smartdns [.] _fast_ping_period_run
0.28% smartdns [.] _tlog_vprintf
0.27% libc.so.6 [.] recvmsg
0.27% libc.so.6 [.] 0x000000000016f047
0.27% smartdns [.] pthread_mutex_lock@plt
0.27% smartdns [.] _dns_server_reply_udp
0.27% smartdns [.] _dns_server_process_answer_A
0.26% libcrypto.so.3 [.] CRYPTO_malloc
0.26% smartdns [.] _dns_get_qr_head
0.26% smartdns [.] _dns_client_process
0.26% smartdns [.] _dns_server_do_query
0.25% smartdns [.] _dns_reply_inpacket
0.25% libc.so.6 [.] __close
0.25% smartdns [.] pthread_mutex_unlock@plt
0.25% smartdns [.] _tlog_root_log_buffer
0.25% smartdns [.] _dns_decode_rr_head
0.25% smartdns [.] jhash
0.24% smartdns [.] _dns_result_callback
0.24% smartdns [.] _dns_read_int
0.24% libc.so.6 [.] pthread_rwlock_rdlock
0.24% libc.so.6 [.] getaddrinfo
0.24% libc.so.6 [.] 0x0000000000058949
0.24% libc.so.6 [.] 0x000000000008f0c0
0.24% smartdns [.] _dns_add_rrs_start
0.24% smartdns [.] _dns_server_ping_result
disable log.
4.09% smartdns [.] _dns_get_domain_from_packet
2.09% smartdns [.] _dns_client_send_packet
1.95% libc.so.6 [.] 0x000000000016d5ca
1.86% smartdns [.] _dns_encode_domain
1.81% libc.so.6 [.] malloc
1.33% smartdns [.] dict_hash
1.31% libc.so.6 [.] pthread_mutex_lock
1.24% libc.so.6 [.] getaddrinfo
1.06% smartdns [.] _dns_server_process_answer
0.99% libc.so.6 [.] cfree
0.93% libc.so.6 [.] pthread_mutex_unlock
0.83% smartdns [.] _dns_get_domain_offset
0.82% smartdns [.] _dns_decode_an
0.77% smartdns [.] _dns_client_create_socket_udp
0.76% libc.so.6 [.] epoll_wait
0.76% smartdns [.] dns_get_rrs_start
0.71% smartdns [.] _dns_client_get_request
0.70% smartdns [.] _dns_server_process_answer_A
0.70% smartdns [.] _dns_ip_address_check_add
0.69% smartdns [.] _dns_read_short
0.68% libc.so.6 [.] pthread_rwlock_unlock
0.65% [vdso] [.] __vdso_clock_gettime
0.63% smartdns [.] pthread_mutex_unlock@plt
0.62% libc.so.6 [.] pthread_cond_wait
0.57% smartdns [.] dns_cache_lookup
0.54% smartdns [.] fast_ping_start
0.54% libc.so.6 [.] connect
0.53% smartdns [.] _dns_client_recv
0.52% smartdns [.] jhash
0.52% smartdns [.] _dns_rr_add_end
0.51% smartdns [.] _dns_client_process
0.50% libc.so.6 [.] pthread_rwlock_rdlock
0.50% smartdns [.] _dns_server_process_udp_one
0.49% smartdns [.] _dns_client_process_udp
0.49% smartdns [.] _fast_ping_process_icmp
0.48% libcrypto.so.3 [.] CRYPTO_free
0.48% smartdns [.] memcpy@plt
0.47% smartdns [.] _dns_client_query_release
0.47% smartdns [.] _fast_ping_close_host_sock
0.45% libc.so.6 [.] clock_gettime
0.42% smartdns [.] jhash
0.42% smartdns [.] tlog_vext
0.42% smartdns [.] _DNS_client_create_socket_tls
0.39% smartdns [.] _dns_add_RAW
0.39% smartdns [.] _dns_decode_qr_head
0.39% smartdns [.] _dns_get_qr_head
0.38% smartdns [.] _dns_decode_body
0.38% libcrypto.so.3 [.] EVP_CIPHER_CTX_get_iv_length
0.36% libc.so.6 [.] 0x00000000000a0cdc
0.36% smartdns [.] tlog_ext
0.36% smartdns [.] _dns_replied_check_add
0.35% libc.so.6 [.] 0x000000000008f0b3
0.35% libcrypto.so.3 [.] CRYPTO_gcm128_decrypt_ctr32
0.34% smartdns [.] _dns_cache_remove_by_domain
0.33% libcrypto.so.3 [.] OPENSSL_sk_pop_free
0.33% smartdns [.] dns_cache_is_soa
0.33% smartdns [.] _dns_server_get_conf_ttl
0.33% libc.so.6 [.] 0x00000000000a0368
0.33% libc.so.6 [.] inet_ntop
0.32% smartdns [.] _dns_server_process_cname_pre
0.32% smartdns [.] _dns_server_period_run
0.32% smartdns [.] dns_cache_invalidate
0.32% smartdns [.] _dns_decode_qd
0.32% libcrypto.so.3 [.] OSSL_PARAM_construct_size_t
0.31% smartdns [.] _dns_decode_head
0.31% libcrypto.so.3 [.] OSSL_PARAM_locate
0.31% libc.so.6 [.] 0x00000000000a10ec
0.31% smartdns [.] _dns_server_ping_result
0.31% smartdns [.] _fast_ping_period_run
0.30% libcrypto.so.3 [.] 0x00000000002de770
0.30% libcrypto.so.3 [.] X509_VERIFY_PARAM_inherit
0.30% libcrypto.so.3 [.] OSSL_PARAM_get_int32
0.30% libcrypto.so.3 [.] CRYPTO_gcm128_setiv
0.28% libc.so.6 [.] 0x000000000016ccb7
0.28% smartdns [.] pthread_mutex_lock@plt
0.28% libc.so.6 [.] 0x000000000008f0c0
0.28% libc.so.6 [.] 0x0000000000039ac7
0.28% smartdns [.] _dns_server_get_answer
Do not filter kernel functions
sorry
28.11% [kernel] [k] __ipv6_dev_get_saddr
11.07% [kernel] [k] l3mdev_master_ifindex_rcu
7.64% [pppoe] [k] pppoe_rcv
3.45% [kernel] [k] ipv6_dev_get_saddr
1.89% [kernel] [k] __dev_queue_xmit
1.59% [kernel] [k] fib_table_lookup
1.46% [nf_tables] [k] nft_do_chain
1.43% [kernel] [k] __netif_receive_skb_core.constprop.0
1.17% [ixgbe] [k] ixgbe_clean_rx_irq
1.09% [kernel] [k] memcmp
1.01% [nf_tables] [k] __nft_rbtree_lookup
1.00% [kernel] [k] dev_queue_xmit_nit
0.99% [kernel] [k] strncpy
0.89% [kernel] [k] skb_clone
0.86% [kernel] [k] __copy_skb_header
0.84% [nf_tables] [k] nft_rhash_lookup
0.80% [kernel] [k] kmem_cache_free
0.80% [nf_conntrack] [k] __nf_conntrack_find_get.isra.0
0.80% [kernel] [k] skb_release_data
0.77% [kernel] [k] __skb_flow_dissect
0.71% [ixgbe] [k] ixgbe_tx_map
0.67% [kernel] [k] csum_partial_copy_generic
0.56% [kernel] [k] ip_finish_output2
0.55% [kernel] [k] kmem_cache_alloc
0.53% [kernel] [k] ip_route_input_slow
0.50% [kernel] [k] nf_hook_slow
0.50% [kernel] [k] ktime_get_with_offset
0.50% [vlan_mon] [k] vlan_pt_recv
0.49% [kernel] [k] dev_hard_start_xmit
0.45% [kernel] [k] kfree_skb_reason
0.43% [ixgbe] [k] ixgbe_xmit_frame_ring
0.42% [kernel] [k] ip_rcv_core.constprop.0
0.41% [nf_tables] [k] nft_lookup_eval
0.38% [kernel] [k] __siphash_unaligned
0.36% [kernel] [k] vlan_do_receive
0.36% [kernel] [k] csum_partial
0.36% [kernel] [k] ip_forward
0.35% [kernel] [k] netif_skb_features
0.34% [kernel] [k] process_backlog
0.34% [kernel] [k] ___slab_alloc.isra.0
0.34% [kernel] [k] validate_xmit_skb.isra.0
0.33% [kernel] [k] get_rps_cpu
0.33% [kernel] [k] page_frag_free
0.30% [nf_tables] [k] nft_meta_get_eval
0.30% [nf_conntrack] [k] nf_conntrack_in
0.29% [kernel] [k] ip_output
0.29% [nf_nat] [k] l4proto_manip_pkt
0.28% [kernel] [k] kmem_cache_free_bulk.part.0
0.28% [kernel] [k] read_tsc
0.28% [kernel] [k] __rcu_read_unlock
0.27% [kernel] [k] netdev_core_pick_tx
0.27% [kernel] [k] _raw_spin_lock_bh
0.27% [kernel] [k] _raw_spin_lock_irqsave
0.26% [kernel] [k] sch_direct_xmit
0.26% [kernel] [k] netdev_pick_tx
0.26% [kernel] [k] __local_bh_enable_ip
0.24% [pppoe] [k] pppoe_xmit
0.23% [kernel] [k] dev_gro_receive
0.22% [kernel] [k] __slab_free.isra.0
0.22% [ixgbe] [k] ixgbe_clean_tx_irq
0.22% [kernel] [k] __rcu_read_lock
0.21% [kernel] [k] skb_network_protocol
0.20% [ppp_generic] [k] __ppp_xmit_process
0.20% [nf_nat] [k] nf_nat_ipv4_manip_pkt
0.20% [kernel] [k] fib_validate_source
0.20% [kernel] [k] dma_map_page_attrs
0.20% [ppp_generic] [k] ppp_push
0.20% [ixgbe] [k] ixgbe_process_skb_fields
0.19% [ppp_generic] [k] ppp_start_xmit
0.19% [kernel] [k] _raw_spin_lock
0.18% [kernel] [k] inet_gro_receive
0.17% [nf_tables] [k] nft_rbtree_lookup
0.17% [kernel] [k] vlan_dev_hard_start_xmit
0.17% [kernel] [k] napi_consume_skb
0.17% [nf_xnatlog] [k] xnatlog_hook
0.17% [kernel] [k] enqueue_to_backlog
0.16% smartdns [.] _dns_get_domain_from_packet
0.16% [kernel] [k] __list_del_entry_valid
modify dns_server.c:1181
static int _dns_server_reply_udp(struct dns_request *request, struct dns_server_conn_udp *udpserver,
unsigned char *inpacket, int inpacket_len)
{
int send_len = 0;
struct iovec iovec[1];
struct msghdr msg;
struct cmsghdr *cmsg;
char msg_control[64];
if (atomic_read(&server.run) == 0 || inpacket == NULL || inpacket_len <= 0) {
return -1;
}
> goto use_send;
iovec[0].iov_base = inpacket;
iovec[0].iov_len = inpacket_len;
same :
28.07% [kernel] [k] __ipv6_dev_get_saddr
11.73% [kernel] [k] l3mdev_master_ifindex_rcu
7.44% [pppoe] [k] pppoe_rcv
3.92% [kernel] [k] ipv6_dev_get_saddr
2.04% [kernel] [k] __dev_queue_xmit
1.55% [kernel] [k] fib_table_lookup
1.49% [nf_tables] [k] nft_do_chain
1.35% [kernel] [k] __netif_receive_skb_core.constprop.0
1.29% [ixgbe] [k] ixgbe_clean_rx_irq
1.09% [kernel] [k] memcmp
1.07% [nf_tables] [k] __nft_rbtree_lookup
1.05% [kernel] [k] strncpy
0.89% [kernel] [k] kmem_cache_free
0.89% [kernel] [k] dev_queue_xmit_nit
0.86% [kernel] [k] __skb_flow_dissect
0.83% [kernel] [k] skb_clone
0.80% [kernel] [k] csum_partial_copy_generic
0.78% [nf_tables] [k] nft_rhash_lookup
0.72% [nf_conntrack] [k] __nf_conntrack_find_get.isra.0
0.70% [kernel] [k] skb_release_data
0.70% [kernel] [k] __copy_skb_header
0.66% [ixgbe] [k] ixgbe_tx_map
0.52% [kernel] [k] ip_finish_output2
0.52% [kernel] [k] ip_route_input_slow
0.51% [kernel] [k] kmem_cache_alloc
0.48% [kernel] [k] dev_hard_start_xmit
0.47% [kernel] [k] nf_hook_slow
0.46% [kernel] [k] kfree_skb_reason
0.44% [vlan_mon] [k] vlan_pt_recv
0.43% [kernel] [k] __siphash_unaligned
0.38% [kernel] [k] ip_rcv_core.constprop.0
0.35% [kernel] [k] vlan_do_receive
0.35% [kernel] [k] page_frag_free
0.35% [nf_tables] [k] nft_lookup_eval
0.34% [kernel] [k] netif_skb_features
0.34% [ixgbe] [k] ixgbe_xmit_frame_ring
0.33% [kernel] [k] process_backlog
0.33% [kernel] [k] kmem_cache_free_bulk.part.0
0.33% [kernel] [k] _raw_spin_lock_irqsave
0.32% [kernel] [k] validate_xmit_skb.isra.0
0.32% [kernel] [k] ___slab_alloc.isra.0
0.31% [kernel] [k] get_rps_cpu
0.31% [nf_tables] [k] nft_meta_get_eval
0.30% [kernel] [k] __rcu_read_unlock
0.29% [kernel] [k] _raw_spin_lock_bh
0.28% [kernel] [k] sch_direct_xmit
0.28% [nf_nat] [k] l4proto_manip_pkt
0.27% [kernel] [k] ip_forward
0.27% [kernel] [k] __local_bh_enable_ip
0.27% [kernel] [k] netdev_core_pick_tx
0.26% [nf_conntrack] [k] nf_conntrack_in
0.26% [kernel] [k] dev_gro_receive
0.25% [kernel] [k] ktime_get_with_offset
0.25% [kernel] [k] ip_output
0.23% [pppoe] [k] pppoe_xmit
0.23% [kernel] [k] fib_validate_source
0.22% [kernel] [k] netdev_pick_tx
0.21% [ppp_generic] [k] ppp_start_xmit
0.21% [kernel] [k] skb_network_protocol
0.21% [kernel] [k] __slab_free.isra.0
0.20% [kernel] [k] __rcu_read_lock
0.20% [kernel] [k] _raw_spin_lock
0.20% [kernel] [k] inet_gro_receive
0.20% [ixgbe] [k] ixgbe_process_skb_fields
0.20% [kernel] [k] dma_map_page_attrs
0.19% [kernel] [k] csum_partial
0.19% [ppp_generic] [k] __ppp_xmit_process
0.19% [nf_nat] [k] nf_nat_ipv4_manip_pkt
0.19% [kernel] [k] read_tsc
0.18% [kernel] [k] __list_del_entry_valid
0.17% [ppp_generic] [k] ppp_push
0.17% [kernel] [k] napi_consume_skb
0.17% [kernel] [k] vlan_dev_hard_start_xmit
0.17% [ixgbe] [k] ixgbe_clean_tx_irq
0.16% [kernel] [k] kmem_cache_alloc_bulk
0.15% [kernel] [k] napi_build_skb
0.15% [nf_tables] [k] nft_rbtree_lookup
0.15% [ppp_generic] [k] ppp_xmit_process
what is you kernel version?
Hi
I try and install latest version Release 43 , but after run smartdns process stay on 100% load
try to clena conf and remove all , remove cache ,remove logs ,..... but still on 100%
in machine have 5000 users