search

pymumu / smartdns

A local DNS server to obtain the fastest website IP for the best Internet experience, support DoT, DoH. 一个本地DNS服务器，获取最快的网站IP，获得最佳上网体验，支持DoH，DoT。
https://pymumu.github.io/smartdns/
GNU General Public License v3.0
8.25k stars 1.06k forks source link

Release 43 / High CPU load #1477

Closed micron10 closed 1 year ago

micron10 commented 1 year ago

Hi

I try and install latest version Release 43 , but after run smartdns process stay on 100% load

try to clena conf and remove all , remove cache ,remove logs ,..... but still on 100%

in machine have 5000 users

pymumu commented 1 year ago

Please add more information, including configuration, cpu usage (top -H -p [pid]), logs, and number of clients.

micron10 commented 1 year ago

Hi @pymumu

Thanks for fast respons,

users is ~ 5600

this is first test config , on time of test i disable log nfset cache , but load is stay on 100% load

top - 12:14:58 up 1 day, 8:56, 1 user, load average: 1.36, 1.02, 0.81 Threads: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.7 sy, 0.0 ni, 66.0 id, 0.0 wa, 0.0 hi, 32.6 si, 0.0 st %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 71.2 id, 0.0 wa, 0.0 hi, 28.8 si, 0.0 st %Cpu2 : 0.7 us, 0.7 sy, 0.0 ni, 68.2 id, 0.0 wa, 0.0 hi, 30.4 si, 0.0 st %Cpu3 : 0.7 us, 0.7 sy, 0.0 ni, 63.7 id, 0.0 wa, 0.0 hi, 34.9 si, 0.0 st %Cpu4 : 10.7 us, 54.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 34.7 si, 0.0 st %Cpu5 : 0.7 us, 0.7 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 38.3 si, 0.0 st %Cpu6 : 0.0 us, 0.7 sy, 0.0 ni, 72.6 id, 0.0 wa, 0.0 hi, 26.7 si, 0.0 st %Cpu7 : 0.7 us, 0.7 sy, 0.0 ni, 73.8 id, 0.0 wa, 0.0 hi, 24.8 si, 0.0 st %Cpu8 : 0.7 us, 0.7 sy, 0.7 ni, 66.7 id, 0.0 wa, 0.0 hi, 31.3 si, 0.0 st %Cpu9 : 0.7 us, 0.0 sy, 0.7 ni, 62.1 id, 0.0 wa, 0.0 hi, 36.6 si, 0.0 st %Cpu10 : 0.0 us, 0.0 sy, 0.0 ni, 62.2 id, 0.0 wa, 0.0 hi, 37.8 si, 0.0 st %Cpu11 : 0.7 us, 0.0 sy, 0.0 ni, 66.2 id, 0.0 wa, 0.0 hi, 33.1 si, 0.0 st MiB Mem : 31955.6 total, 28996.4 free, 2539.1 used, 1433.6 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29416.5 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

1125943 root 20 0 315.5m 23.8m 99.9 0.1 0:23.52 R smartdns -c /etc/smartdns/smartdns.conf 1125947 root 20 0 315.5m 23.8m 4.0 0.1 0:01.15 S smartdns -c /etc/smartdns/smartdns.conf 1125944 root 20 0 315.5m 23.8m 1.3 0.1 0:00.20 S smartdns -c /etc/smartdns/smartdns.conf 1125945 root 15 -5 315.5m 23.8m 1.3 0.1 0:00.39 S smartdns -c /etc/smartdns/smartdns.conf 1125946 root 20 0 315.5m 23.8m 0.7 0.1 0:00.31 S smartdns -c /etc/smartdns/smartdns.conf

this is config:

user root

bind [::]:53 -nftset

server 8.8.8.8 server-tls 8.8.8.8 server 8.8.4.4 server-tls 8.8.4.4 server 1.1.1.1 server-tls 1.1.1.1 server 1.0.0.1 server-tls 1.0.0.1 server 9.9.9.9 server 149.112.112.112 server 2620:fe::fe server 2620:fe::9 server 2001:4860:4860::8888 server 2001:4860:4860::8844 server 2606:4700::1111 server-tls 2606:4700::1111 server 2606:4700:4700::1001 server-tls 2606:4700:4700::1001

cache-size 131072 cache-persist yes cache-file /var/run/smartdns.cache cache-checkpoint-time 86400 prefetch-domain yes serve-expired yes serve-expired-ttl 259200 serve-expired-prefetch-time 21600

tcp-idle-time 120

force-qtype-SOA 65

Enable IPV4, IPV6 dual stack IP optimization selection strategy

dualstack-ip-selection-threshold [num] (0~1000)

dualstack-ip-allow-force-AAAA [yes|no]

dualstack-ip-selection yes

set log level

log-level: [level], level=off, fatal, error, warn, notice, info, debug

log-file: file path of log file.

log-console [yes|no]: output log to console.

log-size: size of each log file, support k,m,g

log-num: number of logs, 0 means disable log

log-level info

log-file /var/log/smartdns/smartdns.log log-size 256k log-num 2

enalbe DNS64 feature

dns64 [ip/subnet]

dns64 64:ff9b::/96

nftset-timeout yes

enable nftset debug, check nftset setting result, output log when error.

nftset-debug [yes|no]

nftset-debug no

this is logs :

smartdns.log smartdns.log-20230822-121509.gz smartdns.log-20230822-121511.gz

pymumu commented 1 year ago

Try increasing the cache size, say 1000000.

https://github.com/pymumu/smartdns/blob/1054229efb31a4f9cddb4f2f163a4c225209c45f/src/dns_cache.c#L33-L40

You can also try to modify the 33 lines of code as follows

DECLARE_HASHTABLE(cache_hash, 20);
micron10 commented 1 year ago

Oki change to 20 in dns_cache and change in config to 1000000

but same :

top - 02:11:46 up 1 day, 22:52, 2 users, load average: 0.58, 0.29, 0.21 Threads: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.0 sy, 0.0 ni, 91.0 id, 0.0 wa, 0.0 hi, 8.3 si, 0.0 st %Cpu1 : 0.7 us, 1.3 sy, 0.0 ni, 84.6 id, 0.0 wa, 0.0 hi, 13.4 si, 0.0 st %Cpu2 : 0.0 us, 0.7 sy, 0.0 ni, 85.8 id, 0.0 wa, 0.0 hi, 13.5 si, 0.0 st %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 91.3 id, 0.0 wa, 0.0 hi, 8.7 si, 0.0 st %Cpu4 : 0.7 us, 0.0 sy, 0.0 ni, 89.8 id, 0.0 wa, 0.0 hi, 9.5 si, 0.0 st %Cpu5 : 0.0 us, 0.7 sy, 0.0 ni, 89.2 id, 0.0 wa, 0.0 hi, 10.1 si, 0.0 st %Cpu6 : 0.7 us, 0.7 sy, 0.0 ni, 87.3 id, 0.0 wa, 0.0 hi, 11.3 si, 0.0 st %Cpu7 : 0.7 us, 0.7 sy, 0.0 ni, 87.2 id, 0.0 wa, 0.0 hi, 11.4 si, 0.0 st %Cpu8 : 15.3 us, 42.7 sy, 0.0 ni, 30.7 id, 0.0 wa, 0.0 hi, 11.3 si, 0.0 st %Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 90.5 id, 0.0 wa, 0.0 hi, 9.5 si, 0.0 st %Cpu10 : 0.7 us, 0.7 sy, 0.0 ni, 87.8 id, 0.0 wa, 0.0 hi, 10.8 si, 0.0 st %Cpu11 : 0.0 us, 0.7 sy, 0.0 ni, 93.0 id, 0.0 wa, 0.0 hi, 6.3 si, 0.0 st MiB Mem : 31955.6 total, 28725.1 free, 2519.3 used, 1717.1 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29436.3 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

1630082 root 20 0 322.9m 32.0m 70.0 0.1 0:31.08 R smartdns -c /etc/smartdns/smartdns.conf 1630086 root 20 0 322.9m 32.0m 1.3 0.1 0:00.97 S smartdns -c /etc/smartdns/smartdns.conf 1630085 root 20 0 322.9m 32.0m 0.7 0.1 0:00.23 S smartdns -c /etc/smartdns/smartdns.conf 1630083 root 20 0 322.9m 32.0m 0.0 0.1 0:00.21 S smartdns -c /etc/smartdns/smartdns.conf 1630084 root 15 -5 322.9m 32.0m 0.0 0.1 0:00.24 S smartdns -c /etc/smartdns/smartdns.conf

smartdns.log smartdns.log-20230823-021222.gz smartdns.log-20230823-021224.gz

its same when stop knot-resolve that work on 3-4 % and run smartdns process is go to 100%

m.

pymumu commented 1 year ago

https://github.com/pymumu/smartdns/blob/9307855f7c99c9a9ab0df592bab49712f686fb5b/src/dns_cache.c#L624-L631

try to modify the 626 and 630 lines of code as follows

                ttl = *now - dns_cache->info.replace_time;
                if (ttl < ttl_inactive_pre || inactive_precallback == NULL) {
 626->                       break;
                }

                if (callback_num >= max_callback_num) {
 630->                       break;
                }
micron10 commented 1 year ago

Hi @pymumu

Change both line to break.

load is stay same on 100%

in config set : cache-size 1000000

see logs i run in debug for more info smartdns.log smartdns.log-20230823-085109.gz smartdns.log-20230823-085110.gz

pymumu commented 1 year ago

run perf top -p [pid], collect log.

micron10 commented 1 year ago

PerfTop: 4134 irqs/sec kernel:80.7% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 1853738)

15.54%  [kernel]          [k] l3mdev_master_ifindex_rcu
12.02%  [kernel]          [k] __ipv6_dev_get_saddr
 6.00%  [kernel]          [k] __snmp6_fill_stats64.constprop.0
 3.91%  [kernel]          [k] ipv6_dev_get_saddr
 3.70%  [pppoe]           [k] pppoe_rcv
 2.50%  [kernel]          [k] rtnl_fill_ifinfo.constprop.0.isra.0
 2.11%  [kernel]          [k] inet6_fill_ifla6_attrs
 1.75%  [kernel]          [k] inet_dump_ifaddr
 1.63%  libc.so.6         [.] 0x0000000000137a10
 1.46%  libc.so.6         [.] 0x0000000000137cb8
 1.39%  libc.so.6         [.] 0x0000000000137ca1
 1.33%  libc.so.6         [.] 0x00000000001379f9
 1.33%  [kernel]          [k] nla_put
 1.12%  libc.so.6         [.] 0x0000000000137a00
 1.06%  libc.so.6         [.] 0x0000000000137ca8
 1.04%  [kernel]          [k] __dev_queue_xmit
 0.95%  libc.so.6         [.] 0x0000000000137a09
 0.82%  [nf_tables]       [k] nft_do_chain
 0.81%  libc.so.6         [.] 0x0000000000137cb1
 0.80%  [kernel]          [k] fib_table_lookup
 0.76%  [kernel]          [k] __netif_receive_skb_core.constprop.0
 0.75%  libc.so.6         [.] 0x00000000001379f0
 0.74%  libc.so.6         [.] 0x00000000001379fc
 0.72%  [kernel]          [k] memset_orig
 0.63%  [ixgbe]           [k] ixgbe_clean_rx_irq
 0.61%  [kernel]          [k] memcmp
 0.58%  libc.so.6         [.] 0x0000000000137c98
 0.57%  [nf_tables]       [k] __nft_rbtree_lookup
 0.57%  libc.so.6         [.] 0x0000000000137aa8
 0.57%  [kernel]          [k] strncpy
pymumu commented 1 year ago

Use the following command to compile smartdns to the debug version. make DEBUG=1 all -j8 Use the perf top -p [pid] command to view function information. [pid] is the pid of smartdns process.

micron10 commented 1 year ago

yes i set smartdns pid

make with DEBUG=1

PerfTop: 4065 irqs/sec kernel:82.9% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 1857172)

16.71%  [kernel]          [k] l3mdev_master_ifindex_rcu
12.39%  [kernel]          [k] __ipv6_dev_get_saddr
 5.29%  [kernel]          [k] __snmp6_fill_stats64.constprop.0
 4.36%  [kernel]          [k] ipv6_dev_get_saddr
 3.11%  [pppoe]           [k] pppoe_rcv
 2.40%  [kernel]          [k] rtnl_fill_ifinfo.constprop.0.isra.0
 1.85%  [kernel]          [k] inet6_fill_ifla6_attrs
 1.57%  [kernel]          [k] inet_dump_ifaddr
 1.44%  libc.so.6         [.] 0x0000000000137a10
 1.36%  libc.so.6         [.] 0x0000000000137cb8
 1.25%  [kernel]          [k] __dev_queue_xmit
 1.19%  libc.so.6         [.] 0x00000000001379f9
 1.18%  [kernel]          [k] nla_put
 1.17%  libc.so.6         [.] 0x0000000000137ca1
 1.01%  libc.so.6         [.] 0x0000000000137ca8
 0.99%  libc.so.6         [.] 0x0000000000137a00
 0.89%  [kernel]          [k] fib_table_lookup
 0.83%  [nf_tables]       [k] nft_do_chain
 0.82%  libc.so.6         [.] 0x0000000000137a09
 0.80%  [kernel]          [k] __netif_receive_skb_core.constprop.0
 0.67%  [kernel]          [k] memset_orig
 0.66%  libc.so.6         [.] 0x00000000001379f0
 0.65%  libc.so.6         [.] 0x0000000000137cb1
 0.64%  [kernel]          [k] memcmp
 0.64%  [kernel]          [k] strncpy
 0.64%  [kernel]          [k] csum_partial_copy_generic
 0.62%  [nf_tables]       [k] __nft_rbtree_lookup
 0.61%  [kernel]          [k] dev_queue_xmit_nit
 0.58%  libc.so.6         [.] 0x00000000001379fc
 0.56%  libc.so.6         [.] 0x0000000000137c98
 0.56%  [ixgbe]           [k] ixgbe_clean_rx_irq
 0.56%  [kernel]          [k] kmem_cache_free
 0.55%  [kernel]          [k] __copy_skb_header
 0.54%  libc.so.6         [.] 0x0000000000137aa8
 0.53%  [kernel]          [k] skb_put
 0.52%  [kernel]          [k] skb_clone
pymumu commented 1 year ago

There is no smartdns function in the log, are you sure there is no mistake?

micron10 commented 1 year ago

yes all is fine on system have two other dns server

  1. Bind
  2. knot-resolver.

both work ~ good with 1-2% load

i whan to change with smartdns for me look more better , fast and good .

if have options to debug and find what make this load

pymumu commented 1 year ago

The above logs are all functions of the kernel or libc, and it is impossible to analyze where the CPU is occupied by smartdns.

Is the CPU usage normal when there is no load?

micron10 commented 1 year ago

yes cpu is fine when no load :

this is perf when not run smartdns and top

PerfTop: 38013 irqs/sec kernel:97.5% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (all, 12 CPUs)

12.31%  [pppoe]         [k] pppoe_rcv
 3.62%  [kernel]        [k] __dev_queue_xmit
 2.66%  [nf_tables]     [k] nft_do_chain
 2.54%  [kernel]        [k] __netif_receive_skb_core.constprop.0
 2.53%  [kernel]        [k] fib_table_lookup
 2.49%  [kernel]        [k] memcmp
 2.41%  [ixgbe]         [k] ixgbe_clean_rx_irq
 2.30%  [nf_tables]     [k] __nft_rbtree_lookup
 1.82%  [kernel]        [k] strncpy
 1.65%  [kernel]        [k] csum_partial_copy_generic
 1.60%  [kernel]        [k] dev_queue_xmit_nit
 1.57%  [kernel]        [k] skb_clone
 1.55%  [kernel]        [k] kmem_cache_free
 1.48%  [kernel]        [k] __copy_skb_header
 1.31%  [kernel]        [k] __skb_flow_dissect
 1.28%  [kernel]        [k] skb_release_data
 1.21%  [nf_tables]     [k] nft_rhash_lookup
 1.15%  [ixgbe]         [k] ixgbe_tx_map
 1.09%  [nf_conntrack]  [k] __nf_conntrack_find_get.isra.0
 0.90%  [kernel]        [k] dev_hard_start_xmit
 0.86%  [kernel]        [k] kmem_cache_alloc
 0.85%  [kernel]        [k] __siphash_unaligned
 0.84%  [kernel]        [k] kfree_skb_reason
 0.81%  [vlan_mon]      [k] vlan_pt_recv
 0.80%  [kernel]        [k] ip_route_input_slow
 0.78%  [kernel]        [k] ip_finish_output2
 0.71%  [nf_tables]     [k] nft_lookup_eval
 0.70%  [kernel]        [k] nf_hook_slow
 0.68%  [kernel]        [k] _raw_spin_lock_irqsave
 0.66%  [ixgbe]         [k] ixgbe_xmit_frame_ring
 0.65%  [kernel]        [k] netif_skb_features
 0.60%  [kernel]        [k] vlan_do_receive
 0.59%  [kernel]        [k] ip_rcv_core.constprop.0
 0.58%  [kernel]        [k] dev_gro_receive
 0.58%  [kernel]        [k] menu_select
 0.58%  [kernel]        [k] ___slab_alloc.isra.0
 0.56%  [kernel]        [k] get_rps_cpu
 0.55%  [kernel]        [k] process_backlog
 0.55%  [nf_nat]        [k] nf_nat_inet_fn
 0.55%  [kernel]        [k] validate_xmit_skb.isra.0
 0.54%  [kernel]        [k] page_frag_free
 0.52%  [kernel]        [k] ip_forward
 0.52%  [kernel]        [k] __rcu_read_unlock
 0.52%  [kernel]        [k] sch_direct_xmit
 0.51%  [kernel]        [k] __local_bh_enable_ip

Tasks: 250 total, 1 running, 249 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.0 sy, 0.0 ni, 71.3 id, 0.0 wa, 0.0 hi, 28.0 si, 0.0 st %Cpu1 : 0.7 us, 0.7 sy, 0.0 ni, 71.6 id, 0.0 wa, 0.0 hi, 27.0 si, 0.0 st %Cpu2 : 0.7 us, 0.0 sy, 0.7 ni, 74.0 id, 0.0 wa, 0.0 hi, 24.7 si, 0.0 st %Cpu3 : 0.7 us, 0.7 sy, 0.0 ni, 72.5 id, 0.0 wa, 0.0 hi, 26.2 si, 0.0 st %Cpu4 : 1.4 us, 0.7 sy, 0.0 ni, 73.6 id, 0.0 wa, 0.0 hi, 24.3 si, 0.0 st %Cpu5 : 1.4 us, 0.0 sy, 0.0 ni, 72.3 id, 0.0 wa, 0.0 hi, 26.4 si, 0.0 st %Cpu6 : 0.7 us, 0.7 sy, 0.0 ni, 68.5 id, 0.0 wa, 0.0 hi, 30.1 si, 0.0 st %Cpu7 : 1.4 us, 0.7 sy, 0.0 ni, 74.7 id, 0.0 wa, 0.0 hi, 23.3 si, 0.0 st %Cpu8 : 0.7 us, 0.7 sy, 0.0 ni, 70.7 id, 0.0 wa, 0.0 hi, 27.9 si, 0.0 st %Cpu9 : 0.7 us, 0.0 sy, 0.0 ni, 69.6 id, 0.0 wa, 0.0 hi, 29.7 si, 0.0 st %Cpu10 : 1.4 us, 0.0 sy, 0.0 ni, 71.2 id, 0.0 wa, 0.0 hi, 27.4 si, 0.0 st %Cpu11 : 0.0 us, 0.0 sy, 0.0 ni, 74.5 id, 0.0 wa, 0.0 hi, 25.5 si, 0.0 st MiB Mem : 31955.6 total, 26487.0 free, 4621.1 used, 3905.0 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 27334.5 avail Mem

pymumu commented 1 year ago

try remove some upstream servers，keep one upstream server.

micron10 commented 1 year ago

try remove all only two stay 8.8.8.8 and 8.8.4.4

still same

micron10 commented 1 year ago

one note with 1 user all is fine may be when have to much users is not fine

pymumu commented 1 year ago

Very strange, if smartdns occupies the CPU, perf tool should be able to see which function of smartdns is occupying it.

The performance of smartdns has been tested, it is about 40000qps. If there are only 5000 users, and all the caches are hit, the CPU usage should be <10%

PikuZheng commented 1 year ago

Only 5 processes when using top query? Are you using docker? What is the operating system? Are you using a precompiled version of release43 for the first time? In addition to the high CPU usage, are the functions normal in actual use?

micron10 commented 1 year ago

Hi yes in top is show only 5 process, machine have 6 core + 6HT Debian base os with cudtom kernel. Yes first time use and compile Smartdns.

I think yes dns function normal

micron10 commented 1 year ago

Hi try and get binary from here same stay on 100%

and one more i try on time when run service to make dig :

;; communications error to 127.0.0.1#53: timed out ;; communications error to 127.0.0.1#53: timed out ;; communications error to 127.0.0.1#53: connection refused

i think and not work

micron10 commented 1 year ago

perf top -p PID after remove kernel item in this moment smartdns is on 100%

PerfTop: 4454 irqs/sec kernel:94.4% exact: 5.6% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 2905711)

 3.18%  smartdns          [.] 0x0000000000215b80
 2.47%  smartdns          [.] 0x000000000021db90
 2.37%  smartdns          [.] 0x000000000021dc6a
 1.56%  smartdns          [.] 0x000000000021db98
 1.40%  smartdns          [.] 0x00000000000176a8
 1.39%  smartdns          [.] 0x000000000021abac
 1.00%  smartdns          [.] 0x0000000000215beb
 0.96%  smartdns          [.] 0x000000000021db9d
 0.92%  smartdns          [.] 0x0000000000212ff2
 0.89%  smartdns          [.] 0x0000000000215b89
 0.84%  smartdns          [.] 0x0000000000212e5f
 0.77%  smartdns          [.] 0x000000000021b839
 0.75%  smartdns          [.] 0x0000000000215bde
 0.69%  smartdns          [.] 0x0000000000215b4d
 0.58%  smartdns          [.] 0x000000000021308f
 0.57%  smartdns          [.] 0x0000000000213997
 0.54%  smartdns          [.] 0x000000000021db73
 0.52%  smartdns          [.] 0x0000000000215d18
 0.52%  smartdns          [.] 0x000000000001e38c
 0.51%  smartdns          [.] 0x000000000021ddcb
 0.48%  smartdns          [.] 0x0000000000215b47
 0.47%  smartdns          [.] 0x0000000000212e3b
 0.47%  smartdns          [.] 0x0000000000215b44
 0.45%  smartdns          [.] 0x000000000021c363
 0.43%  smartdns          [.] 0x0000000000215ce7
 0.42%  smartdns          [.] 0x000000000021353f
 0.41%  smartdns          [.] 0x000000000021f55a
 0.39%  smartdns          [.] 0x00000000002139b2
 0.39%  smartdns          [.] 0x0000000000215bfa
 0.36%  smartdns          [.] 0x000000000021b852
 0.35%  smartdns          [.] 0x000000000021dd83
 0.35%  smartdns          [.] 0x000000000021db7d
 0.35%  smartdns          [.] 0x00000000000176b5
 0.34%  smartdns          [.] 0x000000000021bb08
 0.34%  smartdns          [.] 0x000000000021dcdc
 0.33%  smartdns          [.] 0x000000000021f087
 0.33%  smartdns          [.] 0x000000000021a388
 0.30%  smartdns          [.] 0x000000000021f09a
 0.29%  smartdns          [.] 0x0000000000213b44
 0.29%  smartdns          [.] 0x0000000000215f25
 0.28%  smartdns          [.] 0x0000000000212a77
 0.25%  smartdns          [.] 0x000000000001e51b
 0.25%  smartdns          [.] 0x000000000001e519
 0.25%  smartdns          [.] 0x0000000000212a83
 0.24%  smartdns          [.] 0x000000000021b888
 0.24%  smartdns          [.] 0x000000000021b896
 0.23%  smartdns          [.] 0x0000000000009914
 0.23%  smartdns          [.] 0x000000000021c158
 0.23%  smartdns          [.] 0x000000000002f4a0
 0.22%  smartdns          [.] 0x000000000021a313
 0.22%  [vdso]            [.] 0x00000000000006e5
 0.22%  smartdns          [.] 0x000000000021b8ff
 0.22%  smartdns          [.] 0x000000000021bb36
 0.21%  smartdns          [.] 0x000000000021a38d
 0.21%  smartdns          [.] 0x0000000000212efe
 0.21%  smartdns          [.] 0x0000000000215c15
 0.21%  smartdns          [.] 0x000000000021a30e
 0.20%  smartdns          [.] 0x0000000000215b60
 0.20%  smartdns          [.] 0x000000000021aba9
 0.20%  smartdns          [.] 0x000000000000bcd0
 0.19%  smartdns          [.] 0x000000000021a331
 0.19%  smartdns          [.] 0x00000000000178c5
 0.19%  smartdns          [.] 0x000000000021abf5
 0.18%  smartdns          [.] 0x000000000021b826
 0.18%  smartdns          [.] 0x000000000021fc1b
 0.18%  smartdns          [.] 0x000000000021abf8
 0.18%  smartdns          [.] 0x000000000021a385
 0.18%  smartdns          [.] 0x000000000001e217
 0.18%  smartdns          [.] 0x000000000021b834
 0.18%  [vdso]            [.] 0x00000000000006d5
 0.18%  smartdns          [.] 0x000000000021b7e7
 0.17%  smartdns          [.] 0x000000000021a3bc
 0.17%  smartdns          [.] 0x000000000021db89
 0.17%  smartdns          [.] 0x000000000021bdbe
 0.17%  smartdns          [.] 0x0000000000213c48
pymumu commented 1 year ago

run perf top with debug version of smartdns

micron10 commented 1 year ago

this get :

PerfTop: 4266 irqs/sec kernel:89.9% exact: 10.1% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 2962595)

 7.25%  libc.so.6         [.] 0x0000000000137cb8
 6.70%  libc.so.6         [.] 0x0000000000137a10
 5.96%  libc.so.6         [.] 0x00000000001379f9
 5.62%  libc.so.6         [.] 0x0000000000137ca1
 5.14%  libc.so.6         [.] 0x0000000000137ca8
 4.83%  libc.so.6         [.] 0x0000000000137a00
 4.43%  libc.so.6         [.] 0x0000000000137a09
 4.22%  libc.so.6         [.] 0x0000000000137cb1
 3.33%  libc.so.6         [.] 0x00000000001379f0
 3.04%  libc.so.6         [.] 0x0000000000137c98
 2.80%  libc.so.6         [.] 0x00000000001379fc
 2.74%  libc.so.6         [.] 0x0000000000137aa8
 2.50%  libc.so.6         [.] 0x0000000000137ca4
 1.76%  libc.so.6         [.] 0x00000000001377da
 1.31%  libc.so.6         [.] 0x000000000016d5ca
 0.70%  libc.so.6         [.] 0x0000000000137cdf
 0.61%  smartdns          [.] 0x0000000000015492
 0.54%  libc.so.6         [.] 0x0000000000137755
 0.53%  libc.so.6         [.] 0x00000000000a1952
 0.39%  libc.so.6         [.] 0x00000000000a0039
 0.39%  libc.so.6         [.] 0x000000000016cca0
 0.31%  libc.so.6         [.] 0x000000000016cca9
 0.30%  libc.so.6         [.] 0x0000000000137acf
 0.29%  libc.so.6         [.] 0x00000000001377bf
 0.23%  libc.so.6         [.] 0x000000000016cca4
 0.23%  libc.so.6         [.] 0x000000000016cb91
 0.22%  libc.so.6         [.] 0x000000000008f051
 0.21%  libc.so.6         [.] 0x0000000000137a93
 0.20%  libc.so.6         [.] 0x000000000016ccbb
 0.20%  libc.so.6         [.] 0x00000000001377d1
 0.18%  libc.so.6         [.] 0x000000000016ccb7
 0.18%  libc.so.6         [.] 0x0000000000137a9e
 0.18%  libc.so.6         [.] 0x000000000016ccc0
 0.17%  libc.so.6         [.] 0x000000000016ccae
 0.17%  libc.so.6         [.] 0x00000000000a2c16
 0.17%  libc.so.6         [.] 0x0000000000137ada
 0.16%  libc.so.6         [.] 0x000000000016ccc5
 0.16%  libc.so.6         [.] 0x000000000016cb09
 0.15%  libc.so.6         [.] 0x00000000001377ce
 0.15%  libc.so.6         [.] 0x00000000001374ba
 0.15%  libc.so.6         [.] 0x000000000009798a
 0.15%  libc.so.6         [.] 0x000000000016cb07
 0.14%  libc.so.6         [.] 0x0000000000095fd0
 0.14%  libc.so.6         [.] 0x00000000001377b4
 0.14%  libc.so.6         [.] 0x000000000008f0c0
 0.13%  libc.so.6         [.] 0x0000000000058949
 0.13%  libc.so.6         [.] 0x000000000005895a
 0.13%  libc.so.6         [.] 0x00000000000a2bf0
 0.13%  smartdns          [.] 0x0000000000021d3d
 0.12%  libc.so.6         [.] 0x0000000000137a9b
 0.12%  smartdns          [.] 0x00000000000320bd
 0.11%  libc.so.6         [.] 0x00000000001377d4
 0.11%  libc.so.6         [.] 0x0000000000137a98
 0.11%  libc.so.6         [.] 0x0000000000137ae1
 0.11%  smartdns          [.] 0x0000000000021d38
 0.11%  smartdns          [.] 0x000000000001548a
 0.10%  smartdns          [.] 0x0000000000031621
 0.10%  libc.so.6         [.] 0x0000000000137ab9
 0.10%  libc.so.6         [.] 0x0000000000096018
 0.10%  libc.so.6         [.] 0x00000000001377c8
 0.10%  libc.so.6         [.] 0x00000000000a0cdc
 0.09%  libc.so.6         [.] 0x000000000016cb80
 0.09%  smartdns          [.] 0x0000000000005650
 0.09%  smartdns          [.] 0x0000000000031640
 0.09%  [vdso]            [.] 0x00000000000006a0
 0.09%  libc.so.6         [.] 0x000000000008f0c9
 0.09%  libc.so.6         [.] 0x000000000016f047
 0.09%  libc.so.6         [.] 0x0000000000026290
 0.09%  libc.so.6         [.] 0x000000000016cbd6
 0.08%  smartdns          [.] 0x0000000000041130
 0.08%  libc.so.6         [.] 0x0000000000137718
 0.08%  libc.so.6         [.] 0x000000000005a3ca
 0.08%  libc.so.6         [.] 0x000000000016ee5f
 0.08%  libc.so.6         [.] 0x00000000000984c0
 0.08%  libc.so.6         [.] 0x000000000008f0dd
 0.08%  smartdns          [.] 0x000000000000fd8f
 0.08%  libc.so.6         [.] 0x000000000005a32b
 0.08%  libc.so.6         [.] 0x0000000000137aac
micron10 commented 1 year ago

see this is knot-resolver :

PerfTop: 160 irqs/sec kernel:63.1% exact: 36.9% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 2966215)

 2.01%  liblmdb.so            [.] 0x00000000000037e5
 1.46%  liblmdb.so            [.] 0x000000000000700e
 0.71%  libc.so.6             [.] 0x000000000016c695
 0.51%  libkres.so.9          [.] 0x000000000002242c
 0.50%  libuv.so.1.0.0        [.] 0x00000000000090f0
 0.38%  liblmdb.so            [.] 0x0000000000003eb4
 0.32%  libuv.so.1.0.0        [.] 0x000000000000c240
 0.32%  kresd                 [.] 0x0000000000028570
 0.30%  libkres.so.9          [.] 0x00000000000223f3
 0.29%  libc.so.6             [.] 0x000000000016cd25
 0.28%  liblmdb.so            [.] 0x0000000000003ee6
 0.27%  [JIT] tid 2966215     [.] 0x00007f4992a4f0cc
 0.26%  liblmdb.so            [.] 0x0000000000003ed1
 0.25%  libluajit.so          [.] 0x000000000001c061
 0.25%  libc.so.6             [.] 0x0000000000027f70
 0.25%  libjemalloc.so.2      [.] 0x000000000001d5a6
 0.25%  libuv.so.1.0.0        [.] 0x000000000002be7b
 0.24%  libhogweed.so         [.] 0x000000000001ccf7
 0.24%  libjemalloc.so.2      [.] 0x000000000002094e
 0.23%  libgmp.so.10.5.0      [.] 0x0000000000068c39
 0.23%  libhogweed.so         [.] 0x000000000001ccb5
 0.23%  libluajit.so          [.] 0x000000000001bfd0
 0.23%  libknot.so.13.0.0     [.] 0x000000000000dfb5
 0.23%  libc.so.6             [.] 0x0000000000095fd0
 0.23%  libc.so.6             [.] 0x000000000016c69d
 0.22%  libluajit.so          [.] 0x000000000000a68d
 0.22%  libkres.so.9          [.] 0x0000000000025f3c
 0.22%  libgnutls.so.30.34.2  [.] 0x000000000004b700
 0.21%  libgmp.so.10.5.0      [.] 0x000000000006b6a7
 0.21%  libgnutls.so.30.34.2  [.] 0x00000000001340f0
 0.20%  libluajit.so          [.] 0x0000000000012093
 0.20%  liblmdb.so            [.] 0x0000000000003ef0
 0.20%  [JIT] tid 2966215     [.] 0x00007f4992a4ef78
 0.20%  kresd                 [.] 0x000000000001a3b6
 0.20%  libgnutls.so.30.34.2  [.] 0x00000000001340e8
 0.19%  libkres.so.9          [.] 0x0000000000007d40
 0.19%  libkres.so.9          [.] 0x000000000002345c
 0.18%  libc.so.6             [.] 0x00000000000de6e7
 0.18%  libkres.so.9          [.] 0x00000000000235f8
 0.18%  libgmp.so.10.5.0      [.] 0x000000000006710b
 0.18%  libgmp.so.10.5.0      [.] 0x000000000006b6a0
 0.18%  libjemalloc.so.2      [.] 0x00000000000240fa
 0.18%  libkres.so.9          [.] 0x00000000000247ba
 0.18%  libuv.so.1.0.0        [.] 0x00000000000137fe
 0.18%  kresd                 [.] 0x0000000000009b10
 0.18%  libknot.so.13.0.0     [.] 0x0000000000009d80
 0.18%  libjemalloc.so.2      [.] 0x000000000001e4c7
 0.18%  libhogweed.so         [.] 0x000000000000a6f0
 0.17%  libkres.so.9          [.] 0x000000000001a16b
 0.17%  stats.so              [.] 0x0000000000002250
 0.17%  libjemalloc.so.2      [.] 0x000000000001e4f0
 0.17%  libhogweed.so         [.] 0x0000000000015574
 0.17%  libc.so.6             [.] 0x000000000016d5ca
 0.17%  libc.so.6             [.] 0x0000000000044a20
 0.17%  libkres.so.9          [.] 0x0000000000013e10
 0.17%  libgnutls.so.30.34.2  [.] 0x000000000013e0a0
 0.17%  libkres.so.9          [.] 0x000000000001a2cf
 0.17%  kresd                 [.] 0x0000000000028062
 0.17%  liblmdb.so            [.] 0x0000000000003220

Threads: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.0 sy, 0.0 ni, 51.4 id, 0.0 wa, 0.0 hi, 47.8 si, 0.0 st %Cpu1 : 2.0 us, 1.3 sy, 0.0 ni, 59.1 id, 0.0 wa, 0.0 hi, 37.6 si, 0.0 st %Cpu2 : 1.4 us, 0.7 sy, 0.0 ni, 63.3 id, 0.0 wa, 0.0 hi, 34.7 si, 0.0 st %Cpu3 : 0.7 us, 0.7 sy, 0.0 ni, 62.2 id, 0.0 wa, 0.0 hi, 36.5 si, 0.0 st %Cpu4 : 1.4 us, 0.7 sy, 0.0 ni, 59.6 id, 0.0 wa, 0.0 hi, 38.4 si, 0.0 st %Cpu5 : 0.7 us, 0.7 sy, 0.0 ni, 59.9 id, 0.0 wa, 0.0 hi, 38.8 si, 0.0 st %Cpu6 : 1.4 us, 0.0 sy, 0.0 ni, 53.4 id, 0.0 wa, 0.0 hi, 45.3 si, 0.0 st %Cpu7 : 1.3 us, 0.7 sy, 0.0 ni, 53.0 id, 0.0 wa, 0.0 hi, 45.0 si, 0.0 st %Cpu8 : 1.4 us, 0.7 sy, 0.0 ni, 51.0 id, 0.0 wa, 0.0 hi, 46.9 si, 0.0 st %Cpu9 : 0.7 us, 1.4 sy, 0.0 ni, 58.1 id, 0.0 wa, 0.0 hi, 39.9 si, 0.0 st %Cpu10 : 0.0 us, 0.7 sy, 0.7 ni, 57.8 id, 0.0 wa, 0.0 hi, 40.8 si, 0.0 st %Cpu11 : 0.7 us, 0.0 sy, 0.0 ni, 54.9 id, 0.0 wa, 0.0 hi, 44.4 si, 0.0 st MiB Mem : 31955.6 total, 25799.1 free, 4671.3 used, 4546.1 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 27284.2 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

2966215 root 20 0 2364.7m 48.5m 4.0 0.2 0:42.58 S /usr/bin/kresd -c /etc/knot-resolver/kresd.conf --forks=6 2966225 root 20 0 2364.7m 48.5m 0.0 0.2 0:00.00 S /usr/bin/kresd -c /etc/knot-resolver/kresd.conf --forks=6

micron10 commented 1 year ago

Hi @pymumu is there options to add more debugs to find from where is come this load

i make some test and service work fine with 1 to 100 users after that retest and try to find from where is problem but not success.

PikuZheng commented 1 year ago

I suspect it has some problems with the compiler or the runtime environment, which is why I recommend using a precompiled (static) version. but 127.0.0.1？any proxy between smartdns and end users?

micron10 commented 1 year ago

Hi @PikuZheng

as i write , try with download static file from here but is same

PikuZheng commented 1 year ago

Hi @PikuZheng

as i write , try with download static file from here but is same

sorry to hear that :(

pymumu commented 1 year ago

when running perf top, there should be the following output, and you can see the function names of the process.

Samples: 91  of event 'cpu-clock', 4000 Hz, Event count (approx.): 1036339
Overhead  Shared Object       Symbol
  20.09%  libpthread-2.28.so  [.] __pthread_mutex_lock
  12.01%  smartdns            [.] dns_cache_invalidate
  10.84%  [kernel]            [k] nft_do_chain
  10.83%  [kernel]            [k] do_epoll_wait
  10.83%  libc-2.28.so        [.] __strnlen_avx2
   5.55%  smartdns            [.] _dns_server_process_cache_packet
   5.00%  [kernel]            [k] wait_for_stable_page
   4.87%  libc-2.28.so        [.] epoll_wait
   1.91%  [kernel]            [k] __fget
   1.91%  [kernel]            [k] __virt_addr_valid
   1.91%  libc-2.28.so        [.] __memset_avx2_unaligned_erms
   1.91%  smartdns            [.] _dns_encode_domain
   1.58%  [kernel]            [k] finish_task_switch
   1.28%  [kernel]            [k] sock_poll
   1.28%  [vdso]              [.] time
   1.00%  smartdns            [.] dns_server_run
   0.86%  [kernel]            [k] aa_sk_perm
   0.70%  [kernel]            [k] ep_poll
   0.67%  libc-2.28.so        [.] _IO_vfprintf
   0.67%  libc-2.28.so        [.] calloc
   0.66%  smartdns            [.] _dns_get_domain_offset
   0.66%  [kernel]            [k] nf_conntrack_tuple_taken
   0.66%  libc-2.28.so        [.] clock_gettime
   0.66%  smartdns            [.] New_Prefix2
   0.38%  libc-2.28.so        [.] __libc_disable_asynccancel
   0.34%  libpthread-2.28.so  [.] pthread_cond_signal@@GLIBC_2.3.2
   0.20%  [kernel]            [k] crc32c_pcl_intel_update
   0.20%  libpthread-2.28.so  [.] __errno_location
   0.10%  [kernel]            [k] netif_skb_features
   0.07%  [kernel]            [k] _raw_spin_unlock_irqrestore
   0.06%  smartdns            [.] _dns_client_work
   0.05%  smartdns            [.] _fast_ping_work

Only in this way I can locate where the CPU is being used.

In your log, there are only addresses, which cannot locate the problem. The possible reason is

  1. The version of perf is too low.
  2. smartdns does not use the debug version.

or you can try callgrind

valgrind --tool=callgrind ./smartdns

by the way, you can run dnsperf to verify that the performance is ok or not.

make -C test test.bin -j8
test/test.bin --gtest_filter=Perf.*
micron10 commented 1 year ago

Hi @pymumu

is there options to build one debug version in your system and to test that show all function name. if show to send this binary to test here

pymumu commented 1 year ago

smartdns-x86_64.zip

micron10 commented 1 year ago

Huh

ok i try

in gdb not found debug :

0 0x000000000062a39c in epoll_pwait ()

No symbol table info available.

1 0x0000000000c2a09c in ?? ()

No symbol table info available.

2 0x00000000004289f5 in dns_server_run ()

No symbol table info available.

3 0x00000000004037d7 in _smartdns_run ()

No symbol table info available.

4 0x0000000000404149 in main ()

No symbol table info available.

in perf top -p PID same :

PerfTop: 1 irqs/sec kernel:100.0% exact: 0.0% lost: 0/0 drop: 2/8 [4000Hz cycles], (target_pid: 112410)

 5.78%  smartdns          [.] 0x0000000000235bca
 5.09%  smartdns          [.] 0x00000000000597ee
 4.48%  smartdns          [.] 0x000000000022bad8
 3.73%  smartdns          [.] 0x00000000001e4770
 3.64%  smartdns          [.] 0x000000000014d0b2
 3.61%  smartdns          [.] 0x0000000000010467
 3.61%  smartdns          [.] 0x00000000000ae25e
 3.53%  smartdns          [.] 0x000000000016b788
 3.51%  [vdso]            [.] 0x00000000000006b3
 3.46%  smartdns          [.] 0x00000000001ff817
 3.42%  smartdns          [.] 0x0000000000235390
 3.40%  smartdns          [.] 0x00000000000154d8
 3.28%  smartdns          [.] 0x00000000001ff1d0
 3.20%  smartdns          [.] 0x000000000023352a
 3.16%  smartdns          [.] 0x0000000000042a4c
 3.15%  smartdns          [.] 0x0000000000009bd9
 3.14%  smartdns          [.] 0x000000000003032a
 2.88%  smartdns          [.] 0x0000000000235af9
 2.87%  smartdns          [.] 0x00000000001e61d6
 2.87%  smartdns          [.] 0x0000000000133d14
 2.84%  smartdns          [.] 0x0000000000035617
 2.71%  smartdns          [.] 0x00000000001e4f89
 2.55%  smartdns          [.] 0x000000000000d4e2
 2.43%  smartdns          [.] 0x0000000000117363
 2.38%  smartdns          [.] 0x00000000001e7962
 2.35%  smartdns          [.] 0x0000000000079fd1
 2.02%  smartdns          [.] 0x00000000001aa796
 1.97%  smartdns          [.] 0x00000000000dbe61
 1.75%  smartdns          [.] 0x000000000001aa1e
 1.66%  smartdns          [.] 0x000000000008e596
 1.10%  smartdns          [.] 0x0000000000234b82
 0.95%  smartdns          [.] 0x000000000001a9f0
 0.88%  smartdns          [.] 0x00000000002361fd
 0.77%  smartdns          [.] 0x000000000002aeff
 0.37%  smartdns          [.] 0x0000000000235b42
 0.36%  smartdns          [.] 0x00000000002361f0
 0.35%  smartdns          [.] 0x000000000002a6d0
 0.35%  smartdns          [.] 0x0000000000233797
 0.14%  smartdns          [.] 0x000000000001606e
 0.12%  smartdns          [.] 0x00000000002374be
 0.06%  smartdns          [.] 0x0000000000015d77
 0.04%  smartdns          [.] 0x00000000000090cd
 0.04%  smartdns          [.] 0x0000000000004358

with other process where have debug enable i see all function

in kernel is need to enable options to show this ? or other options to not display function name ?"

micron10 commented 1 year ago

Hi @pymumu @PikuZheng

first I want to apologize to you. problem is in my perf tool not build with libelf and not show function name ....

After see function name i think know where is a problem :

netlink_msg_to_ifaddr i think problem is when smartdns run and have pppoe interface make this problem. i miss to write on this machine this 5000 users is over pppoe interface and this make problem .

is there options to fix this . i think smartdns try to run on all ppp interfacess..

PerfTop: 4285 irqs/sec kernel:95.0% exact: 5.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 313135)

11.77%  smartdns       [.] netlink_msg_to_ifaddr
 8.30%  smartdns       [.] printf_core
 5.25%  smartdns       [.] memcpy
 3.39%  smartdns       [.] _dns_get_domain_from_packet
 3.32%  smartdns       [.] memset
 2.89%  smartdns       [.] _dns_server_process_local_ptr
 2.79%  smartdns       [.] __fwritex
 2.58%  smartdns       [.] __libc_free
 2.38%  smartdns       [.] enframe
 2.32%  smartdns       [.] alloc_slot
 2.19%  smartdns       [.] pad
 2.17%  smartdns       [.] fmt_u
 1.91%  smartdns       [.] dict_hash
 1.56%  smartdns       [.] get_meta
 1.24%  smartdns       [.] out
 1.16%  smartdns       [.] __libc_malloc_impl
 0.87%  smartdns       [.] _dns_read_short
 0.84%  smartdns       [.] _dns_left_len
 0.83%  smartdns       [.] vfprintf
 0.75%  smartdns       [.] _dns_encode_domain
 0.73%  smartdns       [.] sn_write
 0.72%  [vdso]         [.] __vdso_clock_gettime
 0.72%  smartdns       [.] size_to_class
 0.63%  smartdns       [.] _dns_get_domain_offset
 0.60%  smartdns       [.] __stpncpy
 0.60%  smartdns       [.] sccp
 0.60%  smartdns       [.] strncmp
 0.59%  smartdns       [.] _dns_server_process_answer_A
 0.56%  smartdns       [.] __malloc_allzerop
 0.55%  smartdns       [.] _dns_write_short
 0.54%  smartdns       [.] __pthread_mutex_lock
 0.54%  smartdns       [.] _dns_server_process_answer
 0.50%  smartdns       [.] vsnprintf
 0.48%  smartdns       [.] _dns_decode_an
 0.47%  smartdns       [.] pop_arg
 0.46%  smartdns       [.] tlog_vext
 0.44%  smartdns       [.] jhash
 0.43%  smartdns       [.] __clock_gettime
 0.42%  smartdns       [.] _dns_client_send_packet
 0.41%  smartdns       [.] _dns_rr_add_end
 0.40%  smartdns       [.] freeifaddrs
 0.38%  smartdns       [.] __lock
 0.36%  smartdns       [.] _dns_client_recv
 0.35%  smartdns       [.] tlog_ext
 0.32%  smartdns       [.] calloc
 0.32%  smartdns       [.] do_ssl3_write
 0.31%  smartdns       [.] _dns_ip_address_check_add
 0.30%  smartdns       [.] jhash
 0.30%  smartdns       [.] dns_cache_lookup
 0.29%  smartdns       [.] _dns_decode_body
 0.28%  smartdns       [.] __netlink_enumerate
 0.28%  smartdns       [.] copy_addr
 0.28%  smartdns       [.] jhash
 0.28%  smartdns       [.] strtox
 0.28%  smartdns       [.] rol32
 0.27%  smartdns       [.] _dns_client_work
 0.27%  smartdns       [.] nontrivial_free
 0.26%  smartdns       [.] _fast_ping_process_icmp
 0.26%  smartdns       [.] _dns_init_context_by_rrs
 0.26%  smartdns       [.] __unlock
 0.25%  smartdns       [.] _dns_client_process_udp
 0.25%  smartdns       [.] _dns_decode_raw
 0.23%  smartdns       [.] getint
 0.23%  smartdns       [.] ssl3_get_record
 0.23%  smartdns       [.] _dns_server_reply_passthrough
 0.22%  smartdns       [.] __pthread_mutex_unlock
 0.22%  smartdns       [.] dns_server_resolve_callback
 0.22%  smartdns       [.] _dns_decode_head
 0.20%  smartdns       [.] __strerror_l
 0.20%  smartdns       [.] ssl3_clear
 0.20%  smartdns       [.] aes_gcm_ctrl
 0.19%  smartdns       [.] ssl3_read_bytes
 0.19%  smartdns       [.] hash_string_initval
 0.19%  smartdns       [.] EVP_DecryptUpdate
 0.19%  smartdns       [.] memchr
 0.19%  smartdns       [.] CRYPTO_gcm128_decrypt_ctr32
 0.19%  smartdns       [.] setsockopt
 0.19%  smartdns       [.] _dns_add_qr_head
PikuZheng commented 1 year ago

Are you meaning that the device run as a pppoe server and smartdns needs to handle 5000 interfaces? I never seen this situation. Perhaps you could bind it to one interface and use iptables to forward all ppp requests to this interface

micron10 commented 1 year ago

how isc bind and knot-resolver make this ?

yes on machine have pppoe server that handle 5-6k users (pppXXXX ) interfaces

is it possible to add in code if interface is pppoe or ipoe to not server ?

PikuZheng commented 1 year ago

I'm guessing they have different processing method. For example, unbound only binds the physical interface corresponding to the pppoe interface, and ignores the connection and disconnection of pppoe. I'm not sure about smartdns, but maybe the problem handles when the number of pppoe interfaces changes?

micron10 commented 1 year ago

Same problem i see in snmpd and after team add options : include_ifmib_iface_prefix eth vlan lo bond team with this options enable to see info for only this device, if not set and have 5k pppoe or ipoe interface daemon stay same on 100% load or more.... idea is ppp and ipoe interface is virtual and no need to binds smartdns , only on this type need to bind : eth vlan lo bond team or to have options to set .

P.S. in time when run smartdns for test users stay on 5200-5250 no disconnect or connect (may be 1-2 users connect disconnect).

P.S.2 this is knot-resolver :

-- Network interface configuration net.listen('0.0.0.0', 53, { kind = 'dns' }) net.listen('0.0.0.0', 853, { kind = 'tls' })

i think bind same use default not binds to all interfacess

micron10 commented 1 year ago

Edit:

after set in config :

bind [::]:53@bond0.0101 -nftset bind [::]:53@lo -nftset

all is fine but need to set specific interface where is run IP + Default GW.

If you have time to add options if device is pppoe or ipoe to not list i think will fix all problems...

One more now make test

set in conf file ip address on uplink interface but is same :

bind 103.21.32.1:53 -nftset

server 8.8.8.8 server-tls 8.8.8.8 server 8.8.4.4 server-tls 8.8.4.4 server 1.1.1.1 server-tls 1.1.1.1 server 1.0.0.1 server-tls 1.0.0.1 server 9.9.9.9 server 149.112.112.112 server 2620:fe::fe server 2620:fe::9 server 2001:4860:4860::8888 server 2001:4860:4860::8844 server 2606:4700::1111 server-tls 2606:4700::1111 server 2606:4700:4700::1001 server-tls 2606:4700:4700::1001

cache-size 1000000 cache-persist yes cache-file /var/run/smartdns.cache cache-checkpoint-time 86400 prefetch-domain yes serve-expired yes serve-expired-ttl 259200 serve-expired-prefetch-time 21600

tcp-idle-time 120

force-qtype-SOA 65

Enable IPV4, IPV6 dual stack IP optimization selection strategy

dualstack-ip-selection-threshold [num] (0~1000)

dualstack-ip-allow-force-AAAA [yes|no]

dualstack-ip-selection yes

set log level

log-level: [level], level=off, fatal, error, warn, notice, info, debug

log-file: file path of log file.

log-console [yes|no]: output log to console.

log-size: size of each log file, support k,m,g

log-num: number of logs, 0 means disable log

log-level error

log-file /var/log/smartdns/smartdns.log log-size 256k log-num 2

enalbe DNS64 feature

dns64 [ip/subnet]

dns64 64:ff9b::/96

PerfTop: 4341 irqs/sec kernel:94.8% exact: 5.2% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 393012)

11.43%  smartdns       [.] netlink_msg_to_ifaddr
 7.35%  smartdns       [.] printf_core
 5.24%  smartdns       [.] memcpy
 3.77%  smartdns       [.] _dns_get_domain_from_packet
 3.32%  smartdns       [.] __fwritex
 2.98%  smartdns       [.] memset
 2.53%  smartdns       [.] __libc_free
 2.11%  smartdns       [.] alloc_slot
 2.06%  smartdns       [.] _dns_server_process_local_ptr
 1.98%  smartdns       [.] fmt_u
 1.96%  smartdns       [.] dict_hash
 1.71%  smartdns       [.] enframe
 1.61%  smartdns       [.] pad
 1.53%  smartdns       [.] get_meta
 1.24%  smartdns       [.] vfprintf
 1.10%  smartdns       [.] _dns_encode_domain
 1.00%  smartdns       [.] _dns_left_len
 0.94%  smartdns       [.] _dns_read_short
 0.92%  smartdns       [.] __libc_malloc_impl
 0.89%  smartdns       [.] __malloc_allzerop
 0.84%  smartdns       [.] jhash
 0.72%  smartdns       [.] _dns_decode_an
 0.70%  smartdns       [.] vsnprintf
 0.64%  smartdns       [.] _dns_get_domain_offset
 0.64%  smartdns       [.] _dns_client_send_packet
 0.62%  [vdso]         [.] __vdso_clock_gettime
 0.57%  smartdns       [.] strncmp
 0.55%  smartdns       [.] out
 0.51%  smartdns       [.] __pthread_mutex_unlock
 0.51%  smartdns       [.] tlog_vext
 0.49%  smartdns       [.] freeifaddrs
 0.46%  smartdns       [.] pop_arg
 0.45%  smartdns       [.] getint
 0.44%  smartdns       [.] _dns_decode_body
 0.44%  smartdns       [.] size_to_class
 0.44%  smartdns       [.] _dns_client_recv
 0.44%  smartdns       [.] __netlink_enumerate
 0.43%  smartdns       [.] _dns_server_process_answer
 0.42%  smartdns       [.] _dns_decode_qr_head
 0.38%  smartdns       [.] sn_write
 0.37%  smartdns       [.] _dns_write_short
 0.36%  smartdns       [.] sccp
 0.36%  smartdns       [.] __pthread_rwlock_trywrlock
 0.36%  smartdns       [.] memchr
 0.34%  smartdns       [.] _dns_init_context_by_rrs
 0.34%  smartdns       [.] __stpncpy
 0.33%  smartdns       [.] _fast_ping_period_run
 0.32%  smartdns       [.] __pthread_mutex_lock
 0.32%  smartdns       [.] _dns_server_process_answer_A
 0.30%  smartdns       [.] _dns_get_RAW
 0.30%  smartdns       [.] jhash
 0.29%  smartdns       [.] ntohs
 0.29%  smartdns       [.] dns_get_rrs_start
 0.29%  smartdns       [.] tlog_ext
 0.29%  smartdns       [.] snprintf
 0.28%  smartdns       [.] _dns_client_create_socket_udp
 0.28%  smartdns       [.] _dns_rr_add_end
 0.27%  smartdns       [.] __clock_gettime
 0.27%  smartdns       [.] do_ssl3_write
 0.27%  smartdns       [.] rol32
 0.27%  smartdns       [.] _dns_server_get_conf_ttl
 0.26%  smartdns       [.] __syscall_ret
 0.23%  smartdns       [.] tls13_enc
 0.22%  smartdns       [.] _dns_client_query_release
 0.22%  smartdns       [.] dns_server_resolve_callback
 0.22%  smartdns       [.] calloc
 0.22%  smartdns       [.] get_stride
 0.22%  smartdns       [.] _dns_read_int
 0.22%  smartdns       [.] BIO_read
 0.22%  smartdns       [.] _dns_server_process_cache_packet
 0.22%  smartdns       [.] dns_cache_lookup
 0.22%  smartdns       [.] copy_addr
 0.21%  smartdns       [.] _dns_get_qr_head
 0.21%  smartdns       [.] jhash
 0.21%  smartdns       [.] __intscan
 0.21%  smartdns       [.] aes_gcm_cipher
 0.21%  smartdns       [.] gcm_ghash_avx
 0.20%  smartdns       [.] _dns_add_RAW
PikuZheng commented 1 year ago

could you try to bind interface like bind [::]:53@eth0.1

micron10 commented 1 year ago

but i think after i bind to :

bind [::]:53@bond0.0101 -nftset bind [::]:53@lo -nftset

users that is on pppoe and ipoe not resolv like not see dns .

is there options to debug why service is load

pymumu commented 1 year ago

https://github.com/pymumu/smartdns/blob/887ef7b20e416909ed31ee8b37dad23c7cf197d5/src/dns_server.c#L3818-L3826

add line 3825:

static int _dns_server_process_local_ptr(struct dns_request *request)
{
    struct ifaddrs *ifaddr = NULL;
    struct ifaddrs *ifa = NULL;
    unsigned char *addr = NULL;
    char reverse_addr[128] = {0};
    int found = 0;

>       return -1; 

    if (getifaddrs(&ifaddr) == -1) {
micron10 commented 1 year ago

after add see down is info

try to make dig local from server dig cisco.com 127.0.0.1

;; communications error to 127.0.0.1#53: timed out

and in config set : bind [::]:53 -nftset

Threads: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.7 us, 0.7 sy, 0.0 ni, 33.1 id, 0.0 wa, 0.0 hi, 65.4 si, 0.0 st %Cpu1 : 1.3 us, 1.3 sy, 0.0 ni, 47.7 id, 0.0 wa, 0.0 hi, 49.7 si, 0.0 st %Cpu2 : 2.0 us, 2.0 sy, 0.0 ni, 39.9 id, 0.0 wa, 0.0 hi, 56.1 si, 0.0 st %Cpu3 : 2.0 us, 46.7 sy, 0.0 ni, 8.7 id, 0.0 wa, 0.0 hi, 42.7 si, 0.0 st %Cpu4 : 2.8 us, 3.5 sy, 0.0 ni, 48.6 id, 0.0 wa, 0.0 hi, 45.1 si, 0.0 st %Cpu5 : 1.4 us, 0.7 sy, 0.0 ni, 42.9 id, 0.0 wa, 0.0 hi, 55.1 si, 0.0 st %Cpu6 : 2.0 us, 2.7 sy, 0.0 ni, 38.7 id, 0.0 wa, 0.0 hi, 56.7 si, 0.0 st %Cpu7 : 1.4 us, 2.0 sy, 0.0 ni, 49.0 id, 0.0 wa, 0.0 hi, 47.6 si, 0.0 st %Cpu8 : 2.1 us, 8.9 sy, 0.7 ni, 39.0 id, 0.0 wa, 0.0 hi, 49.3 si, 0.0 st %Cpu9 : 1.4 us, 1.4 sy, 0.0 ni, 49.7 id, 0.0 wa, 0.0 hi, 47.6 si, 0.0 st %Cpu10 : 2.0 us, 2.0 sy, 0.7 ni, 47.7 id, 0.0 wa, 0.0 hi, 47.7 si, 0.0 st %Cpu11 : 1.4 us, 0.7 sy, 0.0 ni, 44.9 id, 0.7 wa, 0.0 hi, 52.2 si, 0.0 st MiB Mem : 31955.6 total, 27079.2 free, 2736.0 used, 3190.4 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29219.6 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

429254 root 20 0 304.9m 16.8m 99.9 0.1 1:27.11 R smartdns -c smartdns.conf 429258 root 20 0 304.9m 16.8m 6.0 0.1 0:05.09 S smartdns -c smartdns.conf 429256 root 15 -5 304.9m 16.8m 2.6 0.1 0:01.76 S smartdns -c smartdns.conf 429255 root 20 0 304.9m 16.8m 2.0 0.1 0:01.34 S smartdns -c smartdns.conf 429257 root 20 0 304.9m 16.8m 2.0 0.1 0:01.28 S smartdns -c smartdns.conf

PerfTop: 824 irqs/sec kernel:95.8% exact: 4.2% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 429254)

 3.90%  libc.so.6         [.] 0x000000000016d5ca
 3.10%  smartdns          [.] _dns_get_domain_from_packet
 1.77%  smartdns          [.] dict_hash
 1.74%  smartdns          [.] _dns_encode_domain
 1.63%  libc.so.6         [.] malloc
 1.33%  smartdns          [.] _dns_client_send_packet
 1.32%  libc.so.6         [.] pthread_mutex_lock
 1.22%  libc.so.6         [.] pthread_mutex_unlock
 1.16%  smartdns          [.] dns_cache_lookup
 1.06%  smartdns          [.] dns_cache_invalidate
 0.88%  smartdns          [.] _dns_decode_an
 0.86%  smartdns          [.] _dns_server_process_answer
 0.82%  libc.so.6         [.] cfree
 0.80%  libc.so.6         [.] clock_gettime
 0.78%  smartdns          [.] _dns_get_domain_offset
 0.60%  libc.so.6         [.] connect
 0.58%  smartdns          [.] dns_packet_init
 0.58%  smartdns          [.] _dns_client_get_request
 0.56%  smartdns          [.] tlog_vext
 0.55%  [vdso]            [.] __vdso_clock_gettime
 0.55%  smartdns          [.] _dns_read_short
 0.54%  libcrypto.so.3    [.] EVP_CIPHER_CTX_get_iv_length
 0.50%  smartdns          [.] _dns_server_request_release_complete
 0.48%  libc.so.6         [.] pthread_cond_signal
 0.47%  libc.so.6         [.] pthread_cond_wait
 0.42%  smartdns          [.] _dns_client_recv
 0.41%  smartdns          [.] _dns_ip_address_check_add
 0.41%  smartdns          [.] _dns_client_create_socket_udp
 0.41%  smartdns          [.] jhash
 0.41%  smartdns          [.] _dns_server_get_answer
 0.40%  smartdns          [.] dns_decode
 0.40%  libc.so.6         [.] 0x000000000016ee5f
 0.40%  smartdns          [.] get_tick_count
 0.39%  smartdns          [.] tlog_ext
 0.39%  libcrypto.so.3    [.] 0x0000000000339a74
 0.37%  smartdns          [.] _dns_client_query_release
 0.37%  [vdso]            [.] __vdso_time
 0.37%  libc.so.6         [.] sprintf
 0.36%  libc.so.6         [.] 0x000000000008f051
 0.35%  [vdso]            [.] __vdso_gettimeofday
 0.35%  smartdns          [.] _dns_rr_add_end
 0.33%  smartdns          [.] _dns_server_process_udp_one
 0.33%  smartdns          [.] _dns_client_work
 0.33%  smartdns          [.] _dns_decode_qr_head
 0.32%  smartdns          [.] _dns_server_reply_passthrough
 0.32%  libc.so.6         [.] __errno_location
 0.31%  smartdns          [.] _dns_add_RAW
 0.31%  smartdns          [.] _dns_server_get_conf_ttl
 0.31%  smartdns          [.] fast_ping_start
 0.31%  libcrypto.so.3    [.] BIO_write
 0.29%  smartdns          [.] dns_get_rrs_start
 0.29%  smartdns          [.] dns_cache_update
 0.29%  libc.so.6         [.] pthread_rwlock_unlock
 0.28%  smartdns          [.] _fast_ping_period_run
 0.28%  smartdns          [.] _tlog_vprintf
 0.27%  libc.so.6         [.] recvmsg
 0.27%  libc.so.6         [.] 0x000000000016f047
 0.27%  smartdns          [.] pthread_mutex_lock@plt
 0.27%  smartdns          [.] _dns_server_reply_udp
 0.27%  smartdns          [.] _dns_server_process_answer_A
 0.26%  libcrypto.so.3    [.] CRYPTO_malloc
 0.26%  smartdns          [.] _dns_get_qr_head
 0.26%  smartdns          [.] _dns_client_process
 0.26%  smartdns          [.] _dns_server_do_query
 0.25%  smartdns          [.] _dns_reply_inpacket
 0.25%  libc.so.6         [.] __close
 0.25%  smartdns          [.] pthread_mutex_unlock@plt
 0.25%  smartdns          [.] _tlog_root_log_buffer
 0.25%  smartdns          [.] _dns_decode_rr_head
 0.25%  smartdns          [.] jhash
 0.24%  smartdns          [.] _dns_result_callback
 0.24%  smartdns          [.] _dns_read_int
 0.24%  libc.so.6         [.] pthread_rwlock_rdlock
 0.24%  libc.so.6         [.] getaddrinfo
 0.24%  libc.so.6         [.] 0x0000000000058949
 0.24%  libc.so.6         [.] 0x000000000008f0c0
 0.24%  smartdns          [.] _dns_add_rrs_start
 0.24%  smartdns          [.] _dns_server_ping_result
pymumu commented 1 year ago

disable log.

micron10 commented 1 year ago

set log to off - same on 100%

PerfTop: 4399 irqs/sec kernel:96.3% exact: 3.7% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 434535)

 4.09%  smartdns          [.] _dns_get_domain_from_packet
 2.09%  smartdns          [.] _dns_client_send_packet
 1.95%  libc.so.6         [.] 0x000000000016d5ca
 1.86%  smartdns          [.] _dns_encode_domain
 1.81%  libc.so.6         [.] malloc
 1.33%  smartdns          [.] dict_hash
 1.31%  libc.so.6         [.] pthread_mutex_lock
 1.24%  libc.so.6         [.] getaddrinfo
 1.06%  smartdns          [.] _dns_server_process_answer
 0.99%  libc.so.6         [.] cfree
 0.93%  libc.so.6         [.] pthread_mutex_unlock
 0.83%  smartdns          [.] _dns_get_domain_offset
 0.82%  smartdns          [.] _dns_decode_an
 0.77%  smartdns          [.] _dns_client_create_socket_udp
 0.76%  libc.so.6         [.] epoll_wait
 0.76%  smartdns          [.] dns_get_rrs_start
 0.71%  smartdns          [.] _dns_client_get_request
 0.70%  smartdns          [.] _dns_server_process_answer_A
 0.70%  smartdns          [.] _dns_ip_address_check_add
 0.69%  smartdns          [.] _dns_read_short
 0.68%  libc.so.6         [.] pthread_rwlock_unlock
 0.65%  [vdso]            [.] __vdso_clock_gettime
 0.63%  smartdns          [.] pthread_mutex_unlock@plt
 0.62%  libc.so.6         [.] pthread_cond_wait
 0.57%  smartdns          [.] dns_cache_lookup
 0.54%  smartdns          [.] fast_ping_start
 0.54%  libc.so.6         [.] connect
 0.53%  smartdns          [.] _dns_client_recv
 0.52%  smartdns          [.] jhash
 0.52%  smartdns          [.] _dns_rr_add_end
 0.51%  smartdns          [.] _dns_client_process
 0.50%  libc.so.6         [.] pthread_rwlock_rdlock
 0.50%  smartdns          [.] _dns_server_process_udp_one
 0.49%  smartdns          [.] _dns_client_process_udp
 0.49%  smartdns          [.] _fast_ping_process_icmp
 0.48%  libcrypto.so.3    [.] CRYPTO_free
 0.48%  smartdns          [.] memcpy@plt
 0.47%  smartdns          [.] _dns_client_query_release
 0.47%  smartdns          [.] _fast_ping_close_host_sock
 0.45%  libc.so.6         [.] clock_gettime
 0.42%  smartdns          [.] jhash
 0.42%  smartdns          [.] tlog_vext
 0.42%  smartdns          [.] _DNS_client_create_socket_tls
 0.39%  smartdns          [.] _dns_add_RAW
 0.39%  smartdns          [.] _dns_decode_qr_head
 0.39%  smartdns          [.] _dns_get_qr_head
 0.38%  smartdns          [.] _dns_decode_body
 0.38%  libcrypto.so.3    [.] EVP_CIPHER_CTX_get_iv_length
 0.36%  libc.so.6         [.] 0x00000000000a0cdc
 0.36%  smartdns          [.] tlog_ext
 0.36%  smartdns          [.] _dns_replied_check_add
 0.35%  libc.so.6         [.] 0x000000000008f0b3
 0.35%  libcrypto.so.3    [.] CRYPTO_gcm128_decrypt_ctr32
 0.34%  smartdns          [.] _dns_cache_remove_by_domain
 0.33%  libcrypto.so.3    [.] OPENSSL_sk_pop_free
 0.33%  smartdns          [.] dns_cache_is_soa
 0.33%  smartdns          [.] _dns_server_get_conf_ttl
 0.33%  libc.so.6         [.] 0x00000000000a0368
 0.33%  libc.so.6         [.] inet_ntop
 0.32%  smartdns          [.] _dns_server_process_cname_pre
 0.32%  smartdns          [.] _dns_server_period_run
 0.32%  smartdns          [.] dns_cache_invalidate
 0.32%  smartdns          [.] _dns_decode_qd
 0.32%  libcrypto.so.3    [.] OSSL_PARAM_construct_size_t
 0.31%  smartdns          [.] _dns_decode_head
 0.31%  libcrypto.so.3    [.] OSSL_PARAM_locate
 0.31%  libc.so.6         [.] 0x00000000000a10ec
 0.31%  smartdns          [.] _dns_server_ping_result
 0.31%  smartdns          [.] _fast_ping_period_run
 0.30%  libcrypto.so.3    [.] 0x00000000002de770
 0.30%  libcrypto.so.3    [.] X509_VERIFY_PARAM_inherit
 0.30%  libcrypto.so.3    [.] OSSL_PARAM_get_int32
 0.30%  libcrypto.so.3    [.] CRYPTO_gcm128_setiv
 0.28%  libc.so.6         [.] 0x000000000016ccb7
 0.28%  smartdns          [.] pthread_mutex_lock@plt
 0.28%  libc.so.6         [.] 0x000000000008f0c0
 0.28%  libc.so.6         [.] 0x0000000000039ac7
 0.28%  smartdns          [.] _dns_server_get_answer
pymumu commented 1 year ago

Do not filter kernel functions

micron10 commented 1 year ago

sorry

PerfTop: 4364 irqs/sec kernel:96.9% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 438327)

28.11%  [kernel]        [k] __ipv6_dev_get_saddr
11.07%  [kernel]        [k] l3mdev_master_ifindex_rcu
 7.64%  [pppoe]         [k] pppoe_rcv
 3.45%  [kernel]        [k] ipv6_dev_get_saddr
 1.89%  [kernel]        [k] __dev_queue_xmit
 1.59%  [kernel]        [k] fib_table_lookup
 1.46%  [nf_tables]     [k] nft_do_chain
 1.43%  [kernel]        [k] __netif_receive_skb_core.constprop.0
 1.17%  [ixgbe]         [k] ixgbe_clean_rx_irq
 1.09%  [kernel]        [k] memcmp
 1.01%  [nf_tables]     [k] __nft_rbtree_lookup
 1.00%  [kernel]        [k] dev_queue_xmit_nit
 0.99%  [kernel]        [k] strncpy
 0.89%  [kernel]        [k] skb_clone
 0.86%  [kernel]        [k] __copy_skb_header
 0.84%  [nf_tables]     [k] nft_rhash_lookup
 0.80%  [kernel]        [k] kmem_cache_free
 0.80%  [nf_conntrack]  [k] __nf_conntrack_find_get.isra.0
 0.80%  [kernel]        [k] skb_release_data
 0.77%  [kernel]        [k] __skb_flow_dissect
 0.71%  [ixgbe]         [k] ixgbe_tx_map
 0.67%  [kernel]        [k] csum_partial_copy_generic
 0.56%  [kernel]        [k] ip_finish_output2
 0.55%  [kernel]        [k] kmem_cache_alloc
 0.53%  [kernel]        [k] ip_route_input_slow
 0.50%  [kernel]        [k] nf_hook_slow
 0.50%  [kernel]        [k] ktime_get_with_offset
 0.50%  [vlan_mon]      [k] vlan_pt_recv
 0.49%  [kernel]        [k] dev_hard_start_xmit
 0.45%  [kernel]        [k] kfree_skb_reason
 0.43%  [ixgbe]         [k] ixgbe_xmit_frame_ring
 0.42%  [kernel]        [k] ip_rcv_core.constprop.0
 0.41%  [nf_tables]     [k] nft_lookup_eval
 0.38%  [kernel]        [k] __siphash_unaligned
 0.36%  [kernel]        [k] vlan_do_receive
 0.36%  [kernel]        [k] csum_partial
 0.36%  [kernel]        [k] ip_forward
 0.35%  [kernel]        [k] netif_skb_features
 0.34%  [kernel]        [k] process_backlog
 0.34%  [kernel]        [k] ___slab_alloc.isra.0
 0.34%  [kernel]        [k] validate_xmit_skb.isra.0
 0.33%  [kernel]        [k] get_rps_cpu
 0.33%  [kernel]        [k] page_frag_free
 0.30%  [nf_tables]     [k] nft_meta_get_eval
 0.30%  [nf_conntrack]  [k] nf_conntrack_in
 0.29%  [kernel]        [k] ip_output
 0.29%  [nf_nat]        [k] l4proto_manip_pkt
 0.28%  [kernel]        [k] kmem_cache_free_bulk.part.0
 0.28%  [kernel]        [k] read_tsc
 0.28%  [kernel]        [k] __rcu_read_unlock
 0.27%  [kernel]        [k] netdev_core_pick_tx
 0.27%  [kernel]        [k] _raw_spin_lock_bh
 0.27%  [kernel]        [k] _raw_spin_lock_irqsave
 0.26%  [kernel]        [k] sch_direct_xmit
 0.26%  [kernel]        [k] netdev_pick_tx
 0.26%  [kernel]        [k] __local_bh_enable_ip
 0.24%  [pppoe]         [k] pppoe_xmit
 0.23%  [kernel]        [k] dev_gro_receive
 0.22%  [kernel]        [k] __slab_free.isra.0
 0.22%  [ixgbe]         [k] ixgbe_clean_tx_irq
 0.22%  [kernel]        [k] __rcu_read_lock
 0.21%  [kernel]        [k] skb_network_protocol
 0.20%  [ppp_generic]   [k] __ppp_xmit_process
 0.20%  [nf_nat]        [k] nf_nat_ipv4_manip_pkt
 0.20%  [kernel]        [k] fib_validate_source
 0.20%  [kernel]        [k] dma_map_page_attrs
 0.20%  [ppp_generic]   [k] ppp_push
 0.20%  [ixgbe]         [k] ixgbe_process_skb_fields
 0.19%  [ppp_generic]   [k] ppp_start_xmit
 0.19%  [kernel]        [k] _raw_spin_lock
 0.18%  [kernel]        [k] inet_gro_receive
 0.17%  [nf_tables]     [k] nft_rbtree_lookup
 0.17%  [kernel]        [k] vlan_dev_hard_start_xmit
 0.17%  [kernel]        [k] napi_consume_skb
 0.17%  [nf_xnatlog]    [k] xnatlog_hook
 0.17%  [kernel]        [k] enqueue_to_backlog
 0.16%  smartdns        [.] _dns_get_domain_from_packet
 0.16%  [kernel]        [k] __list_del_entry_valid
pymumu commented 1 year ago

modify dns_server.c:1181

static int _dns_server_reply_udp(struct dns_request *request, struct dns_server_conn_udp *udpserver,
                                 unsigned char *inpacket, int inpacket_len)
{
    int send_len = 0;
    struct iovec iovec[1];
    struct msghdr msg;
    struct cmsghdr *cmsg;
    char msg_control[64];

    if (atomic_read(&server.run) == 0 || inpacket == NULL || inpacket_len <= 0) {
        return -1;
    }

>   goto use_send;

    iovec[0].iov_base = inpacket;
    iovec[0].iov_len = inpacket_len;
micron10 commented 1 year ago

same :

PerfTop: 4361 irqs/sec kernel:96.4% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 444156)

28.07%  [kernel]        [k] __ipv6_dev_get_saddr
11.73%  [kernel]        [k] l3mdev_master_ifindex_rcu
 7.44%  [pppoe]         [k] pppoe_rcv
 3.92%  [kernel]        [k] ipv6_dev_get_saddr
 2.04%  [kernel]        [k] __dev_queue_xmit
 1.55%  [kernel]        [k] fib_table_lookup
 1.49%  [nf_tables]     [k] nft_do_chain
 1.35%  [kernel]        [k] __netif_receive_skb_core.constprop.0
 1.29%  [ixgbe]         [k] ixgbe_clean_rx_irq
 1.09%  [kernel]        [k] memcmp
 1.07%  [nf_tables]     [k] __nft_rbtree_lookup
 1.05%  [kernel]        [k] strncpy
 0.89%  [kernel]        [k] kmem_cache_free
 0.89%  [kernel]        [k] dev_queue_xmit_nit
 0.86%  [kernel]        [k] __skb_flow_dissect
 0.83%  [kernel]        [k] skb_clone
 0.80%  [kernel]        [k] csum_partial_copy_generic
 0.78%  [nf_tables]     [k] nft_rhash_lookup
 0.72%  [nf_conntrack]  [k] __nf_conntrack_find_get.isra.0
 0.70%  [kernel]        [k] skb_release_data
 0.70%  [kernel]        [k] __copy_skb_header
 0.66%  [ixgbe]         [k] ixgbe_tx_map
 0.52%  [kernel]        [k] ip_finish_output2
 0.52%  [kernel]        [k] ip_route_input_slow
 0.51%  [kernel]        [k] kmem_cache_alloc
 0.48%  [kernel]        [k] dev_hard_start_xmit
 0.47%  [kernel]        [k] nf_hook_slow
 0.46%  [kernel]        [k] kfree_skb_reason
 0.44%  [vlan_mon]      [k] vlan_pt_recv
 0.43%  [kernel]        [k] __siphash_unaligned
 0.38%  [kernel]        [k] ip_rcv_core.constprop.0
 0.35%  [kernel]        [k] vlan_do_receive
 0.35%  [kernel]        [k] page_frag_free
 0.35%  [nf_tables]     [k] nft_lookup_eval
 0.34%  [kernel]        [k] netif_skb_features
 0.34%  [ixgbe]         [k] ixgbe_xmit_frame_ring
 0.33%  [kernel]        [k] process_backlog
 0.33%  [kernel]        [k] kmem_cache_free_bulk.part.0
 0.33%  [kernel]        [k] _raw_spin_lock_irqsave
 0.32%  [kernel]        [k] validate_xmit_skb.isra.0
 0.32%  [kernel]        [k] ___slab_alloc.isra.0
 0.31%  [kernel]        [k] get_rps_cpu
 0.31%  [nf_tables]     [k] nft_meta_get_eval
 0.30%  [kernel]        [k] __rcu_read_unlock
 0.29%  [kernel]        [k] _raw_spin_lock_bh
 0.28%  [kernel]        [k] sch_direct_xmit
 0.28%  [nf_nat]        [k] l4proto_manip_pkt
 0.27%  [kernel]        [k] ip_forward
 0.27%  [kernel]        [k] __local_bh_enable_ip
 0.27%  [kernel]        [k] netdev_core_pick_tx
 0.26%  [nf_conntrack]  [k] nf_conntrack_in
 0.26%  [kernel]        [k] dev_gro_receive
 0.25%  [kernel]        [k] ktime_get_with_offset
 0.25%  [kernel]        [k] ip_output
 0.23%  [pppoe]         [k] pppoe_xmit
 0.23%  [kernel]        [k] fib_validate_source
 0.22%  [kernel]        [k] netdev_pick_tx
 0.21%  [ppp_generic]   [k] ppp_start_xmit
 0.21%  [kernel]        [k] skb_network_protocol
 0.21%  [kernel]        [k] __slab_free.isra.0
 0.20%  [kernel]        [k] __rcu_read_lock
 0.20%  [kernel]        [k] _raw_spin_lock
 0.20%  [kernel]        [k] inet_gro_receive
 0.20%  [ixgbe]         [k] ixgbe_process_skb_fields
 0.20%  [kernel]        [k] dma_map_page_attrs
 0.19%  [kernel]        [k] csum_partial
 0.19%  [ppp_generic]   [k] __ppp_xmit_process
 0.19%  [nf_nat]        [k] nf_nat_ipv4_manip_pkt
 0.19%  [kernel]        [k] read_tsc
 0.18%  [kernel]        [k] __list_del_entry_valid
 0.17%  [ppp_generic]   [k] ppp_push
 0.17%  [kernel]        [k] napi_consume_skb
 0.17%  [kernel]        [k] vlan_dev_hard_start_xmit
 0.17%  [ixgbe]         [k] ixgbe_clean_tx_irq
 0.16%  [kernel]        [k] kmem_cache_alloc_bulk
 0.15%  [kernel]        [k] napi_build_skb
 0.15%  [nf_tables]     [k] nft_rbtree_lookup
 0.15%  [ppp_generic]   [k] ppp_xmit_process
pymumu commented 1 year ago

what is you kernel version?