search

pymumu / smartdns

A local DNS server to obtain the fastest website IP for the best Internet experience, support DoT, DoH. 一个本地DNS服务器，获取最快的网站IP，获得最佳上网体验，支持DoH，DoT。
https://pymumu.github.io/smartdns/
GNU General Public License v3.0
8.3k stars 1.06k forks source link

Release 43 / High CPU load #1477

Closed micron10 closed 1 year ago

micron10 commented 1 year ago

Hi

I try and install latest version Release 43 , but after run smartdns process stay on 100% load

try to clena conf and remove all , remove cache ,remove logs ,..... but still on 100%

in machine have 5000 users

micron10 commented 1 year ago

6.4.11

pymumu commented 1 year ago 
28.07%  [kernel]        [k] __ipv6_dev_get_saddr
11.73%  [kernel]        [k] l3mdev_master_ifindex_rcu
 7.44%  [pppoe]         [k] pppoe_rcv

These three functions take up a lot of CPU, and we need to find out what triggers these three functions. These functions seem to be related to obtaining the source of the ipv6 address.

pymumu commented 1 year ago

Is there any ip6table or nftable rules configured?

micron10 commented 1 year ago

for ipv6 no rull yes have nftables rull but only for ipv4.

and one more after stop smartdns and back to knot or bind this process miss.

one more try to disable ipv6 Nd still same

PikuZheng commented 1 year ago 
28.07%  [kernel]        [k] __ipv6_dev_get_saddr
11.73%  [kernel]        [k] l3mdev_master_ifindex_rcu
 7.44%  [pppoe]         [k] pppoe_rcv

These three functions take up a lot of CPU, and we need to find out what triggers these three functions. These functions seem to be related to obtaining the source of the ipv6 address.

It seems that cpu is wasted processing multiple ipv6 addresses (ipv6 routing problem

this is knot-resolver : … -- Network interface configuration net.listen('0.0.0.0', 53, { kind = 'dns' }) net.listen('0.0.0.0', 853, { kind = 'tls' }) i think bind same use default not binds to all interfacess

Does this mean knot is not handling ipv6?

micron10 commented 1 year ago 
28.07%  [kernel]        [k] __ipv6_dev_get_saddr
11.73%  [kernel]        [k] l3mdev_master_ifindex_rcu
 7.44%  [pppoe]         [k] pppoe_rcv

These three functions take up a lot of CPU, and we need to find out what triggers these three functions. These functions seem to be related to obtaining the source of the ipv6 address.

It seems that cpu is wasted processing multiple ipv6 addresses (ipv6 routing problem

But why this problem miss with other dns server

this is knot-resolver : … -- Network interface configuration net.listen('0.0.0.0', 53, { kind = 'dns' }) net.listen('0.0.0.0', 853, { kind = 'tls' }) i think bind same use default not binds to all interfacess

Does this mean knot is not handling ipv6?

no i copy to show how is set in on of config. knot list on all interface.

one note

after i set

bind [::]:53@bond0.0101

if customer from ppp try to get dns request from ip address on interface not work

only if try nslookup from out side or if set lo dev and work on local host

i think need to find this ipv6 problem and to fix.

is there options to disable ipv6 in smartdns for test . or may be this is kernel bug

PikuZheng commented 1 year ago

net.listen('0.0.0.0', 53, { kind = 'dns' }) means ipv4 only bind [::]:53 means ipv4 and ipv6

micron10 commented 1 year ago

send as example

in real case knot and bind work on both ipv4 and ipv6.

in smartdns conf i try to remove ipv6

bind :53

but in perf top stay same process

https://knot-resolver.readthedocs.io/en/stable/daemon-bindings-net_server.html

PikuZheng commented 1 year ago

in smartdns conf i try to remove ipv6

bind :53

it should be bind 0.0.0.0:53

micron10 commented 1 year ago

set in config : bind 0.0.0.0:53

and disable ipv6 on system : sysctl net.ipv6.conf.all.disable_ipv6=1

and still is same

and one note after i set

bind 0.0.0.0:53@eth0

user that is connect on pppXXX not see dns

but if test from out side of system dns resolv

PerfTop: 4195 irqs/sec kernel:96.0% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 1191054)

28.15%  [kernel]          [k] __ipv6_dev_get_saddr
17.41%  [kernel]          [k] l3mdev_master_ifindex_rcu
 5.45%  [pppoe]           [k] pppoe_rcv
 5.05%  [kernel]          [k] ipv6_dev_get_saddr
 1.79%  [kernel]          [k] __dev_queue_xmit
 1.38%  [nf_tables]       [k] nft_do_chain
 1.26%  [kernel]          [k] fib_table_lookup
 1.18%  [kernel]          [k] __netif_receive_skb_core.constprop.0
 1.07%  [nf_tables]       [k] __nft_rbtree_lookup
 1.02%  [kernel]          [k] memcmp
 0.93%  [kernel]          [k] strncpy
 0.88%  [kernel]          [k] dev_queue_xmit_nit
 0.82%  [ixgbe]           [k] ixgbe_clean_rx_irq
 0.77%  [kernel]          [k] kmem_cache_free
 0.74%  [nf_tables]       [k] nft_rhash_lookup
 0.73%  [kernel]          [k] csum_partial_copy_generic
 0.72%  [kernel]          [k] __skb_flow_dissect
 0.72%  [kernel]          [k] __copy_skb_header
 0.69%  [kernel]          [k] skb_clone
 0.66%  [nf_conntrack]    [k] __nf_conntrack_find_get.isra.0
 0.58%  [kernel]          [k] skb_release_data
 0.57%  [ixgbe]           [k] ixgbe_tx_map
 0.49%  [kernel]          [k] ip_finish_output2
 0.48%  [kernel]          [k] ip_route_input_slow
 0.46%  [kernel]          [k] kmem_cache_alloc
 0.38%  [kernel]          [k] dev_hard_start_xmit
 0.36%  [kernel]          [k] ktime_get_with_offset
 0.35%  [kernel]          [k] nf_hook_slow
 0.35%  [vlan_mon]        [k] vlan_pt_recv
 0.35%  [kernel]          [k] kfree_skb_reason
 0.35%  [nf_tables]       [k] nft_lookup_eval
 0.32%  [ixgbe]           [k] ixgbe_xmit_frame_ring
 0.32%  [kernel]          [k] page_frag_free
 0.32%  [kernel]          [k] kmem_cache_free_bulk.part.0
 0.32%  [kernel]          [k] __siphash_unaligned
 0.31%  [kernel]          [k] ___slab_alloc.isra.0
 0.31%  [kernel]          [k] _raw_spin_lock_irqsave
 0.30%  [kernel]          [k] process_backlog
 0.30%  [kernel]          [k] vlan_do_receive
 0.28%  [kernel]          [k] netif_skb_features
 0.28%  [nf_conntrack]    [k] nf_conntrack_in
 0.27%  [kernel]          [k] get_rps_cpu
 0.27%  [kernel]          [k] read_tsc
 0.27%  [kernel]          [k] netdev_pick_tx
 0.26%  [kernel]          [k] _raw_spin_lock_bh
 0.26%  [kernel]          [k] ip_output
 0.25%  [kernel]          [k] __local_bh_enable_ip
 0.25%  [kernel]          [k] ip_forward
 0.25%  [kernel]          [k] ip_rcv_core.constprop.0
 0.24%  [kernel]          [k] netdev_core_pick_tx
 0.23%  [nf_tables]       [k] nft_meta_get_eval
 0.23%  [kernel]          [k] __rcu_read_unlock
 0.23%  [kernel]          [k] validate_xmit_skb.isra.0
 0.22%  [nf_nat]          [k] l4proto_manip_pkt
 0.22%  [kernel]          [k] sch_direct_xmit
 0.21%  [kernel]          [k] skb_network_protocol
 0.21%  [kernel]          [k] __slab_free.isra.0
 0.20%  [ppp_generic]     [k] __ppp_xmit_process
 0.20%  [kernel]          [k] dev_gro_receive
 0.20%  [kernel]          [k] __rcu_read_lock
 0.18%  [kernel]          [k] _raw_spin_lock
 0.18%  [ppp_generic]     [k] ppp_start_xmit
 0.17%  [kernel]          [k] fib_validate_source
 0.16%  [kernel]          [k] vlan_dev_hard_start_xmit
 0.16%  [kernel]          [k] __list_del_entry_valid
 0.15%  [pppoe]           [k] pppoe_xmit
 0.15%  [ixgbe]           [k] ixgbe_clean_tx_irq
 0.15%  smartdns          [.] _dns_get_domain_from_packet
 0.14%  [ppp_generic]     [k] ppp_push
 0.14%  [kernel]          [k] dma_map_page_attrs
 0.14%  [nf_nat]          [k] nf_nat_ipv4_manip_pkt
 0.13%  libc.so.6         [.] 0x000000000016d5ca
 0.13%  [nf_tables]       [k] nft_rbtree_lookup
 0.13%  [kernel]          [k] dst_release
 0.13%  [kernel]          [k] csum_partial
 0.12%  [nf_conntrack]    [k] nf_conntrack_tcp_packet
 0.12%  [nf_nat]          [k] nf_nat_manip_pkt
 0.12%  [kernel]          [k] skb_segment
PikuZheng commented 1 year ago

28.15% [kernel] [k] __ipv6_dev_get_saddr 17.41% [kernel] [k] l3mdev_master_ifindex_rcu

it's strange after disable ipv6... now I believe it is relate with multiple ipv6 addresses

PikuZheng commented 1 year ago

could you try to disable speedtest bind...... -no-speed-check or check how many outbound (to internet) connections there are

micron10 commented 1 year ago

set -no-speed-check but is same.

if you write how i will check how many outbound there are

micron10 commented 1 year ago

Hi @PikuZheng @pymumu

check attached src : https://easyupload.io/efluq3

i disable part of IPv6. code in dns server after that dns start work and here is statistic and perf :

Threads: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 87.6 id, 0.0 wa, 0.0 hi, 12.4 si, 0.0 st %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 91.2 id, 0.0 wa, 0.0 hi, 8.8 si, 0.0 st %Cpu2 : 0.7 us, 0.7 sy, 0.0 ni, 87.9 id, 0.0 wa, 0.0 hi, 10.7 si, 0.0 st %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 90.5 id, 0.0 wa, 0.0 hi, 9.5 si, 0.0 st %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni, 91.2 id, 0.0 wa, 0.0 hi, 8.8 si, 0.0 st %Cpu5 : 0.0 us, 0.7 sy, 0.0 ni, 88.6 id, 0.0 wa, 0.0 hi, 10.7 si, 0.0 st %Cpu6 : 0.0 us, 0.7 sy, 0.0 ni, 88.4 id, 0.0 wa, 0.0 hi, 10.9 si, 0.0 st %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni, 90.6 id, 0.0 wa, 0.0 hi, 9.4 si, 0.0 st %Cpu8 : 0.0 us, 0.7 sy, 0.0 ni, 89.3 id, 0.0 wa, 0.0 hi, 10.1 si, 0.0 st %Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 89.8 id, 0.0 wa, 0.0 hi, 10.2 si, 0.0 st %Cpu10 : 0.7 us, 0.0 sy, 0.0 ni, 87.8 id, 0.0 wa, 0.0 hi, 11.5 si, 0.0 st %Cpu11 : 0.0 us, 0.0 sy, 0.0 ni, 91.8 id, 0.0 wa, 0.0 hi, 8.2 si, 0.0 st MiB Mem : 31955.6 total, 26634.0 free, 2739.4 used, 3635.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29216.2 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

1431343 root 20 0 240.4m 17.3m 3.3 0.1 0:15.68 R smartdns -c smartdns.conf 1431347 root 20 0 240.4m 17.3m 1.3 0.1 0:02.48 S smartdns -c smartdns.conf 1431344 root 20 0 240.4m 17.3m 0.0 0.1 0:00.00 S smartdns -c smartdns.conf 1431345 root 15 -5 240.4m 17.3m 0.0 0.1 0:00.62 S smartdns -c smartdns.conf 1431346 root 20 0 240.4m 17.3m 0.0 0.1 0:00.55 S smartdns -c smartdns.conf

Perf :

PerfTop: 107 irqs/sec kernel:57.0% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 1431343)

 3.11%  libc.so.6         [.] 0x000000000016d5ca
 2.95%  [kernel]          [k] __dev_queue_xmit
 1.87%  smartdns          [.] dns_cache_invalidate
 1.75%  smartdns          [.] _dns_get_domain_from_packet
 1.30%  [nf_tables]       [k] nft_rhash_lookup
 1.19%  [kernel]          [k] fib_table_lookup
 1.14%  smartdns          [.] dict_hash
 1.03%  [kernel]          [k] memcmp
 0.98%  [kernel]          [k] kmem_cache_free
 0.97%  [kernel]          [k] udp_sendmsg
 0.96%  [kernel]          [k] __fdget
 0.92%  libc.so.6         [.] pthread_mutex_lock
 0.90%  [kernel]          [k] ip_finish_output2
 0.90%  [kernel]          [k] __ip_append_data
 0.89%  [nf_tables]       [k] nft_do_chain
 0.86%  [nf_conntrack]    [k] __nf_conntrack_find_get.isra.0
 0.86%  libc.so.6         [.] malloc
 0.84%  [kernel]          [k] csum_partial_copy_generic
 0.75%  [kernel]          [k] skb_release_data
 0.74%  [kernel]          [k] __skb_flow_dissect
 0.74%  [nf_conntrack]    [k] nf_conntrack_in
 0.74%  [kernel]          [k] __copy_skb_header
 0.73%  smartdns          [.] _dns_decode_an
 0.73%  [kernel]          [k] kmem_cache_alloc_node
 0.72%  [kernel]          [k] timerqueue_add
 0.67%  [ixgbe]           [k] ixgbe_tx_map
 0.66%  [kernel]          [k] _raw_spin_lock
 0.64%  [kernel]          [k] __netif_receive_skb_core.constprop.0
 0.63%  smartdns          [.] _dns_encode_domain
 0.62%  smartdns          [.] _dns_server_process_udp_one
 0.62%  [kernel]          [k] page_frag_free
 0.61%  [kernel]          [k] ip_send_skb
 0.59%  [nf_tables]       [k] __nft_rbtree_lookup
 0.58%  [kernel]          [k] __rcu_read_unlock
 0.58%  [kernel]          [k] ep_poll
 0.57%  [kernel]          [k] __siphash_unaligned
 0.57%  [kernel]          [k] __schedule
 0.55%  [kernel]          [k] _raw_spin_lock_irqsave
 0.55%  [kernel]          [k] netdev_core_pick_tx
 0.53%  [kernel]          [k] dev_queue_xmit_nit
 0.49%  [kernel]          [k] __list_del_entry_valid
 0.49%  [kernel]          [k] skb_clone
 0.49%  [ixgbe]           [k] ixgbe_xmit_frame_ring
 0.49%  smartdns          [.] _dns_decode_head
 0.48%  [ppp_generic]     [k] ppp_xmit_process
 0.48%  smartdns          [.] dns_packet_init
 0.47%  [ixgbe]           [k] ixgbe_poll
 0.46%  [kernel]          [k] rep_movs_alternative
 0.45%  smartdns          [.] _dns_server_process_cache_packet
 0.45%  [kernel]          [k] __kmem_cache_alloc_node
 0.44%  smartdns          [.] dns_decode
 0.44%  smartdns          [.] dns_server_run
 0.44%  libc.so.6         [.] sprintf
 0.44%  [kernel]          [k] __update_load_avg_se
 0.43%  smartdns          [.] memcpy@plt
 0.41%  smartdns          [.] _dns_add_qr_head
 0.40%  [kernel]          [k] __ip_make_skb
 0.39%  [kernel]          [k] exit_to_user_mode_prepare
 0.39%  [kernel]          [k] skb_network_protocol
 0.39%  smartdns          [.] dns_cache_update
 0.39%  [kernel]          [k] sch_direct_xmit
 0.39%  smartdns          [.] dns_cache_lookup
 0.38%  [kernel]          [k] process_backlog
 0.38%  [kernel]          [k] __x64_sys_epoll_wait
 0.37%  [vdso]            [.] __vdso_clock_gettime
 0.36%  smartdns          [.] _dns_rr_add_end
 0.36%  libc.so.6         [.] 0x000000000008f0b3
 0.36%  [kernel]          [k] kmem_cache_alloc
 0.36%  libc.so.6         [.] clock_gettime
 0.36%  libc.so.6         [.] recvmsg
 0.35%  [kernel]          [k] newidle_balance
 0.35%  libssl.so.3       [.] 0x000000000003c5c0
 0.35%  [kernel]          [k] schedule_hrtimeout_range_clock
 0.35%  smartdns          [.] _dns_server_reply_passthrough
 0.34%  smartdns          [.] _dns_decode_rr_head
 0.33%  [kernel]          [k] kfree_skb_reason
 0.33%  smartdns          [.] dns_cache_is_soa
 0.33%  smartdns          [.] _dns_server_qtype_soa

DIG :

; <<>> DiG 9.18.18 <<>> cisco.com 127.0.0.1 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12174 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;cisco.com. IN A

;; ANSWER SECTION: cisco.com. 409 IN A 72.163.4.185

;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Mon Aug 28 02:23:54 IST 2023 ;; MSG SIZE rcvd: 43

;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 6870 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION: ;127.0.0.1. IN A

;; AUTHORITY SECTION: 127.0.0.1. 60 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2023082701 1800 900 604800 86400

;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Mon Aug 28 02:23:54 IST 2023 ;; MSG SIZE rcvd: 103

pymumu commented 1 year ago

In order to determine whether it is a smartdns issue or kernel issue, you can use the top command to check the us and sy values ​​of the CPU usage. us stands for CPU usage in user mode, and sy stands for CPU usage in kernel mode

top -H -p [pid]
micron10 commented 1 year ago

Hi @pymumu

see attached perf.data - this with ver where all is enable in dns_server.c (top -H -p same)

https://easyupload.io/k3ep8l - this is perf.data

this is top :

Threads: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.0 hi, 38.8 si, 0.0 st %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 68.1 id, 0.0 wa, 0.0 hi, 31.9 si, 0.0 st %Cpu2 : 0.0 us, 0.7 sy, 0.0 ni, 66.7 id, 0.0 wa, 0.0 hi, 32.6 si, 0.0 st %Cpu3 : 0.7 us, 0.0 sy, 0.0 ni, 60.8 id, 0.0 wa, 0.0 hi, 38.5 si, 0.0 st %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 36.1 si, 0.0 st %Cpu5 : 0.0 us, 0.7 sy, 0.0 ni, 71.9 id, 0.0 wa, 0.0 hi, 27.4 si, 0.0 st %Cpu6 : 30.5 us, 35.1 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 34.4 si, 0.0 st %Cpu7 : 0.0 us, 0.7 sy, 0.0 ni, 63.3 id, 0.0 wa, 0.0 hi, 36.1 si, 0.0 st %Cpu8 : 0.7 us, 0.7 sy, 0.0 ni, 64.6 id, 0.0 wa, 0.0 hi, 34.0 si, 0.0 st %Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 64.6 id, 0.0 wa, 0.0 hi, 35.4 si, 0.0 st %Cpu10 : 0.0 us, 0.7 sy, 0.0 ni, 61.4 id, 0.0 wa, 0.0 hi, 37.9 si, 0.0 st %Cpu11 : 0.0 us, 0.7 sy, 0.0 ni, 65.2 id, 0.0 wa, 0.0 hi, 34.0 si, 0.0 st MiB Mem : 31955.6 total, 26567.0 free, 2758.1 used, 3686.1 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29197.5 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

1688745 root 20 0 251.5m 23.1m 99.9 0.1 0:25.80 R ./smartdns -c smartdns.conf 1688749 root 20 0 251.5m 23.1m 2.0 0.1 0:00.82 S ./smartdns -c smartdns.conf 1688747 root 15 -5 251.5m 23.1m 1.3 0.1 0:00.38 S ./smartdns -c smartdns.conf 1688748 root 20 0 251.5m 23.1m 0.7 0.1 0:00.28 S ./smartdns -c smartdns.conf 1688746 root 20 0 251.5m 23.1m 0.0 0.1 0:00.00 S ./smartdns -c smartdns.conf

micron10 commented 1 year ago

do you have time to see

pymumu commented 1 year ago

Too late, I will find another time to locate the problem.

by the way, the -nftset option may not working, you can remove it, and you can try latest code.

bind [::]:53@bond0.0101 -nftset
bind [::]:53@lo -nftset
micron10 commented 1 year ago

oki will try latest code... will remove optikns.

and if still stay on 100% will remove ipv6 support for now i will update you after test

micron10 commented 1 year ago

Hi @pymumu

after update src to latest master server work load reduce to 60-70%

One note after update src and upload new binary and try to start

daemon start try to read cache but not succes and kill :

run daemon process failed, wait child timeout run daemon failed, please check log at /var/log/smartdns/smartdns.log

[2023-08-31 03:25:21,025][NOTICE][ smartdns.c:485 ] smartdns starting...(Copyright (C) Nick Peng pymumu@gmail.com, build: Aug 30 2023 21:51:44) [2023-08-31 03:25:21,028][ INFO][ dns_cache.c:866 ] load cache file /var/run/smartdns.cache, total 953374 records

ls -la /var/run/smartdns.cache -rw-r----- 1 root root 452292659 Aug 30 10:26 /var/run/smartdns.cache

432M /var/run/smartdns.cache

after remove cache file service start

here is perf info :

PerfTop: 2724 irqs/sec kernel:60.4% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 3819392)

11.70%  [kernel]          [k] __snmp6_fill_stats64.constprop.0
 5.13%  [kernel]          [k] rtnl_fill_ifinfo.constprop.0.isra.0
 4.26%  [kernel]          [k] inet6_fill_ifla6_attrs
 3.55%  [kernel]          [k] inet_dump_ifaddr
 3.36%  libc.so.6         [.] 0x0000000000137cb8
 3.15%  libc.so.6         [.] 0x0000000000137a10
 2.90%  libc.so.6         [.] 0x0000000000137ca1
 2.86%  libc.so.6         [.] 0x00000000001379f9
 2.49%  [kernel]          [k] nla_put
 2.43%  libc.so.6         [.] 0x0000000000137ca8
 2.31%  libc.so.6         [.] 0x0000000000137a00
 1.86%  [pppoe]           [k] pppoe_rcv
 1.72%  libc.so.6         [.] 0x0000000000137a09
 1.56%  libc.so.6         [.] 0x00000000001379f0
 1.55%  libc.so.6         [.] 0x00000000001379fc
 1.42%  [kernel]          [k] entry_SYSCALL_64_after_hwframe
 1.27%  [kernel]          [k] memset_orig
 1.25%  libc.so.6         [.] 0x0000000000137cb1
 1.18%  libc.so.6         [.] 0x0000000000137c98
 1.16%  [vdso]            [.] __vdso_clock_gettime
 1.02%  [kernel]          [k] inet_fill_link_af
 1.01%  libc.so.6         [.] 0x0000000000137ca4
 0.96%  [kernel]          [k] skb_put
 0.85%  libc.so.6         [.] epoll_wait
 0.83%  [kernel]          [k] memcpy_orig
 0.82%  [kernel]          [k] __x64_sys_epoll_wait
 0.79%  libc.so.6         [.] 0x00000000001377da
 0.77%  [kernel]          [k] _raw_read_lock_bh
 0.77%  [kernel]          [k] rtnl_dump_ifinfo
 0.76%  libc.so.6         [.] 0x0000000000137aa8
 0.67%  [kernel]          [k] ep_poll
 0.60%  smartdns          [.] get_tick_count
 0.53%  [kernel]          [k] _raw_spin_lock_bh
 0.49%  [kernel]          [k] rtnl_fill_vf
 0.47%  [kernel]          [k] do_syscall_64
 0.46%  [kernel]          [k] __fdget
 0.45%  [kernel]          [k] __dev_queue_xmit
 0.41%  [kernel]          [k] syscall_enter_from_user_mode
 0.40%  [kernel]          [k] syscall_exit_to_user_mode
 0.40%  [kernel]          [k] inet6_dump_addr
 0.40%  [ppp_generic]     [k] ppp_get_stats64
 0.38%  [kernel]          [k] clear_page_erms
 0.38%  smartdns          [.] dns_server_run
 0.37%  [kernel]          [k] _find_next_bit
 0.36%  [kernel]          [k] entry_SYSCALL_64_safe_stack
 0.35%  [kernel]          [k] __entry_text_start
 0.33%  [kernel]          [k] nla_reserve
 0.32%  [kernel]          [k] syscall_return_via_sysret
 0.32%  libc.so.6         [.] 0x000000000008f051
 0.31%  [kernel]          [k] rep_movs_alternative
 0.31%  libc.so.6         [.] 0x000000000008f0dd
 0.30%  [ixgbe]           [k] ixgbe_clean_rx_irq
 0.30%  [kernel]          [k] __memset
 0.28%  [nf_tables]       [k] nft_do_chain
 0.28%  libc.so.6         [.] 0x000000000016d5ca
 0.27%  [kernel]          [k] fib_table_lookup
 0.26%  [kernel]          [k] _raw_spin_lock
 0.25%  libc.so.6         [.] 0x0000000000137755
 0.25%  [kernel]          [k] native_irq_return_iret
 0.25%  [kernel]          [k] __netif_receive_skb_core.constprop.0
 0.25%  [kernel]          [k] __list_del_entry_valid
 0.23%  [kernel]          [k] jiffies_to_msecs
 0.23%  [kernel]          [k] __memcpy
 0.23%  libc.so.6         [.] 0x00000000000a1952
 0.23%  [kernel]          [k] __skb_flow_dissect
 0.21%  [kernel]          [k] kmem_cache_free
 0.21%  [kernel]          [k] strlen
 0.21%  [kernel]          [k] __rcu_read_unlock
 0.20%  [nf_tables]       [k] __nft_rbtree_lookup
 0.20%  libc.so.6         [.] 0x000000000016cca0
 0.19%  [kernel]          [k] strncpy
 0.18%  [kernel]          [k] dev_queue_xmit_nit
 0.17%  smartdns          [.] _dns_server_process_local_ptr
 0.17%  [kernel]          [k] __rcu_read_lock
 0.17%  [kernel]          [k] __copy_skb_header
 0.17%  [kernel]          [k] skb_clone
 0.17%  [kernel]          [k] __local_bh_enable_ip
 0.17%  [nf_tables]       [k] nft_rhash_lookup
pymumu commented 1 year ago

What is the CPU usage of kernel mode and user mode?

micron10 commented 1 year ago

Hi @pymumu

See changes down in dns_server.c this changes apply over lastest master :

--- dns_server.c.orig 2023-09-01 10:34:38.199564546 +0000 +++ dns_server.c 2023-09-01 10:34:31.095421531 +0000 @@ -1179,6 +1179,8 @@ static int _dns_server_reply_udp(struct return -1; }

in other hand

see perf

PerfTop: 1162 irqs/sec kernel:59.4% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 813332)

29.51%  smartdns          [.] _dns_cache_insert_sorted
 8.22%  [pppoe]           [k] pppoe_rcv
 2.41%  [kernel]          [k] __dev_queue_xmit
 1.38%  [nf_tables]       [k] nft_do_chain
 1.33%  [nf_tables]       [k] __nft_rbtree_lookup
 1.20%  [kernel]          [k] __netif_receive_skb_core.constprop.0
 1.12%  [kernel]          [k] __skb_flow_dissect
 1.12%  [kernel]          [k] memcmp
 1.12%  [kernel]          [k] dev_queue_xmit_nit
 1.03%  [kernel]          [k] fib_table_lookup
 1.02%  [kernel]          [k] skb_clone
 1.00%  [kernel]          [k] kmem_cache_free
 0.98%  [ixgbe]           [k] ixgbe_clean_rx_irq
 0.95%  [nf_tables]       [k] nft_rhash_lookup
 0.87%  [kernel]          [k] csum_partial_copy_generic
 0.86%  [kernel]          [k] strncpy
 0.86%  [nf_conntrack]    [k] __nf_conntrack_find_get.isra.0
 0.85%  [kernel]          [k] __copy_skb_header
 0.79%  [kernel]          [k] ktime_get_with_offset
 0.78%  [ixgbe]           [k] ixgbe_tx_map
 0.78%  [kernel]          [k] skb_release_data
 0.67%  libc.so.6         [.] 0x000000000016d5ca
 0.59%  [kernel]          [k] ip_finish_output2
 0.55%  [kernel]          [k] kfree_skb_reason
 0.53%  smartdns          [.] _dns_get_domain_from_packet
 0.52%  [kernel]          [k] __siphash_unaligned
 0.50%  [kernel]          [k] _raw_spin_lock_bh
 0.50%  [nf_tables]       [k] nft_lookup_eval
 0.47%  [kernel]          [k] netif_skb_features
 0.44%  [kernel]          [k] kmem_cache_alloc
 0.44%  [kernel]          [k] get_rps_cpu
 0.44%  [vlan_mon]        [k] vlan_pt_recv
 0.42%  [kernel]          [k] nf_hook_slow
 0.42%  [kernel]          [k] netdev_core_pick_tx
 0.41%  [ixgbe]           [k] ixgbe_xmit_frame_ring
 0.41%  [kernel]          [k] vlan_do_receive
 0.40%  [kernel]          [k] kmem_cache_free_bulk.part.0
 0.39%  [kernel]          [k] dev_hard_start_xmit
 0.39%  [kernel]          [k] ip_rcv_core.constprop.0
 0.39%  [kernel]          [k] validate_xmit_skb.isra.0
 0.38%  [kernel]          [k] __local_bh_enable_ip
 0.35%  [kernel]          [k] ___slab_alloc.isra.0
 0.35%  [kernel]          [k] ip_route_input_slow
 0.35%  [kernel]          [k] ip_output
 0.31%  smartdns          [.] dns_cache_lookup
 0.31%  [kernel]          [k] read_tsc
 0.30%  [nf_nat]          [k] nf_nat_ipv4_manip_pkt
 0.30%  [kernel]          [k] dev_gro_receive
 0.29%  [nf_nat]          [k] l4proto_manip_pkt
 0.29%  [pppoe]           [k] pppoe_xmit
 0.29%  [kernel]          [k] _raw_spin_lock
 0.29%  [kernel]          [k] skb_network_protocol
 0.29%  [kernel]          [k] vlan_dev_hard_start_xmit
 0.28%  [kernel]          [k] csum_partial
 0.28%  [kernel]          [k] page_frag_free
 0.26%  [kernel]          [k] dma_map_page_attrs
 0.26%  [kernel]          [k] process_backlog
 0.25%  [kernel]          [k] __fdget
 0.25%  smartdns          [.] dict_hash
 0.24%  [nf_conntrack]    [k] nf_conntrack_in
 0.24%  [nf_tables]       [k] nft_meta_get_eval
 0.23%  [ixgbe]           [k] ixgbe_process_skb_fields
 0.23%  [ppp_generic]     [k] __ppp_xmit_process
 0.23%  [kernel]          [k] __slab_free.isra.0
 0.23%  [kernel]          [k] ip_route_output_key_hash_rcu
 0.22%  [kernel]          [k] ep_poll
 0.22%  [kernel]          [k] ip_rcv_finish_core.isra.0
 0.22%  [kernel]          [k] __rcu_read_unlock
 0.21%  [ppp_generic]     [k] ppp_xmit_process
 0.21%  [kernel]          [k] dma_unmap_page_attrs
 0.20%  [kernel]          [k] kmem_cache_alloc_node
 0.20%  [kernel]          [k] napi_gro_receive
 0.20%  [kernel]          [k] _raw_spin_lock_irqsave
 0.20%  [kernel]          [k] __lock_text_start
 0.19%  smartdns          [.] _dns_encode_domain
 0.19%  [kernel]          [k] inet_gro_receive
 0.19%  [kernel]          [k] netdev_pick_tx
 0.19%  [nf_xnatlog]      [k] xnatlog_hook

top :

Threads: 5 total, 0 running, 5 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 43.7 id, 0.0 wa, 0.0 hi, 56.3 si, 0.0 st %Cpu1 : 0.7 us, 0.7 sy, 0.0 ni, 40.5 id, 0.0 wa, 0.0 hi, 58.1 si, 0.0 st %Cpu2 : 1.3 us, 0.7 sy, 0.0 ni, 45.3 id, 0.0 wa, 0.0 hi, 52.7 si, 0.0 st %Cpu3 : 2.0 us, 0.7 sy, 0.0 ni, 51.7 id, 0.0 wa, 0.0 hi, 45.6 si, 0.0 st %Cpu4 : 1.4 us, 0.7 sy, 0.0 ni, 51.4 id, 0.0 wa, 0.0 hi, 46.6 si, 0.0 st %Cpu5 : 1.3 us, 0.7 sy, 0.0 ni, 49.0 id, 0.0 wa, 0.0 hi, 49.0 si, 0.0 st %Cpu6 : 1.3 us, 0.7 sy, 0.0 ni, 45.0 id, 0.0 wa, 0.0 hi, 53.0 si, 0.0 st %Cpu7 : 0.7 us, 0.7 sy, 0.7 ni, 51.0 id, 0.0 wa, 0.0 hi, 46.9 si, 0.0 st %Cpu8 : 1.3 us, 0.7 sy, 0.0 ni, 53.0 id, 0.0 wa, 0.0 hi, 45.0 si, 0.0 st %Cpu9 : 2.0 us, 0.7 sy, 0.0 ni, 52.3 id, 0.0 wa, 0.0 hi, 45.0 si, 0.0 st %Cpu10 : 0.0 us, 0.0 sy, 0.0 ni, 51.7 id, 0.0 wa, 0.0 hi, 48.3 si, 0.0 st %Cpu11 : 0.7 us, 0.0 sy, 0.7 ni, 46.9 id, 0.0 wa, 0.0 hi, 51.7 si, 0.0 st MiB Mem : 31955.6 total, 26194.9 free, 2928.9 used, 3967.1 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29026.7 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

813332 root 20 0 243.4m 34.0m 27.3 0.1 1:49.14 S ./smartdns -c smartdns.conf 813336 root 20 0 243.4m 34.0m 14.0 0.1 0:57.25 S ./smartdns -c smartdns.conf 813335 root 20 0 243.4m 34.0m 3.3 0.1 0:10.50 S ./smartdns -c smartdns.conf 813334 root 15 -5 243.4m 34.0m 0.7 0.1 0:03.20 S ./smartdns -c smartdns.conf 813333 root 20 0 243.4m 34.0m 0.0 0.1 0:00.00 S ./smartdns -c smartdns.conf

see load

Edit:

One more is there options to auto load list of domain for nftset .

like in config file to set

conf-file nftset.conf

nftset.conf :

nftset /google.com/#4:ip#filter#allows nftset /cisco.com/#4:ip#filter#allows

and after add new row :

nftset /yahoo.com/#4:ip#filter#allows

to auto load new domain

now to load this need to restart service.

pymumu commented 1 year ago

Use the latest code, delete the cache file, and add the following configuration items.

rr-ttl 600
micron10 commented 1 year ago

added:

Threads: 5 total, 0 running, 5 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.7 sy, 0.0 ni, 43.1 id, 0.7 wa, 0.0 hi, 55.6 si, 0.0 st %Cpu1 : 0.7 us, 1.4 sy, 0.0 ni, 58.2 id, 0.0 wa, 0.0 hi, 39.7 si, 0.0 st %Cpu2 : 0.7 us, 0.7 sy, 0.0 ni, 53.7 id, 0.0 wa, 0.0 hi, 44.9 si, 0.0 st %Cpu3 : 1.4 us, 0.7 sy, 0.0 ni, 50.0 id, 0.0 wa, 0.0 hi, 48.0 si, 0.0 st %Cpu4 : 1.4 us, 0.7 sy, 0.0 ni, 57.9 id, 0.0 wa, 0.0 hi, 40.0 si, 0.0 st %Cpu5 : 0.0 us, 1.4 sy, 0.0 ni, 44.2 id, 0.0 wa, 0.0 hi, 54.4 si, 0.0 st %Cpu6 : 2.0 us, 0.7 sy, 0.0 ni, 41.9 id, 0.0 wa, 0.0 hi, 55.4 si, 0.0 st %Cpu7 : 0.7 us, 0.7 sy, 0.0 ni, 54.1 id, 0.0 wa, 0.0 hi, 44.6 si, 0.0 st %Cpu8 : 1.4 us, 0.7 sy, 0.0 ni, 58.2 id, 0.0 wa, 0.0 hi, 39.7 si, 0.0 st %Cpu9 : 0.7 us, 0.7 sy, 0.0 ni, 56.2 id, 0.0 wa, 0.0 hi, 42.5 si, 0.0 st %Cpu10 : 0.0 us, 0.7 sy, 0.0 ni, 47.6 id, 0.0 wa, 0.0 hi, 51.7 si, 0.0 st %Cpu11 : 0.0 us, 0.7 sy, 0.0 ni, 49.6 id, 0.0 wa, 0.0 hi, 49.6 si, 0.0 st MiB Mem : 31955.6 total, 26204.3 free, 2911.2 used, 3975.4 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29044.3 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

866812 root 20 0 243.0m 20.0m 18.7 0.1 0:13.40 S ./smartdns -c smartdns.conf 866816 root 20 0 243.0m 20.0m 5.3 0.1 0:04.82 S ./smartdns -c smartdns.conf 866814 root 15 -5 243.0m 20.0m 1.3 0.1 0:01.03 S ./smartdns -c smartdns.conf 866815 root 20 0 243.0m 20.0m 1.3 0.1 0:00.92 S ./smartdns -c smartdns.conf 866813 root 20 0 243.0m 20.0m 0.0 0.1 0:00.00 S ./smartdns -c smartdns.conf

perf :

PerfTop: 847 irqs/sec kernel:65.9% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 866812)

10.16%  smartdns          [.] _dns_cache_insert_sorted
 6.55%  [pppoe]           [k] pppoe_rcv
 2.79%  [kernel]          [k] __dev_queue_xmit
 1.49%  [kernel]          [k] fib_table_lookup
 1.39%  [nf_tables]       [k] nft_do_chain
 1.21%  smartdns          [.] _dns_get_domain_from_packet
 1.19%  [kernel]          [k] __skb_flow_dissect
 1.17%  [ixgbe]           [k] ixgbe_clean_rx_irq
 1.15%  [kernel]          [k] __netif_receive_skb_core.constprop.0
 1.14%  libc.so.6         [.] 0x000000000016d5ca
 1.09%  [kernel]          [k] skb_clone
 0.99%  [nf_conntrack]    [k] __nf_conntrack_find_get.isra.0
 0.95%  [nf_tables]       [k] nft_rhash_lookup
 0.94%  [nf_tables]       [k] __nft_rbtree_lookup
 0.94%  [kernel]          [k] strncpy
 0.92%  [kernel]          [k] kmem_cache_free
 0.91%  [kernel]          [k] dev_queue_xmit_nit
 0.85%  [kernel]          [k] csum_partial_copy_generic
 0.82%  [ixgbe]           [k] ixgbe_tx_map
 0.74%  [kernel]          [k] memcmp
 0.73%  [kernel]          [k] __copy_skb_header
 0.69%  [kernel]          [k] nf_hook_slow
 0.67%  [kernel]          [k] kmem_cache_alloc
 0.66%  [kernel]          [k] ip_finish_output2
 0.65%  [kernel]          [k] skb_release_data
 0.62%  smartdns          [.] dict_hash
 0.61%  [kernel]          [k] __siphash_unaligned
 0.54%  [kernel]          [k] dev_hard_start_xmit
 0.54%  [ixgbe]           [k] ixgbe_xmit_frame_ring
 0.49%  [kernel]          [k] kfree_skb_reason
 0.48%  [vlan_mon]        [k] vlan_pt_recv
 0.47%  [nf_tables]       [k] nft_lookup_eval
 0.46%  [kernel]          [k] timerqueue_add
 0.46%  smartdns          [.] _dns_encode_domain
 0.45%  libc.so.6         [.] malloc
 0.44%  [kernel]          [k] _raw_spin_lock_irqsave
 0.42%  [kernel]          [k] _raw_spin_lock_bh
 0.41%  [kernel]          [k] ___slab_alloc.isra.0
 0.39%  [kernel]          [k] page_frag_free
 0.38%  [kernel]          [k] ip_rcv_core.constprop.0
 0.38%  [nf_conntrack]    [k] nf_conntrack_in
 0.38%  [kernel]          [k] __rcu_read_unlock
 0.37%  [kernel]          [k] ip_route_input_slow
 0.36%  [kernel]          [k] netdev_core_pick_tx
 0.36%  libc.so.6         [.] pthread_mutex_lock
 0.36%  [kernel]          [k] __rcu_read_lock
 0.35%  [kernel]          [k] validate_xmit_skb.isra.0
 0.34%  [kernel]          [k] kmem_cache_free_bulk.part.0
 0.33%  [kernel]          [k] __local_bh_enable_ip
 0.32%  [kernel]          [k] _raw_spin_lock
 0.32%  [kernel]          [k] ip_output
 0.32%  [kernel]          [k] __slab_free.isra.0
 0.32%  [kernel]          [k] netdev_pick_tx
 0.31%  [kernel]          [k] ep_poll
 0.31%  [kernel]          [k] get_rps_cpu
 0.31%  [kernel]          [k] process_backlog
 0.31%  [kernel]          [k] sch_direct_xmit
 0.31%  [kernel]          [k] kmem_cache_alloc_node
 0.30%  [kernel]          [k] dev_gro_receive
 0.30%  [kernel]          [k] rep_movs_alternative
 0.30%  [kernel]          [k] ip_route_output_key_hash_rcu
 0.29%  [kernel]          [k] ktime_get_with_offset
 0.29%  [kernel]          [k] __list_del_entry_valid
 0.28%  [kernel]          [k] netif_skb_features
 0.27%  smartdns          [.] dns_cache_lookup
 0.27%  [kernel]          [k] __fdget
 0.27%  [kernel]          [k] __ip_select_ident
 0.27%  [pppoe]           [k] pppoe_xmit
 0.26%  [kernel]          [k] entry_SYSCALL_64_after_hwframe
 0.26%  [vdso]            [.] __vdso_clock_gettime
 0.26%  [ppp_generic]     [k] __ppp_xmit_process
 0.24%  smartdns          [.] _dns_get_domain_offset
 0.24%  [kernel]          [k] vlan_do_receive
 0.23%  [kernel]          [k] schedule_hrtimeout_range_clock
 0.23%  [kernel]          [k] ip_forward
 0.22%  [nf_nat]          [k] l4proto_manip_pkt
 0.22%  smartdns          [.] _dns_decode_an
 0.22%  [nf_conntrack]    [k] nf_conntrack_tcp_packet
pymumu commented 1 year ago

Increase cache size.

micron10 commented 1 year ago

one more see in log :

[2023-09-01 17:18:45,781][ERROR][ dns_server.c:3467] update packet ttl failed. [2023-09-01 17:18:49,081][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:18:49,081][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:18:54,082][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:18:54,082][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:18:57,485][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:18:57,485][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:19:02,482][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:19:02,482][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:21:12,682][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:21:12,682][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:21:17,682][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:21:17,682][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:21:20,989][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:21:20,989][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:21:20,989][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:21:20,989][ERROR][ dns_server.c:3467] update packet ttl failed. [2023-09-01 17:21:24,388][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:21:24,388][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:21:29,387][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:21:29,387][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:21:32,685][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:21:32,685][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:21:37,988][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:21:37,988][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:23:40,487][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:23:40,487][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 17:23:45,487][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 17:23:45,487][ERROR][ dns_server.c:1930] update packet ttl failed.

micron10 commented 1 year ago

After increse :

cache-size 50000000 cache-persist yes cache-file /var/run/smartdns.cache cache-checkpoint-time 86400 prefetch-domain yes serve-expired yes serve-expired-ttl 259200 serve-expired-prefetch-time 21600

tcp-idle-time 120

force-qtype-SOA 65

rr-ttl 600

perf :

PerfTop: 843 irqs/sec kernel:66.9% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 868695)

 9.17%  smartdns          [.] _dns_cache_insert_sorted
 6.99%  [pppoe]           [k] pppoe_rcv
 2.61%  [kernel]          [k] __dev_queue_xmit
 1.43%  [kernel]          [k] fib_table_lookup
 1.42%  [nf_tables]       [k] nft_do_chain
 1.29%  [ixgbe]           [k] ixgbe_clean_rx_irq
 1.17%  libc.so.6         [.] 0x000000000016d5ca
 1.16%  smartdns          [.] _dns_get_domain_from_packet
 1.15%  [kernel]          [k] __netif_receive_skb_core.constprop.0
 1.09%  [kernel]          [k] strncpy
 1.06%  [kernel]          [k] __skb_flow_dissect
 1.00%  [kernel]          [k] dev_queue_xmit_nit
 0.98%  [kernel]          [k] kmem_cache_free
 0.93%  [kernel]          [k] skb_clone
 0.89%  [nf_conntrack]    [k] __nf_conntrack_find_get.isra.0
 0.88%  [nf_tables]       [k] __nft_rbtree_lookup
 0.84%  [kernel]          [k] skb_release_data
 0.81%  [ixgbe]           [k] ixgbe_tx_map

top :

Threads: 5 total, 0 running, 5 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 46.8 id, 0.0 wa, 0.0 hi, 53.2 si, 0.0 st %Cpu1 : 0.7 us, 0.0 sy, 0.0 ni, 61.0 id, 0.0 wa, 0.0 hi, 38.4 si, 0.0 st %Cpu2 : 1.4 us, 0.7 sy, 0.0 ni, 52.4 id, 0.0 wa, 0.0 hi, 45.6 si, 0.0 st %Cpu3 : 1.4 us, 0.7 sy, 0.0 ni, 52.0 id, 0.0 wa, 0.0 hi, 45.9 si, 0.0 st %Cpu4 : 1.4 us, 0.7 sy, 0.0 ni, 61.4 id, 0.0 wa, 0.0 hi, 36.6 si, 0.0 st %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni, 57.0 id, 0.0 wa, 0.0 hi, 43.0 si, 0.0 st %Cpu6 : 0.7 us, 0.0 sy, 0.7 ni, 49.0 id, 0.0 wa, 0.0 hi, 49.7 si, 0.0 st %Cpu7 : 0.7 us, 0.7 sy, 0.0 ni, 57.5 id, 0.0 wa, 0.0 hi, 41.1 si, 0.0 st %Cpu8 : 0.0 us, 0.7 sy, 0.0 ni, 55.5 id, 0.0 wa, 0.0 hi, 43.8 si, 0.0 st %Cpu9 : 0.7 us, 0.0 sy, 0.0 ni, 52.1 id, 0.0 wa, 0.0 hi, 47.3 si, 0.0 st %Cpu10 : 1.4 us, 0.0 sy, 0.0 ni, 52.8 id, 0.0 wa, 0.0 hi, 45.8 si, 0.0 st %Cpu11 : 0.7 us, 0.7 sy, 0.0 ni, 45.3 id, 0.0 wa, 0.0 hi, 53.2 si, 0.0 st MiB Mem : 31955.6 total, 26195.1 free, 2919.4 used, 3976.4 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29036.2 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

868695 root 20 0 248.9m 27.5m 21.2 0.1 0:19.32 S ./smartdns -c smartdns.conf 868699 root 20 0 248.9m 27.5m 9.3 0.1 0:07.86 S ./smartdns -c smartdns.conf 868697 root 15 -5 248.9m 27.5m 0.7 0.1 0:01.29 S ./smartdns -c smartdns.conf 868698 root 20 0 248.9m 27.5m 0.7 0.1 0:01.18 S ./smartdns -c smartdns.conf 868696 root 20 0 248.9m 27.5m 0.0 0.1 0:00.00 S ./smartdns -c smartdns.conf

pymumu commented 1 year ago 
static void _dns_cache_insert_sorted(struct dns_cache *dns_cache, struct list_head *head)
{
    time_t ttl;
    struct dns_cache *tmp = NULL;

+   list_add_tail(&dns_cache->list, &tmp->list);
+   return;

    /* ascending order */
    ttl = dns_cache->info.insert_time + dns_cache->info.ttl;
    list_for_each_entry_reverse(tmp, head, list)
    {
        if ((tmp->info.insert_time + tmp->info.ttl) <= ttl) {
            list_add(&dns_cache->list, &tmp->list);
            return;
        }
    }

    list_add(&dns_cache->list, head);
}
micron10 commented 1 year ago

after add :

[2023-09-01 17:50:59,267][FATAL][ smartdns.c:610 ] process exit with signal 11, code = 1, errno = 0, pid = 24, self = 875937, pc = 0xffffffffa0033900, addr = 0x18, build(Sep 1 2023 12:20:35 x86_64) [2023-09-01 17:50:59,267][FATAL][ util.c:1443] Stack: [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #01: 0x42b06b () from ./smartdns+0x2b06b [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #02: 0x40635d () from ./smartdns+0x635d [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #03: 0x7f0daa523d30 () from /lib64/libc.so.6+0x40d30 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #04: 0x4351ff () from ./smartdns+0x351ff [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #05: 0x435c7d () from ./smartdns+0x35c7d [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #06: 0x436113 () from ./smartdns+0x36113 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #07: 0x4166aa () from ./smartdns+0x166aa [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #08: 0x417050 () from ./smartdns+0x17050 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #09: 0x41e718 () from ./smartdns+0x1e718 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #10: 0x41f906 () from ./smartdns+0x1f906 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #11: 0x420e19 () from ./smartdns+0x20e19 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #12: 0x420fd4 () from ./smartdns+0x20fd4 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #13: 0x411710 () from ./smartdns+0x11710 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #14: 0x411d42 () from ./smartdns+0x11d42 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #15: 0x413370 () from ./smartdns+0x13370 [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #16: 0x41356e () from ./smartdns+0x1356e [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #17: 0x7f0daa575eba () from /lib64/libc.so.6+0x92eba [2023-09-01 17:50:59,267][FATAL][ util.c:1455] #18: 0x7f0daa600a88 () from /lib64/libc.so.6+0x11da88

pymumu commented 1 year ago

typo:

list_add_tail(&dns_cache->list, head);
micron10 commented 1 year ago

Great now look good :)

top :

Threads: 5 total, 0 running, 5 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.7 sy, 0.0 ni, 56.6 id, 0.0 wa, 0.0 hi, 42.6 si, 0.0 st %Cpu1 : 0.7 us, 1.4 sy, 0.0 ni, 53.4 id, 0.0 wa, 0.0 hi, 44.6 si, 0.0 st %Cpu2 : 0.7 us, 1.4 sy, 0.0 ni, 57.4 id, 0.0 wa, 0.0 hi, 40.5 si, 0.0 st %Cpu3 : 0.7 us, 0.7 sy, 0.0 ni, 58.8 id, 0.0 wa, 0.0 hi, 39.9 si, 0.0 st %Cpu4 : 1.4 us, 0.7 sy, 0.0 ni, 56.5 id, 0.0 wa, 0.0 hi, 41.5 si, 0.0 st %Cpu5 : 0.7 us, 0.7 sy, 0.0 ni, 57.1 id, 0.0 wa, 0.0 hi, 41.5 si, 0.0 st %Cpu6 : 0.7 us, 0.7 sy, 0.0 ni, 53.5 id, 0.0 wa, 0.0 hi, 45.1 si, 0.0 st %Cpu7 : 0.7 us, 0.7 sy, 0.0 ni, 57.1 id, 0.0 wa, 0.0 hi, 41.5 si, 0.0 st %Cpu8 : 0.7 us, 0.7 sy, 0.0 ni, 60.3 id, 0.0 wa, 0.0 hi, 38.4 si, 0.0 st %Cpu9 : 0.7 us, 0.7 sy, 0.0 ni, 62.2 id, 0.0 wa, 0.0 hi, 36.5 si, 0.0 st %Cpu10 : 0.0 us, 1.4 sy, 0.7 ni, 55.4 id, 0.0 wa, 0.0 hi, 42.6 si, 0.0 st %Cpu11 : 0.0 us, 0.7 sy, 0.0 ni, 56.0 id, 0.0 wa, 0.0 hi, 43.3 si, 0.0 st MiB Mem : 31955.6 total, 26186.7 free, 2922.9 used, 3981.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 29032.7 avail Mem

PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND

878665 root 20 0 250.1m 28.5m 14.0 0.1 0:12.98 S ./smartdns -c smartdns.conf 878669 root 20 0 250.1m 28.5m 3.3 0.1 0:03.75 S ./smartdns -c smartdns.conf 878667 root 15 -5 250.1m 28.5m 0.7 0.1 0:01.14 S ./smartdns -c smartdns.conf 878668 root 20 0 250.1m 28.5m 0.7 0.1 0:00.99 S ./smartdns -c smartdns.conf 878666 root 20 0 250.1m 28.5m 0.0 0.1 0:00.00 S ./smartdns -c smartdns.conf

perf :

PerfTop: 514 irqs/sec kernel:69.1% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 878665)

 6.59%  [pppoe]           [k] pppoe_rcv
 2.56%  [kernel]          [k] __dev_queue_xmit
 1.78%  libc.so.6         [.] 0x000000000016d5ca
 1.57%  [nf_tables]       [k] nft_do_chain
 1.48%  [kernel]          [k] fib_table_lookup
 1.39%  smartdns          [.] _dns_get_domain_from_packet
 1.31%  [kernel]          [k] __skb_flow_dissect
 1.22%  [ixgbe]           [k] ixgbe_clean_rx_irq
 1.19%  [nf_conntrack]    [k] __nf_conntrack_find_get.isra.0
 1.16%  [kernel]          [k] __netif_receive_skb_core.constprop.0
 1.03%  [nf_tables]       [k] nft_rhash_lookup
 1.01%  [kernel]          [k] kmem_cache_free
 0.94%  [kernel]          [k] dev_queue_xmit_nit
 0.91%  [kernel]          [k] ip_finish_output2
 0.90%  [nf_tables]       [k] __nft_rbtree_lookup
 0.80%  [kernel]          [k] __copy_skb_header
 0.79%  [kernel]          [k] strncpy
 0.75%  [kernel]          [k] csum_partial_copy_generic
 0.74%  [kernel]          [k] skb_release_data
 0.74%  smartdns          [.] _dns_encode_domain
 0.73%  [kernel]          [k] skb_clone
 0.71%  [kernel]          [k] memcmp
 0.71%  smartdns          [.] dict_hash
 0.68%  [kernel]          [k] ___slab_alloc.isra.0
 0.65%  libc.so.6         [.] malloc
 0.64%  [ixgbe]           [k] ixgbe_tx_map
 0.62%  [kernel]          [k] kmem_cache_alloc
 0.57%  [ixgbe]           [k] ixgbe_xmit_frame_ring
 0.54%  [kernel]          [k] nf_hook_slow
 0.51%  [kernel]          [k] __fdget
 0.51%  [kernel]          [k] ep_poll
 0.50%  [vlan_mon]        [k] vlan_pt_recv
 0.49%  [kernel]          [k] kfree_skb_reason
 0.46%  [kernel]          [k] ip_route_input_slow
 0.46%  [kernel]          [k] __siphash_unaligned
 0.44%  libc.so.6         [.] pthread_mutex_lock
 0.42%  [kernel]          [k] __local_bh_enable_ip
 0.42%  smartdns          [.] dns_cache_lookup
 0.42%  [nf_conntrack]    [k] nf_conntrack_in
 0.41%  [kernel]          [k] dev_hard_start_xmit
 0.39%  [kernel]          [k] _raw_spin_lock_bh
 0.39%  [kernel]          [k] timerqueue_add
 0.39%  [kernel]          [k] page_frag_free
 0.39%  [kernel]          [k] vlan_do_receive
 0.39%  [kernel]          [k] __list_del_entry_valid
 0.38%  [kernel]          [k] __rcu_read_unlock
 0.37%  smartdns          [.] _dns_get_domain_offset
 0.36%  [kernel]          [k] netdev_pick_tx
 0.35%  [kernel]          [k] netif_skb_features
 0.35%  [kernel]          [k] _raw_spin_lock
 0.35%  [kernel]          [k] sch_direct_xmit
 0.35%  [nf_tables]       [k] nft_lookup_eval
 0.35%  [kernel]          [k] _raw_spin_lock_irqsave
 0.33%  [kernel]          [k] ip_route_output_key_hash_rcu
 0.32%  [kernel]          [k] kmem_cache_free_bulk.part.0
 0.32%  [kernel]          [k] __schedule
 0.32%  [kernel]          [k] csum_partial
 0.31%  [kernel]          [k] kmem_cache_alloc_node
 0.31%  [kernel]          [k] __slab_free.isra.0
 0.31%  [kernel]          [k] validate_xmit_skb.isra.0
 0.30%  [kernel]          [k] netdev_core_pick_tx
 0.30%  [kernel]          [k] ip_output
 0.30%  [kernel]          [k] __ip_select_ident
 0.30%  [kernel]          [k] __ip_append_data
 0.29%  [kernel]          [k] __rcu_read_lock
 0.29%  libc.so.6         [.] pthread_mutex_unlock
 0.28%  [kernel]          [k] process_backlog
 0.28%  [kernel]          [k] get_rps_cpu
 0.28%  smartdns          [.] _dns_decode_an
 0.27%  [nf_nat]          [k] nf_nat_ipv4_manip_pkt
 0.26%  [kernel]          [k] rep_movs_alternative
 0.26%  [kernel]          [k] ip_rcv_core.constprop.0
 0.26%  [kernel]          [k] entry_SYSCALL_64_after_hwframe
 0.25%  smartdns          [.] _dns_server_get_answer
 0.25%  [nf_tables]       [k] nft_meta_get_eval
 0.24%  [kernel]          [k] skb_network_protocol
 0.24%  [ppp_generic]     [k] ppp_push
 0.24%  [kernel]          [k] dev_gro_receive

ok next question is for nftset is it possible to load list dynamic without restart server ,

pymumu commented 1 year ago

There are no support plans for the dynamic configuration loading feature.

In addition, the above modification will cause the prefetch feature to become invalid.

micron10 commented 1 year ago

There are no support plans for the dynamic configuration loading feature. ------> OK

In addition, the above modification will cause the prefetch feature to become invalid. -----> is there a way to do it so that there is no problem?

pymumu commented 1 year ago

Only the prefetch function is not available, other functions should be normal.

micron10 commented 1 year ago

OK Thanks for support @pymumu if you see any other issue will update you!

@PikuZheng thanks

pymumu commented 1 year ago

The prefetch function requires periodic scanning for cache TTL, which results in excessive CPU consumption One idea is to reduce CPU usage in case the user turns off prefetch function.

I will continue to track and solve this problem, maybe next release.

micron10 commented 1 year ago

One note :

when enable in config ipv6 dns server :

server 2001:4860:4860::8888

server 2001:4860:4860::8844

server 2606:4700::1111

server-tls 2606:4700::1111

server 2606:4700:4700::1001

server-tls 2606:4700:4700::1001

and process go to 100%

and perf :

PerfTop: 4315 irqs/sec kernel:95.6% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 889807)

user root 30.55% [kernel] [k] ipv6_dev_get_saddr 14.47% [kernel] [k] l3mdev_master_ifindex_rcu 6.24% [pppoe] [k] pppoe_rcv 4.61% [kernel] [k] ipv6_dev_get_saddr 1.91% [kernel] [k] dev_queue_xmit 1.46% [nf_tables] [k] nft_do_chain 1.31% [kernel] [k] netif_receive_skb_core.constprop.0 0.99% [kernel] [k] strncpy 0.96% [kernel] [k] fib_table_lookup 0.91% [ixgbe] [k] ixgbe_clean_rx_irq 0.87% [kernel] [k] dev_queue_xmit_nit 0.82% [nf_tables] [k] nft_rhash_lookup 0.81% [nf_tables] [k] nft_rbtree_lookup 0.76% [kernel] [k] csum_partial_copy_generic 0.73% [kernel] [k] skb_release_data 0.72% [kernel] [k] kmem_cache_free 0.71% [kernel] [k] skb_clone 0.62% [kernel] [k] skb_flow_dissect 0.62% [kernel] [k] memcmp 0.60% [kernel] [k] __copy_skb_header 0.58% [nf_conntrack] [k] nf_conntrack_find_get.isra.0 0.55% [ixgbe] [k] ixgbe_tx_map 0.52% [kernel] [k] kmem_cache_alloc 0.48% [kernel] [k] ip_route_input_slow 0.42% [kernel] [k] dev_hard_start_xmit 0.41% [vlan_mon] [k] vlan_pt_recv 0.41% [kernel] [k] ktime_get_with_offset 0.39% [kernel] [k] ip_finish_output2 0.39% [kernel] [k] _slab_alloc.isra.0 0.35% [ixgbe] [k] ixgbe_xmit_frame_ring 0.33% [kernel] [k] siphash_unaligned 0.32% [kernel] [k] ip_rcv_core.constprop.0 0.32% [kernel] [k] kmem_cache_free_bulk.part.0 0.32% [kernel] [k] local_bh_enable_ip 0.32% [kernel] [k] nf_hook_slow 0.29% [kernel] [k] vlan_do_receive 0.29% [nf_conntrack] [k] nf_conntrack_in 0.29% [ppp_generic] [k] ppp_xmit_process 0.29% [kernel] [k] ip_forward 0.27% [kernel] [k] kfree_skb_reason 0.27% [nf_tables] [k] nft_lookup_eval 0.27% [kernel] [k] _raw_spin_lock_irqsave 0.26% [kernel] [k] netdev_core_pick_tx 0.26% [kernel] [k] netif_skb_features 0.25% [kernel] [k] _raw_spin_lock_bh 0.25% [kernel] [k] validate_xmit_skb.isra.0 0.25% [kernel] [k] process_backlog 0.25% [kernel] [k] ip_output 0.24% [kernel] [k] sch_direct_xmit 0.24% [kernel] [k] rcu_read_unlock 0.23% [kernel] [k] page_frag_free 0.23% [kernel] [k] get_rps_cpu 0.22% [kernel] [k] skb_network_protocol 0.22% [kernel] [k] netdev_pick_tx 0.21% [nf_tables] [k] nft_meta_get_eval 0.21% [kernel] [k] read_tsc 0.21% [pppoe] [k] pppoe_xmit 0.20% [kernel] [k] list_del_entry_valid 0.20% [kernel] [k] slab_free.isra.0 0.18% [kernel] [k] _raw_spin_lock 0.18% [kernel] [k] dev_gro_receive 0.16% [ppp_generic] [k] ppp_start_xmit 0.16% libc.so.6 [.] 0x000000000016d5ca 0.15% [kernel] [k] vlan_dev_hard_start_xmit 0.15% [ixgbe] [k] ixgbe_clean_tx_irq 0.15% [kernel] [k] inet_gro_receive 0.15% [kernel] [k] native_irq_return_iret 0.15% [ppp_generic] [k] ppp_push 0.15% [kernel] [k] enqueue_to_backlog 0.15% [nf_nat] [k] nf_nat_ipv4_manip_pkt 0.15% [kernel] [k] rcu_read_lock 0.15% [ixgbe] [k] ixgbe_process_skb_fields 0.14% [nf_conntrack] [k] nf_ct_get_tuple 0.14% [kernel] [k] ip_rcv_finish_core.isra.0 0.14% [nf_nat] [k] nf_nat_ipv4_out 0.14% [nf_nat] [k] l4proto_manip_pkt 0.13% [kernel] [k] eth_header

pymumu commented 1 year ago

https://github.com/pymumu/smartdns/blob/c39a7b9b415aff8ef86d92094d0c64012704bdff/src/dns_client.c#L1887-L1889

try remove these lines.

micron10 commented 1 year ago

after remove is same

micron10 commented 1 year ago

and in log still see this :

[2023-09-01 22:50:44,449][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 22:50:44,449][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 22:52:22,724][ERROR][ dns_server.c:4840] decode cache failed, 8192, -265854011 [2023-09-01 22:54:40,848][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 22:54:40,848][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 22:54:43,948][ERROR][ dns_server.c:1866] update packet info failed. [2023-09-01 22:54:43,948][ERROR][ dns_server.c:1930] update packet ttl failed. [2023-09-01 22:56:56,749][ERROR][ dns_server.c:1866] update packet info failed.

pymumu commented 1 year ago

restart smartdns, and attach the cache file.