search

iqiyi / dpvs

DPVS is a high performance Layer-4 load balancer based on DPDK.
Other
3.03k stars 729 forks source link

DPVS doesn't work with DPDK 18.11.2 #641

Closed liweitianux closed 4 years ago

liweitianux commented 4 years ago

Hello.

We've been using DPVS (v1.7.8) in FullNAT mode for more than half a year. It works quite nice. Thank you for the nice work.

Now we're testing and upgrading to the newer v1.8.4 to gain even better performance and IPv6 support. However, we're having the issue that DPVS doesn't work when compiled with DPDK 18.11.2 (with the patches applied). The output messages looks like:

current thread affinity is set to FFF
EAL: Detected 12 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:07:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
DPVS: dpvs version: 1.8-4, build on 2020.08.24.16:14:50
CFG_FILE: Opening configuration file '/etc/dpvs.conf'.
CFG_FILE: log_level = INFO
CFG_FILE: log_file = /var/log/dpvs.log
linux_set_if_mac: fail to set dpdk0.kni's MAC address: Timer expired
Adding back configs for LIP NIC (eth3:dpdk1) ...
Adding 10.252.45.70/24 ...
[dpvs_setsockopt] scoket msg connection error: Connection refused
dpip: <unknow>
linux_set_if_mac: fail to set dpdk1.kni's MAC address: Timer expired
RTNETLINK answers: Timer expired

And the related syslog messages are:

Aug 24 16:18:05 w-lvs02 kernel: igb_uio: Use MSIX interrupt by default
Aug 24 16:18:06 w-lvs02 kernel: i40e 0000:05:00.0: i40e_ptp_stop: removed PHC on eth2
Aug 24 16:18:06 w-lvs02 kernel: igb_uio 0000:05:00.0: mapping 1K dma=0xc95876000 host=ffff932995876000
Aug 24 16:18:06 w-lvs02 kernel: igb_uio 0000:05:00.0: unmapping 1K dma=0xc95876000 host=ffff932995876000
Aug 24 16:18:06 w-lvs02 kernel: i40e 0000:05:00.1: i40e_ptp_stop: removed PHC on eth3
Aug 24 16:18:06 w-lvs02 kernel: igb_uio 0000:05:00.1: mapping 1K dma=0x103bd28000 host=ffff932d3bd28000
Aug 24 16:18:06 w-lvs02 kernel: igb_uio 0000:05:00.1: unmapping 1K dma=0x103bd28000 host=ffff932d3bd28000
Aug 24 16:18:06 w-lvs02 dpvs[5110]: EAL: Probing VFIO support...
Aug 24 16:18:06 w-lvs02 dpvs[5110]: EAL: PCI device 0000:05:00.0 on NUMA socket 0
Aug 24 16:18:06 w-lvs02 dpvs[5110]: EAL:   probe driver: 8086:1572 net_i40e
Aug 24 16:18:06 w-lvs02 kernel: igb_uio 0000:05:00.0: uio device registered with irq 65
Aug 24 16:18:07 w-lvs02 dpvs[5110]: EAL: PCI device 0000:05:00.1 on NUMA socket 0
Aug 24 16:18:07 w-lvs02 dpvs[5110]: EAL:   probe driver: 8086:1572 net_i40e
Aug 24 16:18:07 w-lvs02 kernel: igb_uio 0000:05:00.1: uio device registered with irq 66
Aug 24 16:18:07 w-lvs02 dpvs[5110]: EAL: PCI device 0000:07:00.0 on NUMA socket 0
Aug 24 16:18:07 w-lvs02 dpvs[5110]: EAL:   probe driver: 8086:1521 net_e1000_igb
Aug 24 16:18:07 w-lvs02 dpvs[5110]: EAL: PCI device 0000:07:00.1 on NUMA socket 0
Aug 24 16:18:07 w-lvs02 dpvs[5110]: EAL:   probe driver: 8086:1521 net_e1000_igb
Aug 24 16:18:07 w-lvs02 dpvs[5110]: DPVS: dpvs version: 1.8-4, build on 2020.08.24.16:14:50
Aug 24 16:18:07 w-lvs02 dpvs[5110]: CFG_FILE: Opening configuration file '/etc/dpvs.conf'.
Aug 24 16:18:07 w-lvs02 dpvs[5110]: CFG_FILE: log_level = INFO
Aug 24 16:18:07 w-lvs02 dpvs[5110]: CFG_FILE: log_file = /var/log/dpvs.log
Aug 24 16:18:12 w-lvs02 kernel: rte_kni: Creating kni...
Aug 24 16:18:15 w-lvs02 kernel: rte_kni: kni_net_process_request: wait_event_interruptible timeout
Aug 24 16:18:15 w-lvs02 kernel: rte_kni: kni_net_set_mac request returns -62!
Aug 24 16:18:15 w-lvs02 kernel: rte_kni: Creating kni...
Aug 24 16:18:18 w-lvs02 kernel: rte_kni: kni_net_process_request: wait_event_interruptible timeout
Aug 24 16:18:18 w-lvs02 kernel: rte_kni: kni_net_set_mac request returns -62!
Aug 24 16:18:21 w-lvs02 kernel: rte_kni: kni_net_process_request: wait_event_interruptible timeout
Aug 24 16:18:22 w-lvs02 kernel: IPv6: ADDRCONF(NETDEV_UP): dpdk0.kni: link is not ready

In addition, the ip addr output looks like:

68: dpdk0.kni: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 3c:fd:fe:06:2e:80 brd ff:ff:ff:ff:ff:ff
    inet xx.xx.xx.xx/28 scope global dpdk0.kni
       valid_lft forever preferred_lft forever
    inet xx.xx.xx.yy/32 scope global dpdk0.kni
       valid_lft forever preferred_lft forever
    inet xx.xx.xx.zz/32 scope global dpdk0.kni
       valid_lft forever preferred_lft forever
    inet6 xx:xx:xx:xx::xx/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 xx:xx:xx:xx::yy/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 xx:xx:xx:xx::zz/128 scope global 
       valid_lft forever preferred_lft forever
69: dpdk1.kni: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3c:fd:fe:06:2e:82 brd ff:ff:ff:ff:ff:ff
    inet 10.252.45.70/24 scope global dpdk1.kni
       valid_lft forever preferred_lft forever

So there are the following issues:

This server has i40e NIC, and the old DPDK 17.11.2 (DPVS v1.7.8) cannot configure an IPv6 address on it (the DPDK interfaces).

Thank you, regards.

ywc689 commented 4 years ago
  1. rte_kni.ko has added a param "carrier". Please insmod this kmod with "carrier=on".
  2. "Kernel failed to set KNI's MAC address" can be igonred when starting dpvs.
liweitianux commented 4 years ago

Thank you for your prompt reply. Followed your suggestion to insert the rte_kni module with the carrier=on parameter, DPVS built with DPDK 18.11.2 works great!

Nevertheless, these errors/warnings still scare me a bit, such as:

(screen output)

linux_set_if_mac: fail to set dpdk0.kni's MAC address: Timer expired
[dpvs_setsockopt] scoket msg connection error: Connection refused
dpip: <unknow>
linux_set_if_mac: fail to set dpdk1.kni's MAC address: Timer expired
RTNETLINK answers: Timer expired

(in dpvs.log)

NETIF: dpdk_set_mc_list: rte_eth_dev_set_mc_addr_list is not supported, enable all multicast.
NETIF: dpdk_set_mc_list: rte_eth_dev_set_mc_addr_list is not supported, enable all multicast.
Kni: kni_mc_list_cmp_set: add mc addr: 01:00:5e:00:00:01 dpdk0 OK
Kni: kni_mc_list_cmp_set: del mc addr: 33:33:00:00:00:02 dpdk0 OK
NETIF: dpdk_set_mc_list: rte_eth_dev_set_mc_addr_list is not supported, enable all multicast.
Kni: kni_mc_list_cmp_set: del mc addr: 33:33:00:00:00:02 dpdk1 OK
NETIF: dpdk_set_mc_list: rte_eth_dev_set_mc_addr_list is not supported, enable all multicast.
Kni: kni_mc_list_cmp_set: add mc addr: 33:33:00:00:00:05 dpdk0 OK
Kni: kni_mc_list_cmp_set: add mc addr: 01:00:5e:00:00:05 dpdk0 OK
NETIF: dpdk_set_mc_list: rte_eth_dev_set_mc_addr_list is not supported, enable all multicast.
NETIF: dpdk_set_mc_list: rte_eth_dev_set_mc_addr_list is not supported, enable all multicast.
Kni: kni_mc_list_cmp_set: del mc addr: 33:33:00:00:00:05 dpdk0 OK
Kni: kni_mc_list_cmp_set: del mc addr: 01:00:5e:00:00:05 dpdk0 OK
NETIF: dpdk_set_mc_list: rte_eth_dev_set_mc_addr_list is not supported, enable all multicast.
NETIF: dpdk_set_mc_list: rte_eth_dev_set_mc_addr_list is not supported, enable all multicast.

(in /var/log/messages)

Aug 24 20:18:08 w-lvs02 kernel: rte_kni: Creating kni...
Aug 24 20:18:11 w-lvs02 kernel: rte_kni: kni_net_process_request: wait_event_interruptible timeout
Aug 24 20:18:11 w-lvs02 kernel: rte_kni: kni_net_set_mac request returns -62!
Aug 24 20:18:11 w-lvs02 kernel: rte_kni: Creating kni...
Aug 24 20:18:14 w-lvs02 kernel: rte_kni: kni_net_process_request: wait_event_interruptible timeout
Aug 24 20:18:14 w-lvs02 kernel: rte_kni: kni_net_set_mac request returns -62!
Aug 24 20:18:17 w-lvs02 kernel: rte_kni: kni_net_process_request: wait_event_interruptible timeout

It would be great if you could clarify these errors/warnings a bit, so that I can run the new DPVS/DPDK more confidently :-)

Thank you, regards.

ywc689 commented 4 years ago
  1. The warning from dpdk_set_mc_list may because your nic doesn't support multicast filter. DPVS receive all multicast packets in this case.
  2. Refer to Issues #503 for the KNI log.
liweitianux commented 4 years ago

Thank you for the kind support.