./dpvs

current thread affinity is set to FF EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: Invalid NUMA socket, default to 0 EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:02:00.0 (socket 0) EAL: Invalid NUMA socket, default to 0 EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:02:00.1 (socket 0) EAL: No legacy callbacks, legacy socket not created DPVS: dpvs version: 1.9-0, build on 2022.07.09.06:00:33 CFG_FILE: Opening configuration file '/etc/dpvs.conf'. CFG_FILE: log_level = WARNING NETIF: dpdk0:rx_queue_number = 8 NETIF: dpdk1:rx_queue_number = 8 NETIF: worker cpu1:dpdk0 rx_queue_id += 0 NETIF: worker cpu1:dpdk0 tx_queue_id += 0 NETIF: worker cpu1:dpdk1 rx_queue_id += 0 NETIF: worker cpu1:dpdk1 tx_queue_id += 0 NETIF: worker cpu2:dpdk0 rx_queue_id += 1 NETIF: worker cpu2:dpdk0 tx_queue_id += 1 NETIF: worker cpu2:dpdk1 rx_queue_id += 1 NETIF: worker cpu2:dpdk1 tx_queue_id += 1 NETIF: worker cpu3:dpdk0 rx_queue_id += 2 NETIF: worker cpu3:dpdk0 tx_queue_id += 2 NETIF: worker cpu3:dpdk1 rx_queue_id += 2 NETIF: worker cpu3:dpdk1 tx_queue_id += 2 NETIF: worker cpu4:dpdk0 rx_queue_id += 3 NETIF: worker cpu4:dpdk0 tx_queue_id += 3 NETIF: worker cpu4:dpdk1 rx_queue_id += 3 NETIF: worker cpu4:dpdk1 tx_queue_id += 3 NETIF: worker cpu5:dpdk0 rx_queue_id += 4 NETIF: worker cpu5:dpdk0 tx_queue_id += 4 NETIF: worker cpu5:dpdk1 rx_queue_id += 4 NETIF: worker cpu5:dpdk1 tx_queue_id += 4 NETIF: worker cpu6:dpdk0 rx_queue_id += 5 NETIF: worker cpu6:dpdk0 tx_queue_id += 5 NETIF: worker cpu6:dpdk1 rx_queue_id += 5 NETIF: worker cpu6:dpdk1 tx_queue_id += 5 NETIF: worker cpu7:dpdk0 rx_queue_id += 6 NETIF: worker cpu7:dpdk0 tx_queue_id += 6 NETIF: worker cpu7:dpdk1 rx_queue_id += 6 NETIF: worker cpu7:dpdk1 tx_queue_id += 6 SAPOOL: sapool_filter_enable = on NETIF: Ethdev port_id=0 invalid rss_hf: 0x3afbc, valid value: 0x38d34 NETIF: Ethdev port_id=0 invalid tx_offload: 0x1000e, valid value: 0x2a03f

试了flow_enable off，netif.c port_conf->rxmode.mq_mode == ETH_MQ_RX_RSS_NONE还是报错，大佬帮看看什么问题

xgfone commented 2 years ago

下面以 DPVS 1.9.0 版本为例

关于下面这两行输出，不用管它，这是 DPVS 的警告输出，也只是警告，并不影响使用。

NETIF: Ethdev port_id=0 invalid rss_hf: 0x3afbc, valid value: 0x38d34
NETIF: Ethdev port_id=0 invalid tx_offload: 0x1000e, valid value: 0x2a03f

在启动 DPDK PORT 时（对应函数 netif_port_start），会先检查并适配 RSS 和 OFFLOAD（防止后面调用 DPDK 接口做配置时失败）；而在检查和适配过程（adapt_device_conf）中，会首先通过 DPDK 接口获取网卡设备实际支持的值，如果发现和配置文件中的不一样，则将配置修改为从 DPDK 接口中获取的值，并打印一条警告日志；之后，netif_port_start函数才会调用 DPDK 接口来配置 DPDK PORT。

所以，即使在 dpvs.conf 中配置的不对，DPVS 也会纠正为正确的值（同时打印一条警告信息），并不影响后续的使用。

xgfone commented 2 years ago

只要确认网卡支持DPVS所要求的特性（也可通过 valid value: 0x38d34 中的值推算出是否支持），可以忽略上面的警告输出。

ywc689 commented 2 years ago

这条告警说明网卡不完全支持配置设定的 rss 方式（默认是all)，可以尝试通过调整配置文件中的 rss 配置消除。如果消除不了，只要确认所有 worker 上都有连接且转发正常，就可以忽略该告警。

arnohub commented 2 years ago

感谢这个问题解决了，不过又遇到了新的问题，我搭建了个测试环境，遇到了client端访问vip端口不通的情况，dpvs抓包发现直接回复了RST，rs上抓不到dpvs过来的包，并且ipvsadm -ln --stats OutPkts 没流量，确定dpvs到RS网络是通的，tcp探活没问题。拓扑及详情如下：

[root@inf2-test bin]# cat /etc/dpvs.conf !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ! This is dpvs default configuration file. ! ! The attribute "" denotes the configuration item at initialization stage. Item of ! this type is configured oneshoot and not reloadable. If invalid value configured in the ! file, dpvs would use its default value. ! ! Note that dpvs configuration file supports the following comment type: ! line comment: using '#" or '!' ! inline range comment: using '<' and '>', put comment in between !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

! global config global_defs { log_level WARNING ! log_file /var/log/dpvs.log ! log_async_mode on ! pdump off }

! netif config netif_defs {

pktpool_size 1048575 pktpool_cache 256 fdir_mode perfect device dpdk0 { rx { queue_number 1 descriptor_number 10240 rss all } tx { queue_number 1 descriptor_number 10240 } ! mtu 1500 ! promisc_mode kni_name dpdk0.kni } device dpdk1 { rx { queue_number 1 descriptor_number 10240 rss all } tx { queue_number 1 descriptor_number 10240 } ! mtu 1500 ! promisc_mode kni_name dpdk1.kni } ! bonding bond0 { ! mode 0 ! slave dpdk0 ! slave dpdk1 ! primary dpdk0 ! kni_name bond0.kni ! options dedicated_queues=off # for mode 4 only !} } ! worker config (lcores) worker_defs { worker cpu0 { type master cpu_id 0 } worker cpu1 { type slave cpu_id 1 port dpdk0 { rx_queue_ids 0 tx_queue_ids 0 ! isol_rx_cpu_ids 9 ! isol_rxq_ring_sz 1048576 } port dpdk1 { rx_queue_ids 0 tx_queue_ids 0 ! isol_rx_cpu_ids 9 ! isol_rxq_ring_sz 1048576 } } ! timer config timer_defs { # cpu job loops to schedule dpdk timer management schedule_interval 500 } ! dpvs neighbor config neigh_defs { unres_queue_length 128 timeout 60 } ! dpvs ipv4 config ipv4_defs { forwarding off default_ttl 64 fragment { bucket_number 4096 bucket_entries 16 max_entries 4096 ttl 1 } } ! dpvs ipv6 config ipv6_defs { disable off forwarding off route6 { method hlist recycle_time 10 } } ! control plane config ctrl_defs { lcore_msg { ring_size 4096 sync_msg_timeout_us 20000 priority_level low } ipc_msg { unix_domain /var/run/dpvs_ctrl } } ! ipvs config ipvs_defs { conn { conn_pool_size 2097152 conn_pool_cache 256 conn_init_timeout 3 ! expire_quiescent_template ! fast_xmit_close redirect on } udp { ! defence_udp_drop uoa_mode opp uoa_max_trail 3 timeout { normal 300 last 3 } } tcp { ! defence_tcp_drop timeout { none 2 established 90 syn_sent 3 syn_recv 30 fin_wait 7 time_wait 7 close 3 close_wait 7 last_ack 7 listen 120 synack 30 last 2 } synproxy { synack_options { mss 1452 ttl 63 sack ! wscale ! timestamp } ! defer_rs_syn rs_syn_max_retry 3 ack_storm_thresh 10 max_ack_saved 3 conn_reuse_state { close time_wait ! fin_wait ! close_wait ! last_ack } } } } ! sa_pool config sa_pool { pool_hash_size 16 flow_enable off }

arnohub commented 2 years ago

xgfone commented 2 years ago

也需要通过 dpip 命令将 IP 及其路由在 dpdk 网卡设置一下，因为 DPDK 和 Linux 是两个独立的网络栈。

arnohub commented 2 years ago

你看上面的截图，路由设置了

iqiyi / dpvs

NETIF: Ethdev port_id=0 invalid rss_hf: 0x3afbc, valid value: 0x38d34 NETIF: Ethdev port_id=0 invalid tx_offload: 0x1000e, valid value: 0x2a03f #842

./dpvs