The IP on DPDK is not accessable

[b01213231]

Following the guide, everything looks fine.The dpvs is working on the background. But when I add the IP to the eth which drived by dpdk, the ip is not accessable from client , any other configurations I missed ?

when I display the dpip link , forward2kni-off is there , does that matter ? Pls help..

Here's some output for reference. [root@localhost bin]# ./dpip link show 1: dpdk0: socket 1 mtu 1500 rx-queue 16 tx-queue 16 UP 10000 Mbps full-duplex fixed-nego promisc-off forward2kni-off addr 00:E0:ED:57:08:BA OF_RX_IP_CSUM OF_TX_IP_CSUM OF_TX_TCP_CSUM OF_TX_UDP_CSUM

[root@localhost bin]# ./dpip addr show inet 10.219.100.6/22 scope global dpdk0 broadcast 10.219.103.255 valid_lft forever preferred_lft forever inet 10.219.100.7/32 scope global dpdk0 valid_lft forever preferred_lft forever sa_used 0 sa_free 1032176 sa_miss 0

[root@localhost bin]# ./dpip route show inet 10.219.100.6/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto inet 10.219.100.7/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto inet 10.219.100.0/22 via 0.0.0.0 src 10.219.100.6 dev dpdk0 mtu 1500 tos 0 scope link metric 0 proto auto inet 0.0.0.0/0 via 10.219.103.254 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope global metric 0 proto auto

[root@localhost bin]# ./ipvsadm -ln IP Virtual Server version 0.0.0 (size=0) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.219.100.6:80 rr -> 10.219.100.4:80 FullNat 1 0 0

[root@localhost ~]# curl 10.219.100.6 curl: (7) Failed connect to 10.219.100.6:80; No route to host [root@localhost ~]#

[mscbg]

It seems like the client cannot access to vip. Please do this on your client if your client is based on linux: route -n ping 10.219.100.6

[b01213231]

thanks mscbg. but ping to 10.219.100.6 is not available.

[root@localhost ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.219.100.0 0.0.0.0 255.255.252.0 U 0 0 0 eth2.227 169.254.0.0 0.0.0.0 255.255.0.0 U 1004 0 0 eth2 169.254.0.0 0.0.0.0 255.255.0.0 U 1008 0 0 eth2.227 0.0.0.0 10.219.103.254 0.0.0.0 UG 0 0 0 eth2.227 [root@localhost ~]# ping 10.219.100.6 PING 10.219.100.6 (10.219.100.6) 56(84) bytes of data. From 10.219.100.1 icmp_seq=2 Destination Host Unreachable From 10.219.100.1 icmp_seq=3 Destination Host Unreachable From 10.219.100.1 icmp_seq=4 Destination Host Unreachable From 10.219.100.1 icmp_seq=6 Destination Host Unreachable From 10.219.100.1 icmp_seq=7 Destination Host Unreachable From 10.219.100.1 icmp_seq=8 Destination Host Unreachable ^C --- 10.219.100.6 ping statistics --- 8 packets transmitted, 0 received, +6 errors, 100% packet loss, time 7848ms pipe 3 [root@localhost ~]#

[b01213231]

is it necessary add the vip to dpdk0.kni ?

[mscbg]

Your client didn‘t receive arp packet. And I guess dpdk port didn't receive arp request, either. Please make sure your dpdk port is at the same lan or vlan（10.219.100.0/22）or dpvs cannot receive any packets. Vip should not add to kni. Kni is used to resolve packet which ipvs didn't care. And we usually use kni to make 'ssh' 、keepalived healthy-check、ospf work. Finally，I find your ping echo is from 10.219.100.1，can you find this ip from dpvs？

[mscbg]

use ‘’dpip neigh show‘’ and tcpdump on your client to find more information

[b01213231]

IP 10.219.100.1 is configured under non-dpdk eth in vlan 227, the access port on TOR which connected to dpdk eth is also configured in vlan 227. I think it is in the same vlan. As you said , the packets from client 100.1 is not able to deliver to 100.6 . it's strange.
And dpip neigh shows nothing .

[mscbg]

So I find your environment is vlan. You must configure vlan in dpvs. We will tell you how to configure it later.

[ywc689]

@b01213231 You need to configure vlan interface after dpvs startup, and then set IP addresses/routes on vlan interface. Use the command below to add a vlan interface of id 227 to dpdk0. dpip vlan add dpdk0.227 link dpdk0 proto vlan id 227

[b01213231]

Thanks @mscbg @ywc689 . IP is accessable after configure the vlan tag on dpdk. But ping from client ,only one packet can return ,then it causes the dpvs down and centos crash.All the configuration reset after centos restarts, no hugepage , no igb_uio driver module, dpdk nic changes to ixgbe .Any idea about this ?

[beacer]

Any crash log or coredump output ? All configs by dpdk script, insmod, ipvsadm, and dpip are not exist after system reboot, just like ifconfig and ip command. to reserve them, pls add them to system initial script as needed.

[b01213231]

yes,I find the crash log in /var/crash , but it is too big. here's some information about the nic in crash log. [root@localhost 127.0.0.1-2017.12.28-22:48:55]# more vmcore-dmesg.txt |grep 0000:82:00.1 [ 0.995970] pci 0000:82:00.1: [8086:1528] type 00 class 0x020000 [ 0.995982] pci 0000:82:00.1: reg 0x10: [mem 0xc8000000-0xc81fffff 64bit pref] [ 0.996004] pci 0000:82:00.1: reg 0x20: [mem 0xc8400000-0xc8403fff 64bit pref] [ 0.996011] pci 0000:82:00.1: reg 0x30: [mem 0xfff80000-0xffffffff pref] [ 0.996053] pci 0000:82:00.1: PME# supported from D0 D3hot D3cold [ 0.996079] pci 0000:82:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 0.996092] pci 0000:82:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref] [ 1.026267] pci 0000:82:00.1: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window [ 1.026676] pci 0000:82:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 1.026699] pci 0000:82:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref] [ 1.026723] pci 0000:82:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 1.026728] pci 0000:82:00.1: res[7]=[mem 0x00000000-0xffffffffffffffff 64bit pref] get_res_add_size add_size 100000 [ 1.026730] pci 0000:82:00.1: res[10]=[mem 0x00000000-0xffffffffffffffff 64bit pref] get_res_add_size add_size 100000 [ 1.026734] pci 0000:82:00.1: BAR 6: assigned [mem 0xc8b00000-0xc8b7ffff pref] [ 1.026771] pci 0000:82:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 1.026773] pci 0000:82:00.1: BAR 7: assigned [mem 0xc8d80000-0xc8e7ffff 64bit pref] [ 1.026785] pci 0000:82:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref] [ 1.026787] pci 0000:82:00.1: BAR 10: assigned [mem 0xc8e80000-0xc8f7ffff 64bit pref] [ 1.369943] pci 0000:82:00.1: Signaling PME through PCIe PME interrupt [ 3.742182] ixgbe 0000:82:00.1: irq 169 for MSI/MSI-X [ 3.742194] ixgbe 0000:82:00.1: irq 170 for MSI/MSI-X [ 3.742204] ixgbe 0000:82:00.1: irq 171 for MSI/MSI-X [ 3.742221] ixgbe 0000:82:00.1: irq 172 for MSI/MSI-X [ 3.742231] ixgbe 0000:82:00.1: irq 173 for MSI/MSI-X [ 3.742241] ixgbe 0000:82:00.1: irq 174 for MSI/MSI-X [ 3.742252] ixgbe 0000:82:00.1: irq 175 for MSI/MSI-X [ 3.742262] ixgbe 0000:82:00.1: irq 176 for MSI/MSI-X [ 3.742280] ixgbe 0000:82:00.1: irq 177 for MSI/MSI-X [ 3.742291] ixgbe 0000:82:00.1: irq 178 for MSI/MSI-X [ 3.742301] ixgbe 0000:82:00.1: irq 179 for MSI/MSI-X [ 3.742318] ixgbe 0000:82:00.1: irq 180 for MSI/MSI-X [ 3.742329] ixgbe 0000:82:00.1: irq 181 for MSI/MSI-X [ 3.742339] ixgbe 0000:82:00.1: irq 182 for MSI/MSI-X [ 3.742349] ixgbe 0000:82:00.1: irq 183 for MSI/MSI-X [ 3.742359] ixgbe 0000:82:00.1: irq 184 for MSI/MSI-X [ 3.742370] ixgbe 0000:82:00.1: irq 185 for MSI/MSI-X [ 3.742381] ixgbe 0000:82:00.1: irq 186 for MSI/MSI-X [ 3.742391] ixgbe 0000:82:00.1: irq 187 for MSI/MSI-X [ 3.742407] ixgbe 0000:82:00.1: irq 188 for MSI/MSI-X [ 3.742418] ixgbe 0000:82:00.1: irq 189 for MSI/MSI-X [ 3.742428] ixgbe 0000:82:00.1: irq 190 for MSI/MSI-X [ 3.742438] ixgbe 0000:82:00.1: irq 191 for MSI/MSI-X [ 3.742449] ixgbe 0000:82:00.1: irq 192 for MSI/MSI-X [ 3.742466] ixgbe 0000:82:00.1: irq 193 for MSI/MSI-X [ 3.742476] ixgbe 0000:82:00.1: irq 194 for MSI/MSI-X [ 3.742486] ixgbe 0000:82:00.1: irq 195 for MSI/MSI-X [ 3.742503] ixgbe 0000:82:00.1: irq 196 for MSI/MSI-X [ 3.742513] ixgbe 0000:82:00.1: irq 197 for MSI/MSI-X [ 3.742523] ixgbe 0000:82:00.1: irq 198 for MSI/MSI-X [ 3.742533] ixgbe 0000:82:00.1: irq 199 for MSI/MSI-X [ 3.742543] ixgbe 0000:82:00.1: irq 200 for MSI/MSI-X [ 3.742553] ixgbe 0000:82:00.1: irq 201 for MSI/MSI-X [ 3.742563] ixgbe 0000:82:00.1: irq 202 for MSI/MSI-X [ 3.742573] ixgbe 0000:82:00.1: irq 203 for MSI/MSI-X [ 3.742589] ixgbe 0000:82:00.1: irq 204 for MSI/MSI-X [ 3.742599] ixgbe 0000:82:00.1: irq 205 for MSI/MSI-X [ 3.742610] ixgbe 0000:82:00.1: irq 206 for MSI/MSI-X [ 3.742619] ixgbe 0000:82:00.1: irq 207 for MSI/MSI-X [ 3.742629] ixgbe 0000:82:00.1: irq 208 for MSI/MSI-X [ 3.742646] ixgbe 0000:82:00.1: irq 209 for MSI/MSI-X [ 3.742750] ixgbe 0000:82:00.1: Multiqueue Enabled: Rx Queue count = 40, Tx Queue count = 40 [ 3.802791] ixgbe 0000:82:00.1: (PCI Express:5.0GT/s:Width x8) 00:e0:ed:56:ea:d1 [ 3.842818] ixgbe 0000:82:00.1: MAC: 3, PHY: 0, PBA No: 000000-000 [ 3.842822] ixgbe 0000:82:00.1: PCI Express bandwidth of 32GT/s available [ 3.842824] ixgbe 0000:82:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%) [ 4.085563] ixgbe 0000:82:00.1: Intel(R) 10 Gigabit Network Connection [ 22.543808] ixgbe 0000:82:00.1: registered PHC device on eth3 [85104.601061] ixgbe 0000:82:00.1 eth3: NIC Link is Up 10 Gbps, Flow Control: None [85580.061496] ixgbe 0000:82:00.1: removed PHC on eth3 [85580.486761] ixgbe 0000:82:00.1: complete [85580.487210] igb_uio 0000:82:00.1: irq 169 for MSI/MSI-X [85580.487301] igb_uio 0000:82:00.1: uio device registered with irq a9 [85610.667953] igb_uio 0000:82:00.1: (PCI Express:5.0GT/s:Width x8) [85610.691655] igb_uio 0000:82:00.1 (unregistered net_device): MAC: 3, PHY: 3, PBA No: 000000-000 [85610.691659] igb_uio 0000:82:00.1 (unregistered net_device): Enabled Features: RxQ: 1 TxQ: 1 [85610.705647] igb_uio 0000:82:00.1 (unregistered net_device): Intel(R) 10 Gigabit Network Connection

[beacer]

Which NIC you're using, you may found similar issue by searching DPDK and NIC. The NIC we're using is X540, and currently, we have no other NIC model on hand for test.

[b01213231]

The Nic is X540. I changed the hugepage to 1G and dpvs runs successful. I guess there was something wrong in deliverying 2M hugepages Since dpvs works ,I will close this ticket , thank you all.

[jesson1]

does the ovs cause the same problem?