search

platinasystems / go

Other
9 stars 68 forks source link

Ping between leaf to leaf sometimes does not works while bringing the links up/down during link redundancy testing #120

Open jignesh045 opened 6 years ago

jignesh045 commented 6 years ago

Issue Description: When there are two routes to same destination, ping between leaf-leaf is not working properly when we bring down one of the interface.

Environment: setup2 with below build (build created after resolving issue 109)

root@invader42:/home/jignesh# goes hget platina packages | grep version: version: dce65f7bbc20d4774ceff97e77f9633c0a26b9e9 version: f9c448619d8909ccc356e16ee372491a3f441c54 version: 60f39141fbbf78ddb2260dba74c68f2789374f18 root@invader42:/home/jignesh#

Steps:

(1) Run the following commands to execute playbook for Quagga_BGP_redundancy test case. In this test case, after establishing bgp neighborship we are bringing down (& up) one interface (that is configured to reach other leaf/spine invaders) & make sure that there should be a reachability between leaf-leaf invaders (i.e. between invader44 to invader45) via other available interface.

(a) Login to invader28 (b) cd /root/regtest/ (c) Run the command (This command will execute quagga bgp peering redundancy and won't reset the config on testbed): ansible-playbook -i hosts_testbed2 playbooks/bgp/quagga_bgp_peering_redundancy.yml -K -e reset_config=False (d) Observe the result (In my case, sometime it failed & sometime it passed)

(2) For the failed case, we manually debugged and found that due to connectivity issue (BGP neighborship is OK) it is not able to ping leaf-leaf invaders using the dummy interface. Below is the troubleshooting details.

(a) On invader45, there are two routes (via eth-3-1 & eth-5-1) to 192.168.44.1/32 (dummy0 interface of invader44)

root@invader45:/home/jignesh# ip route default via 172.17.2.1 dev eth0 10.0.1.0/24 dev eth-1-1 proto kernel scope link src 10.0.1.29 linkdown 10.0.2.0/24 dev eth-2-1 proto kernel scope link src 10.0.2.29 10.0.3.0/24 dev eth-3-1 proto kernel scope link src 10.0.3.29 10.0.4.0/24 dev eth-4-1 proto kernel scope link src 10.0.4.29 10.0.5.0/24 dev eth-5-1 proto kernel scope link src 10.0.5.29 10.0.6.0/24 dev eth-6-1 proto kernel scope link src 10.0.6.29 10.0.7.0/24 dev eth-7-1 proto kernel scope link src 10.0.7.29 10.0.8.0/24 dev eth-8-1 proto kernel scope link src 10.0.8.29 10.0.9.0/24 dev eth-9-1 proto kernel scope link src 10.0.9.29 10.0.10.0/24 dev eth-10-1 proto kernel scope link src 10.0.10.29 10.0.11.0/24 dev eth-11-1 proto kernel scope link src 10.0.11.29 10.0.12.0/24 dev eth-12-1 proto kernel scope link src 10.0.12.29 10.0.13.0/24 dev eth-13-1 proto kernel scope link src 10.0.13.29 linkdown 10.0.14.0/24 dev eth-14-1 proto kernel scope link src 10.0.14.29 10.0.15.0/24 dev eth-15-1 proto kernel scope link src 10.0.15.29 10.0.16.0/24 dev eth-16-1 proto kernel scope link src 10.0.16.29 10.0.17.0/24 dev eth-17-1 proto kernel scope link src 10.0.17.29 10.0.18.0/24 dev eth-18-1 proto kernel scope link src 10.0.18.29 10.0.19.0/24 dev eth-19-1 proto kernel scope link src 10.0.19.29 10.0.20.0/24 dev eth-20-1 proto kernel scope link src 10.0.20.29 10.0.21.0/24 dev eth-21-1 proto kernel scope link src 10.0.21.29 10.0.22.0/24 dev eth-22-1 proto kernel scope link src 10.0.22.29 10.0.23.0/24 dev eth-23-1 proto kernel scope link src 10.0.23.29 10.0.24.0/24 dev eth-24-1 proto kernel scope link src 10.0.24.29 10.0.25.0/24 dev eth-25-1 proto kernel scope link src 10.0.25.29 10.0.26.0/24 dev eth-26-1 proto kernel scope link src 10.0.26.29 10.0.27.0/24 dev eth-27-1 proto kernel scope link src 10.0.27.29 10.0.28.0/24 dev eth-28-1 proto kernel scope link src 10.0.28.29 10.0.29.0/24 dev eth-29-1 proto kernel scope link src 10.0.29.29 10.0.30.0/24 dev eth-30-1 proto kernel scope link src 10.0.30.29 10.0.31.0/24 dev eth-31-1 proto kernel scope link src 10.0.31.29 10.0.32.0/24 dev eth-32-1 proto kernel scope link src 10.0.32.29 172.17.2.0/23 dev eth0 proto kernel scope link src 172.17.2.45 172.18.0.0/16 dev docker0 proto kernel scope link src 172.18.0.1 linkdown 192.168.42.1 via 10.0.27.32 dev eth-27-1 192.168.43.1 via 10.0.5.31 dev eth-5-1 192.168.44.1 proto zebra nexthop via 10.0.3.31 dev eth-3-1 weight 1 nexthop via 10.0.5.31 dev eth-5-1 weight 1 root@invader45:/home/jignesh#

(b) It is using eth-3-1 interface to reach invader44's dummy0 interface.

root@invader45:/home/jignesh# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 172.17.2.1 0.0.0.0 UG 0 0 0 eth0 10.0.1.0 255.255.255.0 U 0 0 0 eth-1-1 10.0.2.0 255.255.255.0 U 0 0 0 eth-2-1 10.0.3.0 255.255.255.0 U 0 0 0 eth-3-1 10.0.4.0 255.255.255.0 U 0 0 0 eth-4-1 10.0.5.0 255.255.255.0 U 0 0 0 eth-5-1 10.0.6.0 255.255.255.0 U 0 0 0 eth-6-1 10.0.7.0 255.255.255.0 U 0 0 0 eth-7-1 10.0.8.0 255.255.255.0 U 0 0 0 eth-8-1 10.0.9.0 255.255.255.0 U 0 0 0 eth-9-1 10.0.10.0 255.255.255.0 U 0 0 0 eth-10-1 10.0.11.0 255.255.255.0 U 0 0 0 eth-11-1 10.0.12.0 255.255.255.0 U 0 0 0 eth-12-1 10.0.13.0 255.255.255.0 U 0 0 0 eth-13-1 10.0.14.0 255.255.255.0 U 0 0 0 eth-14-1 10.0.15.0 255.255.255.0 U 0 0 0 eth-15-1 10.0.16.0 255.255.255.0 U 0 0 0 eth-16-1 10.0.17.0 255.255.255.0 U 0 0 0 eth-17-1 10.0.18.0 255.255.255.0 U 0 0 0 eth-18-1 10.0.19.0 255.255.255.0 U 0 0 0 eth-19-1 10.0.20.0 255.255.255.0 U 0 0 0 eth-20-1 10.0.21.0 255.255.255.0 U 0 0 0 eth-21-1 10.0.22.0 255.255.255.0 U 0 0 0 eth-22-1 10.0.23.0 255.255.255.0 U 0 0 0 eth-23-1 10.0.24.0 255.255.255.0 U 0 0 0 eth-24-1 10.0.25.0 255.255.255.0 U 0 0 0 eth-25-1 10.0.26.0 255.255.255.0 U 0 0 0 eth-26-1 10.0.27.0 255.255.255.0 U 0 0 0 eth-27-1 10.0.28.0 255.255.255.0 U 0 0 0 eth-28-1 10.0.29.0 255.255.255.0 U 0 0 0 eth-29-1 10.0.30.0 255.255.255.0 U 0 0 0 eth-30-1 10.0.31.0 255.255.255.0 U 0 0 0 eth-31-1 10.0.32.0 255.255.255.0 U 0 0 0 eth-32-1 172.17.2.0 255.255.254.0 U 0 0 0 eth0 172.18.0.0 255.255.0.0 U 0 0 0 docker0 192.168.42.1 10.0.27.32 255.255.255.255 UGH 0 0 0 eth-27-1 192.168.43.1 10.0.5.31 255.255.255.255 UGH 0 0 0 eth-5-1 192.168.44.1 10.0.3.31 255.255.255.255 UGH 0 0 0 eth-3-1 root@invader45:/home/jignesh#

(c) Try to ping dummy0 interface of invader44 from invader45

root@invader45:/home/jignesh# ping -I 192.168.45.1 192.168.44.1 PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data. ^C --- 192.168.44.1 ping statistics --- 9 packets transmitted, 0 received, 100% packet loss, time 8172ms

root@invader45:/home/jignesh#

(d) Further debugging shows that it is using eth-5-1 interface to reach though the "route" command shows the exit interface to reach 192.168.44.1 is eth-3-1.

root@invader45:/home/jignesh# tcpdump -i eth-5-1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth-5-1, link-type EN10MB (Ethernet), capture size 262144 bytes 05:41:14.224775 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 14894, seq 1, length 64 05:41:15.229625 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 14894, seq 2, length 64 05:41:15.233630 LLDP, length 189: invader45 05:41:15.234576 LLDP, length 189: invader43 05:41:16.253617 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 14894, seq 3, length 64 05:41:17.277623 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 14894, seq 4, length 64 05:41:18.301625 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 14894, seq 5, length 64 05:41:19.325627 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 14894, seq 6, length 64 05:41:20.349586 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 14894, seq 7, length 64 05:41:21.377622 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 14894, seq 8, length 64 05:41:22.397632 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 14894, seq 9, length 64 05:41:27.549586 ARP, Request who-has 10.0.5.31 tell 10.0.5.29, length 28 05:41:27.550668 ARP, Reply 10.0.5.31 is-at 50:18:4c:00:0b:e0 (oui Unknown), length 50 ^C 13 packets captured 13 packets received by filter 0 packets dropped by kernel root@invader45:/home/jignesh#

(e) In our case while executing playbook, when it was showing as pass (playbook result after executing), after few minute i have manually checked the above steps (to check which interface it is using) & found that it is using different interface (as shown above) and at that time again ping was not working.

arthurxiong3 commented 6 years ago

(d) Observe the result (In my case, sometime it failed & sometime it passed)

(e) In our case while executing playbook, when it was showing as pass (playbook result after executing), after few minute i have manually checked the above steps (to check which interface it is using) & found that it is using different interface (as shown above) and at that time again ping was not working

Were routing tables before and after the ping on all the invaders(i44, i45, and the ones in between if exist) logged? See if there's any difference between passed and failed ones.

arthurxiong3 commented 6 years ago

ping sometimes not working is the symptom, what's the cause?

Also It's not clearly stated that what's wrong in the routing info.

As a comparison, a route with multiple nexthop appears the same from a debian virtualbox vm - root@pla-d1:/home/arthur# ip route | grep 192.168.67.0 -A2 192.168.67.0/24 proto zebra metric 20 nexthop via 192.168.56.1 dev enp0s8 weight 1 nexthop via 192.168.57.1 dev enp0s9 weight 1 root@pla-d1:/home/arthur# route | grep 192.168.67.0 192.168.67.0 192.168.56.1 255.255.255.0 UG 20 0 0 enp0s8

jignesh045 commented 6 years ago

Were routing tables before and after the ping on all the invaders(i44, i45, and the ones in between if exist) logged? See if there's any difference between passed and failed ones.

I have attached the logs of all four invaders of failed test case run through playbook. I have captured the output of the following commands: route, ip route, vtysh -c 'sh ip route' & vtysh -c 'sh ip route bgp' during the following steps execution via playbook:

  1. After BGP neighborship established
  2. After interface bringing down interface
  3. After interface bringing back to up state Inv42_quagga_bgp_peering_redundancy_2018041902_1749.log Inv43_quagga_bgp_peering_redundancy_2018041902_1749.log Inv44_quagga_bgp_peering_redundancy_2018041902_1749.log Inv45_quagga_bgp_peering_redundancy_2018041902_1749.log

ping sometimes not working is the symptom, what's the cause?

Also It's not clearly stated that what's wrong in the routing info.

There is nothing wrong observed with routing info but the cause is when we bring down the interface & debug at that point, it is not forwarding request based on the current route available in routing table i.e. displaying in "route" or "ip route" output (which i have explained here in earlier comment).

arthurxiong3 commented 6 years ago

from [https://github.com/platinasystems/go/files/1927931/Inv45_quagga_bgp_peering_redundancy_2018041902_1749.log] -

invader45 2018041902:18:40 route ... 10.0.3.0 255.255.255.0 U 0 0 0 eth-3-1 ... 10.0.5.0 255.255.255.0 U 0 0 0 eth-5-1 ... 192.168.44.1 10.0.3.31 255.255.255.255 UGH 0 0 0 eth-3-1

invader45 2018041902:18:40 ip route ... 10.0.3.0/24 dev eth-3-1 proto kernel scope link src 10.0.3.29 ... 10.0.5.0/24 dev eth-5-1 proto kernel scope link src 10.0.5.29 ... 192.168.44.1 proto zebra nexthop via 10.0.3.31 dev eth-3-1 weight 1 nexthop via 10.0.5.31 dev eth-5-1 weight 1

invader45 2018041902:18:40 vtysh -c 'sh ip route' ... C> 10.0.3.0/24 is directly connected, eth-3-1 ... C> 10.0.5.0/24 is directly connected, eth-5-1 ... B 192.168.44.1/32 [20/0] via 192.168.42.1, 00:00:27 S> 192.168.44.1/32 [1/0] via 10.0.3.31, eth-3-1 * via 10.0.5.31, eth-5-1 C> 192.168.45.1/32 is directly connected, dummy0 ... invader45 2018041902:18:42 ping -w 3 -c 3 -I 192.168.45.1 192.168.44.1 PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data. 64 bytes from 192.168.44.1: icmp_seq=1 ttl=63 time=0.662 ms 64 bytes from 192.168.44.1: icmp_seq=2 ttl=63 time=0.873 ms 64 bytes from 192.168.44.1: icmp_seq=3 ttl=63 time=0.949 ms

So 'ip route' and 'route' outputs are the same comparing ping passed and failed... Which interface, eth-3-1 or eth-5-1, was used when ping passed?

jignesh045 commented 6 years ago

So 'ip route' and 'route' outputs are the same comparing ping passed and failed... Which interface, eth-3-1 or eth-5-1, was used when ping passed?

Based on manually executed steps to debug -

(1) When ping pass (after BGP neighborship established), it use eth-3-1 interface as shown below.

root@invader45:/home/jignesh# ping 192.168.44.1 -I 192.168.45.1
PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data.
64 bytes from 192.168.44.1: icmp_seq=1 ttl=63 time=0.923 ms
64 bytes from 192.168.44.1: icmp_seq=2 ttl=63 time=0.858 ms
64 bytes from 192.168.44.1: icmp_seq=3 ttl=63 time=0.990 ms
64 bytes from 192.168.44.1: icmp_seq=4 ttl=63 time=1.03 ms
64 bytes from 192.168.44.1: icmp_seq=5 ttl=63 time=1.02 ms
64 bytes from 192.168.44.1: icmp_seq=6 ttl=63 time=0.989 ms
64 bytes from 192.168.44.1: icmp_seq=7 ttl=63 time=1.05 ms
64 bytes from 192.168.44.1: icmp_seq=8 ttl=63 time=1.04 ms
^C
--- 192.168.44.1 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7056ms
rtt min/avg/max/mdev = 0.858/0.990/1.059/0.073 ms
root@invader45:/home/jignesh#

root@invader45:/home/jignesh# tcpdump -i eth-3-1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth-3-1, link-type EN10MB (Ethernet), capture size 262144 bytes
02:02:32.705294 LLDP, length 189: invader45
02:02:34.175308 LLDP, length 189: invader43
02:02:34.279725 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 4163, seq 1, length 64
02:02:35.311153 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 4163, seq 2, length 64
02:02:36.331158 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 4163, seq 3, length 64
02:02:37.332323 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 4163, seq 4, length 64
02:02:38.333481 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 4163, seq 5, length 64
02:02:39.334612 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 4163, seq 6, length 64
02:02:40.335159 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 4163, seq 7, length 64
02:02:41.336315 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 4163, seq 8, length 64
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
root@invader45:/home/jignesh#

(2) Bringing down the interface eth-3-1 & eth-19-1 in invader44 & invader45

root@invader45:/home/jignesh# ifconfig eth-3-1 down
root@invader45:/home/jignesh#
root@invader45:/home/jignesh# ifconfig eth-19-1 down
root@invader45:/home/jignesh#
root@invader45:/home/jignesh# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
10.0.1.0        *               255.255.255.0   U     0      0        0 eth-1-1
10.0.2.0        *               255.255.255.0   U     0      0        0 eth-2-1
10.0.4.0        *               255.255.255.0   U     0      0        0 eth-4-1
10.0.5.0        *               255.255.255.0   U     0      0        0 eth-5-1
10.0.6.0        *               255.255.255.0   U     0      0        0 eth-6-1
10.0.7.0        *               255.255.255.0   U     0      0        0 eth-7-1
10.0.8.0        *               255.255.255.0   U     0      0        0 eth-8-1
10.0.9.0        *               255.255.255.0   U     0      0        0 eth-9-1
10.0.10.0       *               255.255.255.0   U     0      0        0 eth-10-1
10.0.11.0       *               255.255.255.0   U     0      0        0 eth-11-1
10.0.12.0       *               255.255.255.0   U     0      0        0 eth-12-1
10.0.13.0       *               255.255.255.0   U     0      0        0 eth-13-1
10.0.14.0       *               255.255.255.0   U     0      0        0 eth-14-1
10.0.15.0       *               255.255.255.0   U     0      0        0 eth-15-1
10.0.16.0       *               255.255.255.0   U     0      0        0 eth-16-1
10.0.17.0       *               255.255.255.0   U     0      0        0 eth-17-1
10.0.18.0       *               255.255.255.0   U     0      0        0 eth-18-1
10.0.20.0       *               255.255.255.0   U     0      0        0 eth-20-1
10.0.21.0       *               255.255.255.0   U     0      0        0 eth-21-1
10.0.22.0       *               255.255.255.0   U     0      0        0 eth-22-1
10.0.23.0       *               255.255.255.0   U     0      0        0 eth-23-1
10.0.24.0       *               255.255.255.0   U     0      0        0 eth-24-1
10.0.25.0       *               255.255.255.0   U     0      0        0 eth-25-1
10.0.26.0       *               255.255.255.0   U     0      0        0 eth-26-1
10.0.27.0       *               255.255.255.0   U     0      0        0 eth-27-1
10.0.28.0       *               255.255.255.0   U     0      0        0 eth-28-1
10.0.29.0       *               255.255.255.0   U     0      0        0 eth-29-1
10.0.30.0       *               255.255.255.0   U     0      0        0 eth-30-1
10.0.31.0       *               255.255.255.0   U     0      0        0 eth-31-1
10.0.32.0       *               255.255.255.0   U     0      0        0 eth-32-1
172.17.2.0      *               255.255.254.0   U     0      0        0 eth0
172.18.0.0      *               255.255.0.0     U     0      0        0 docker0
192.168.42.1    10.0.27.32      255.255.255.255 UGH   0      0        0 eth-27-1
192.168.43.1    10.0.5.31       255.255.255.255 UGH   0      0        0 eth-5-1
192.168.44.1    10.0.5.31       255.255.255.255 UGH   0      0        0 eth-5-1
root@invader45:/home/jignesh#

root@invader44:/home/jignesh# ifconfig eth-3-1 down
root@invader44:/home/jignesh# ifconfig eth-19-1 down
root@invader44:/home/jignesh#
root@invader44:/home/jignesh# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
10.0.1.0        *               255.255.255.0   U     0      0        0 eth-1-1
10.0.2.0        *               255.255.255.0   U     0      0        0 eth-2-1
10.0.4.0        *               255.255.255.0   U     0      0        0 eth-4-1
10.0.5.0        *               255.255.255.0   U     0      0        0 eth-5-1
10.0.6.0        *               255.255.255.0   U     0      0        0 eth-6-1
10.0.7.0        *               255.255.255.0   U     0      0        0 eth-7-1
10.0.8.0        *               255.255.255.0   U     0      0        0 eth-8-1
10.0.9.0        *               255.255.255.0   U     0      0        0 eth-9-1
10.0.10.0       *               255.255.255.0   U     0      0        0 eth-10-1
10.0.11.0       *               255.255.255.0   U     0      0        0 eth-11-1
10.0.12.0       *               255.255.255.0   U     0      0        0 eth-12-1
10.0.13.0       *               255.255.255.0   U     0      0        0 eth-13-1
10.0.14.0       *               255.255.255.0   U     0      0        0 eth-14-1
10.0.15.0       *               255.255.255.0   U     0      0        0 eth-15-1
10.0.16.0       *               255.255.255.0   U     0      0        0 eth-16-1
10.0.17.0       *               255.255.255.0   U     0      0        0 eth-17-1
10.0.18.0       *               255.255.255.0   U     0      0        0 eth-18-1
10.0.20.0       *               255.255.255.0   U     0      0        0 eth-20-1
10.0.21.0       *               255.255.255.0   U     0      0        0 eth-21-1
10.0.22.0       *               255.255.255.0   U     0      0        0 eth-22-1
10.0.23.0       *               255.255.255.0   U     0      0        0 eth-23-1
10.0.24.0       *               255.255.255.0   U     0      0        0 eth-24-1
10.0.25.0       *               255.255.255.0   U     0      0        0 eth-25-1
10.0.26.0       *               255.255.255.0   U     0      0        0 eth-26-1
10.0.27.0       *               255.255.255.0   U     0      0        0 eth-27-1
10.0.28.0       *               255.255.255.0   U     0      0        0 eth-28-1
10.0.29.0       *               255.255.255.0   U     0      0        0 eth-29-1
10.0.30.0       *               255.255.255.0   U     0      0        0 eth-30-1
10.0.31.0       *               255.255.255.0   U     0      0        0 eth-31-1
10.0.32.0       *               255.255.255.0   U     0      0        0 eth-32-1
172.17.2.0      *               255.255.254.0   U     0      0        0 eth0
172.18.0.0      *               255.255.0.0     U     0      0        0 docker0
192.168.42.1    10.0.11.32      255.255.255.255 UGH   0      0        0 eth-11-1
192.168.43.1    10.0.21.31      255.255.255.255 UGH   0      0        0 eth-21-1
192.168.45.1    10.0.11.32      255.255.255.255 UGH   0      0        0 eth-11-1
root@invader44:/home/jignesh#

verified the ping response & found that it is sending the request on eth-5-1 port but on the other side of invader45 i.e. on invader43, i have not seen any icmp packet received on port eth-5-1

root@invader45:/home/jignesh# ping 192.168.44.1 -I 192.168.45.1
PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data.
^C
--- 192.168.44.1 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7160ms

root@invader45:/home/jignesh#

root@invader45:/home/jignesh# tcpdump -i eth-5-1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth-5-1, link-type EN10MB (Ethernet), capture size 262144 bytes
05:17:06.183320 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 7542, seq 1, length 64
05:17:07.200287 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 7542, seq 2, length 64
05:17:08.224285 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 7542, seq 3, length 64
05:17:09.248296 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 7542, seq 4, length 64
05:17:10.272293 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 7542, seq 5, length 64
05:17:11.296289 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 7542, seq 6, length 64
05:17:12.320264 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 7542, seq 7, length 64
05:17:13.344293 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 7542, seq 8, length 64
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel
root@invader45:/home/jignesh#

root@invader43:/home/jignesh# tcpdump -i eth-5-1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth-5-1, link-type EN10MB (Ethernet), capture size 262144 bytes
05:17:02.594040 LLDP, length 189: invader45
05:17:05.082501 LLDP, length 189: invader43
05:17:14.374132 ARP, Request who-has 10.0.5.31 tell 10.0.5.29, length 50
05:17:14.374168 ARP, Reply 10.0.5.31 is-at 50:18:4c:00:0b:e0 (oui Unknown), length 28
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
root@invader43:/home/jignesh#
arthurxiong3 commented 6 years ago

tcpdump can only capture packets targeted locally. Were any ping packets actually captured by tcpdump on inv43 when ping passed from inv45 to inv44?

Could you please try 'goes hdelta' on inv43 and inv44 for each of the interface while doing the ping and compare passed vs. failed cases...

Something like the below: ping - " root@invader43:/home/arthur# ping 10.0.5.29 -c 5 PING 10.0.5.29 (10.0.5.29) 56(84) bytes of data. 64 bytes from 10.0.5.29: icmp_seq=1 ttl=64 time=1.47 ms 64 bytes from 10.0.5.29: icmp_seq=2 ttl=64 time=0.707 ms 64 bytes from 10.0.5.29: icmp_seq=3 ttl=64 time=0.894 ms 64 bytes from 10.0.5.29: icmp_seq=4 ttl=64 time=0.988 ms 64 bytes from 10.0.5.29: icmp_seq=5 ttl=64 time=1.03 ms

--- 10.0.5.29 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4034ms rtt min/avg/max/mdev = 0.707/1.019/1.471/0.254 ms root@invader43:/home/arthur# ping 10.0.3.29 -c 5 PING 10.0.3.29 (10.0.3.29) 56(84) bytes of data. 64 bytes from 10.0.3.29: icmp_seq=1 ttl=64 time=1.25 ms 64 bytes from 10.0.3.29: icmp_seq=2 ttl=64 time=1.04 ms 64 bytes from 10.0.3.29: icmp_seq=3 ttl=64 time=1.01 ms 64 bytes from 10.0.3.29: icmp_seq=4 ttl=64 time=0.972 ms 64 bytes from 10.0.3.29: icmp_seq=5 ttl=64 time=0.905 ms

--- 10.0.3.29 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4004ms rtt min/avg/max/mdev = 0.905/1.038/1.255/0.121 ms "

1st 'goes hdelta' - " root@invader45:/home/arthur# goes hdelta | egrep eth-3-1.port_rx_packets vnet.eth-3-1.port_rx_packets: 2 (0/s) vnet.eth-3-1.port_rx_packets: 1 (1/s) vnet.eth-3-1.port_rx_packets: 1 (1/s) vnet.eth-3-1.port_rx_packets: 1 (1/s) vnet.eth-3-1.port_rx_packets: 1 (1/s) vnet.eth-3-1.port_rx_packets: 1 (1/s) "

2nd 'goes hdelta' from another ssh session - " root@invader45:/home/arthur# goes hdelta | egrep eth-5-1.port_rx_packets vnet.eth-5-1.port_rx_packets: 1 (1/s) vnet.eth-5-1.port_rx_packets: 1 (1/s) vnet.eth-5-1.port_rx_packets: 1 (1/s) vnet.eth-5-1.port_rx_packets: 1 (1/s) vnet.eth-5-1.port_rx_packets: 1 (1/s) vnet.eth-5-1.port_rx_packets: 1 (0/s) vnet.eth-5-1.port_rx_packets: 1 (0/s) "

Note: There might be more rx packets than the ping counts which is normal since there can be other rx packets. The point is to watch if rx packets coming in is at about the same speed as the ping going, or not - to confirm if ping request received, or not.

Then after the above steps are done, run 'goes hdelta' and 'egrep' tx_packets on the same ports, and then ping again...

jignesh045 commented 6 years ago

Below is the output of manually executed commands:

(1) Executed command after BGP neighborship established. ping response is not working (it should working). Also captured Inv43 tx, rx packets at the same time.

root@invader45:/home/jignesh# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
...
192.168.42.1    10.0.19.32      255.255.255.255 UGH   0      0        0 eth-19-1
192.168.43.1    10.0.3.31       255.255.255.255 UGH   0      0        0 eth-3-1
192.168.44.1    10.0.3.31       255.255.255.255 UGH   0      0        0 eth-3-1
root@invader45:/home/jignesh#

root@invader45:/home/jignesh# ip route
default via 172.17.2.1 dev eth0
...
192.168.42.1 proto zebra
        nexthop via 10.0.19.32  dev eth-19-1 weight 1
        nexthop via 10.0.27.32  dev eth-27-1 weight 1
192.168.43.1 proto zebra
        nexthop via 10.0.3.31  dev eth-3-1 weight 1
        nexthop via 10.0.5.31  dev eth-5-1 weight 1
192.168.44.1 proto zebra
        nexthop via 10.0.3.31  dev eth-3-1 weight 1
        nexthop via 10.0.5.31  dev eth-5-1 weight 1
root@invader45:/home/jignesh#

root@invader45:/home/jignesh# ping -I 192.168.45.1 192.168.44.1 -c 5
PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data.

--- 192.168.44.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4101ms

root@invader45:/home/jignesh#

root@invader45:/home/jignesh# tcpdump -i eth-3-1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth-3-1, link-type EN10MB (Ethernet), capture size 262144 bytes
05:02:59.966614 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3842, seq 1, length 64
05:03:00.995833 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3842, seq 2, length 64
05:03:02.019843 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3842, seq 3, length 64
05:03:03.043844 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3842, seq 4, length 64
05:03:03.875112 LLDP, length 189: invader45
05:03:04.067847 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3842, seq 5, length 64
05:03:07.149841 LLDP, length 189: invader43
^C
7 packets captured
7 packets received by filter
0 packets dropped by kernel
root@invader45:/home/jignesh#

root@invader43:/home/jignesh# goes hdelta | egrep eth-3-1.port_rx_packets
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 2 (2/s)
^C
root@invader43:/home/jignesh#

root@invader43:/home/jignesh# goes hdelta | egrep eth-3-1.port_tx_packets
^C
root@invader43:/home/jignesh#

(2) Executed commands after bringing down interface eth-3-1 & eth-19-1 in invader44 & invader45. 

root@invader45:/home/jignesh# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
...
172.18.0.0      *               255.255.0.0     U     0      0        0 docker0
192.168.42.1    10.0.27.32      255.255.255.255 UGH   0      0        0 eth-27-1
192.168.43.1    10.0.5.31       255.255.255.255 UGH   0      0        0 eth-5-1
192.168.44.1    10.0.5.31       255.255.255.255 UGH   0      0        0 eth-5-1
root@invader45:/home/jignesh#

root@invader45:/home/jignesh# ip route
default via 172.17.2.1 dev eth0
...
192.168.42.1 via 10.0.27.32 dev eth-27-1 proto zebra
192.168.43.1 via 10.0.5.31 dev eth-5-1 proto zebra
192.168.44.1 via 10.0.5.31 dev eth-5-1 proto zebra
root@invader45:/home/jignesh#

root@invader45:/home/jignesh# ping -I 192.168.45.1 192.168.44.1 -c 5
PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data.

--- 192.168.44.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4098ms

root@invader45:/home/jignesh#

root@invader45:/home/jignesh# tcpdump -i eth-5-1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth-5-1, link-type EN10MB (Ethernet), capture size 262144 bytes
05:21:22.913626 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3860, seq 1, length 64
05:21:23.939853 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3860, seq 2, length 64
05:21:24.967865 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3860, seq 3, length 64
05:21:25.991822 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3860, seq 4, length 64
05:21:27.011848 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 3860, seq 5, length 64
05:21:28.163803 ARP, Request who-has 10.0.5.31 tell 10.0.5.29, length 28
05:21:28.164825 ARP, Reply 10.0.5.31 is-at 50:18:4c:00:0b:e0 (oui Unknown), length 50
05:21:33.949004 LLDP, length 189: invader45
05:21:36.365481 IP 192.168.45.1.34509 > 192.168.43.1.bgp: Flags [P.], seq 2246034959:2246034978, ack 2307079049, win 108, options [nop,nop,TS val 3472201591 ecr 547395709], length 19: BGP
05:21:36.366884 IP 192.168.43.1.bgp > 192.168.45.1.34509: Flags [P.], seq 1:20, ack 19, win 108, options [nop,nop,TS val 547455708 ecr 3472201591], length 19: BGP
05:21:36.366940 IP 192.168.45.1.34509 > 192.168.43.1.bgp: Flags [.], ack 20, win 108, options [nop,nop,TS val 3472201593 ecr 547455708], length 0
05:21:37.157867 LLDP, length 189: invader43
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel
root@invader45:/home/jignesh#

root@invader43:/home/jignesh# goes hdelta | egrep eth-5-1.port_rx_packets
vnet.eth-5-1.port_rx_packets: 1 (1/s)
vnet.eth-5-1.port_rx_packets: 1 (1/s)
vnet.eth-5-1.port_rx_packets: 1 (1/s)
vnet.eth-5-1.port_rx_packets: 1 (1/s)
vnet.eth-5-1.port_rx_packets: 1 (1/s)
vnet.eth-5-1.port_rx_packets: 1 (0/s)
vnet.eth-5-1.port_rx_packets: 2 (1/s)
vnet.eth-5-1.port_rx_packets: 1 (0/s)
^C
root@invader43:/home/jignesh#

root@invader43:/home/jignesh# goes hdelta | egrep eth-5-1.port_tx_packets
vnet.eth-5-1.port_tx_packets: 1 (0/s)
vnet.eth-5-1.port_tx_packets: 1 (1/s)
vnet.eth-5-1.port_tx_packets: 1 (0/s)
^C
root@invader43:/home/jignesh#

(3) After bringing up the both interface, below is further investigation. Some time got the ping response & sometime there is ping loss & no response. In this case eth-3-1 is used to send packet to 192.168.44.1

(A) when there is ping response:

root@invader45:/home/jignesh# ping -I 192.168.45.1 192.168.44.1 -c 10
PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data.
64 bytes from 192.168.44.1: icmp_seq=1 ttl=63 time=0.943 ms
64 bytes from 192.168.44.1: icmp_seq=2 ttl=63 time=0.859 ms
64 bytes from 192.168.44.1: icmp_seq=3 ttl=63 time=1.01 ms
64 bytes from 192.168.44.1: icmp_seq=4 ttl=63 time=1.06 ms
64 bytes from 192.168.44.1: icmp_seq=5 ttl=63 time=0.993 ms
64 bytes from 192.168.44.1: icmp_seq=6 ttl=63 time=1.12 ms
64 bytes from 192.168.44.1: icmp_seq=7 ttl=63 time=1.01 ms
64 bytes from 192.168.44.1: icmp_seq=8 ttl=63 time=0.965 ms
64 bytes from 192.168.44.1: icmp_seq=9 ttl=63 time=1.02 ms
64 bytes from 192.168.44.1: icmp_seq=10 ttl=63 time=1.00 ms

--- 192.168.44.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9052ms
rtt min/avg/max/mdev = 0.859/1.000/1.121/0.077 ms
root@invader45:/home/jignesh#

root@invader43:/home/jignesh# goes hdelta | egrep eth-3-1
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (104/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (100/s)
vnet.eth-3-1.rx_pipe_ip4_l3_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_ip4_l3_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_ip4_l3_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_ip4_l3_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_ip4_l3_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
^C
root@invader43:/home/jignesh#

(B) when there is a ping loss:

root@invader45:/home/jignesh# ping -I 192.168.45.1 192.168.44.1 -c 5
PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data.
64 bytes from 192.168.44.1: icmp_seq=1 ttl=63 time=0.933 ms
64 bytes from 192.168.44.1: icmp_seq=2 ttl=63 time=1.01 ms

--- 192.168.44.1 ping statistics ---
5 packets transmitted, 2 received, 60% packet loss, time 4037ms
rtt min/avg/max/mdev = 0.933/0.973/1.014/0.051 ms
root@invader45:/home/jignesh#

root@invader43:/home/jignesh# goes hdelta | egrep eth-3-1
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_ip4_l3_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_ip4_l3_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_ip4_l3_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_ip4_l3_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
^C
root@invader43:/home/jignesh#

(C) When there is no ping response:

root@invader45:/home/jignesh# ping -I 192.168.45.1 192.168.44.1 -c 5
PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data.

--- 192.168.44.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4102ms

root@invader45:/home/jignesh#

root@invader43:/home/jignesh# goes hdelta | egrep eth-3-1
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_zero_port_bitmap_drops: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_zero_port_bitmap_drops: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_zero_port_bitmap_drops: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (1/s)
vnet.eth-3-1.port_rx_bytes: 102 (102/s)
vnet.eth-3-1.port_rx_65_to_127_byte_packets: 1 (1/s)
vnet.eth-3-1.port_rx_good_packets: 1 (1/s)
vnet.eth-3-1.port_rx_unicast_packets: 1 (1/s)
vnet.eth-3-1.port_rx_promiscuous_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 98 (98/s)
vnet.eth-3-1.rx_pipe_unicast_packets: 1 (1/s)
vnet.eth-3-1.rx_pipe_zero_port_bitmap_drops: 1 (1/s)
vnet.eth-3-1.port_rx_packets: 1 (0/s)
vnet.eth-3-1.port_rx_bytes: 207 (23/s)
vnet.eth-3-1.port_rx_good_packets: 1 (0/s)
vnet.eth-3-1.rx_pipe_port_table_packets: 1 (0/s)
vnet.eth-3-1.rx_pipe_port_table_bytes: 203 (23/s)
vnet.eth-3-1.tx_pipe_unicast_queue_cos0_packets: 1 (0/s)
vnet.eth-3-1.tx_pipe_unicast_queue_cos0_bytes: 203 (16/s)
vnet.eth-3-1.tx_pipe_port_table_packets: 1 (0/s)
vnet.eth-3-1.tx_pipe_port_table_bytes: 203 (16/s)
vnet.eth-3-1.port_tx_packets: 1 (0/s)
vnet.eth-3-1.port_tx_bytes: 207 (16/s)
vnet.eth-3-1.port_tx_good_packets: 1 (0/s)
vnet.eth-3-1.rx_packets: 1 (0/s)
vnet.eth-3-1.rx_bytes: 203 (16/s)
vnet.eth-3-1.punts: 1 (0/s)
^C
root@invader43:/home/jignesh#

root@invader43:/home/jignesh# goes hdelta | egrep drop
vnet.eth-3-1.rx_pipe_zero_port_bitmap_drops: 1 (1/s)
vnet.eth-3-1.rx_pipe_zero_port_bitmap_drops: 1 (1/s)
vnet.eth-3-1.rx_pipe_zero_port_bitmap_drops: 1 (1/s)
vnet.eth-3-1.rx_pipe_zero_port_bitmap_drops: 1 (1/s)
^C
root@invader43:/home/jignesh#
arthurxiong3 commented 6 years ago

(1) Executed command after BGP neighborship established. ping response is not working (it should working).

Ping not working before interface brought down... Is this a new issue?

jignesh045 commented 6 years ago

Ping not working before interface brought down... Is this a new issue?

I can not say it is new issue. It has random behavior (at any point of test execution) that i have observed (during manual execution) as you can see the point (3) in last comment after interface back to up, sometime ping is working/breaking/not working (while in playbook execution at point (3) it was passed, see the previous attached logs for invader45) that needs to be debug from development prospective.

arthurxiong3 commented 6 years ago

It has random behavior

Could you update the title to reflect this? Also find out which latest version this issue isn't reproducible on?

jignesh045 commented 6 years ago

Also find out which latest version this issue isn't reproducible on?

This is the latest build (after resolving issue-109) in which we have executed regression & found this issue.

jignesh045 commented 6 years ago

Issue verification status: Inconsistent

Issue status: Sometimes the ping between invaders works (after executing playbook & verifying ping status) and sometimes it failed

Network Diagram: bgp peering_redundancy

Issue verify in below build (in all setup2 invaders)

root@invader45:~# goes version
github.com/platinasystems/go: v0.41-244-g49d9e6da
github.com/platinasystems/fe1: v0.2-520-g85494c1
github.com/platinasystems/firmware-fe1a: v0.2
root@invader45:~#

root@invader45:~# dpkg --list | grep kernel
ii  kmod                           18-3                             amd64        tools for managing Linux kernel modules
ii  libdrm2:amd64                  2.4.58-2                         amd64        Userspace interface to kernel DRM services -- runtime
ii  linux-image-4.13.0-platina-mk1 4.13-117-g00814e4e1a1e           amd64        Linux kernel, version 4.13.0-platina-mk1
ii  rsyslog                        8.4.2-1+deb8u2                   amd64        reliable system and kernel logging daemon
root@invader45:~#

Below is execution (through automation) and verification steps (manually after execution):

Steps:

  1. Executed the playbook from invader28 using the command: ansible-playbook -i hosts_testbed2 playbooks/bgp/quagga_bgp_peering_redundancy.yml -K -e reset_config=False

Got the below execution summary

TASK [Print failure summary] ****************************************************************************************************************************************************************************
ok: [server_emulator] => {
    "msg": "Failure summary: \nFrom switch invader45 neighbor ip 192.168.44.1 is not getting pinged\nFrom switch invader45 neighbor ip 192.168.44.1 is not getting pinged\nFrom switch invader45 neighbor ip 192.168.44.1 is not getting pinged\n\nFrom switch invader44 neighbor ip 192.168.45.1 is not getting pinged\nFrom switch invader44 neighbor ip 192.168.45.1 is not getting pinged\nFrom switch invader44 neighbor ip 192.168.45.1 is not getting pinged\n\n\n\n"
}
  1. Observe the route from all invaders (only displaying required route)
root@invader44:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
...
...
192.168.42.1    10.0.3.32       255.255.255.255 UGH   0      0        0 eth-3-1
192.168.43.1    10.0.19.31      255.255.255.255 UGH   0      0        0 eth-19-1
192.168.45.1    10.0.19.31      255.255.255.255 UGH   0      0        0 eth-19-1
root@invader44:~#

root@invader45:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
...
...
192.168.42.1    10.0.19.32      255.255.255.255 UGH   0      0        0 eth-19-1
192.168.43.1    10.0.3.31       255.255.255.255 UGH   0      0        0 eth-3-1
192.168.44.1    10.0.3.31       255.255.255.255 UGH   0      0        0 eth-3-1
root@invader45:~#

root@invader42:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
...
...
192.168.43.1    10.0.19.29      255.255.255.255 UGH   0      0        0 eth-19-1
192.168.44.1    10.0.3.30       255.255.255.255 UGH   0      0        0 eth-3-1
192.168.45.1    10.0.19.29      255.255.255.255 UGH   0      0        0 eth-19-1
root@invader42:~#

root@invader43:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
...
...
192.168.42.1    10.0.3.29       255.255.255.255 UGH   0      0        0 eth-3-1
192.168.44.1    10.0.19.30      255.255.255.255 UGH   0      0        0 eth-19-1
192.168.45.1    10.0.3.29       255.255.255.255 UGH   0      0        0 eth-3-1
root@invader43:~#
  1. Manually initiated ping from invader45 to invader44
root@invader45:~# ping 192.168.44.1 -I 192.168.45.1
PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data.
^C
--- 192.168.44.1 ping statistics ---
1077 packets transmitted, 0 received, 100% packet loss, time 1101823ms

root@invader45:~#

After executing ping, checked that it is using eth-3-1 interface as exit interface to reach destination.

root@invader45:~# tcpdump -i eth-3-1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth-3-1, link-type EN10MB (Ethernet), capture size 262144 bytes
22:49:57.880715 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 1, length 64
22:49:58.903818 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 2, length 64
22:49:59.927824 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 3, length 64
22:50:00.951815 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 4, length 64
22:50:01.975814 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 5, length 64
22:50:02.935769 ARP, Request who-has 10.0.3.31 tell 10.0.3.29, length 28
22:50:03.003814 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 6, length 64
22:50:03.213839 ARP, Reply 10.0.3.31 is-at 50:18:4c:00:0b:d8 (oui Unknown), length 50
22:50:04.023819 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 7, length 64
22:50:05.047817 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 8, length 64
22:50:06.071831 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 9, length 64
22:50:07.095814 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 10, length 64
22:50:08.119772 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 11, length 64
22:50:09.143817 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 12, length 64
22:50:10.167811 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 13, length 64
22:50:11.191816 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 14, length 64
22:50:12.215817 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 15, length 64
22:50:13.239822 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 16, length 64
22:50:14.263823 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 17, length 64
22:50:15.291770 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 18, length 64
22:50:16.315778 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 19, length 64
22:50:17.335815 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 20, length 64
22:50:18.359820 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 21, length 64
22:50:19.383821 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 22, length 64
22:50:20.407810 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 23, length 64
22:50:21.431810 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 24, length 64
22:50:22.455819 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 25, length 64
22:50:23.479821 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 26, length 64
22:50:24.503825 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 27, length 64
22:50:25.527813 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 28, length 64
22:50:26.551816 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 29, length 64
22:50:27.575815 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 30, length 64
22:50:28.599743 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 31, length 64
22:50:29.623820 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 32, length 64
22:50:30.647818 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 33, length 64
22:50:31.607716 ARP, Request who-has 10.0.3.31 tell 10.0.3.29, length 28
22:50:31.675775 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 34, length 64
22:50:31.783762 ARP, Reply 10.0.3.31 is-at 50:18:4c:00:0b:d8 (oui Unknown), length 50
22:50:32.695778 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 35, length 64
22:50:33.719780 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 36, length 64
22:50:34.743819 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 37, length 64
22:50:35.767819 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 38, length 64
22:50:36.795783 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 39, length 64
22:50:37.815814 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 40, length 64
22:50:37.860070 IP 192.168.43.1.46103 > 192.168.45.1.bgp: Flags [.], ack 3893303565, win 115, options [nop,nop,TS val 32874736 ecr 1287915312], length 0
22:50:37.860120 IP 192.168.43.1.46103 > 192.168.45.1.bgp: Flags [P.], seq 0:19, ack 1, win 115, options [nop,nop,TS val 32874736 ecr 1287915312], length 19: BGP
22:50:38.839823 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 9740, seq 41, length 64
...
...
23:13:43.021961 ARP, Request who-has 10.0.3.29 tell 10.0.3.31, length 50
23:13:43.021974 ARP, Reply 10.0.3.29 is-at 50:18:4c:00:15:a4 (oui Unknown), length 28
23:14:37.908056 IP 192.168.43.1.46103 > 192.168.45.1.bgp: Flags [.], ack 457, win 115, options [nop,nop,TS val 34314738 ecr 1289355315], length 0
23:14:37.908102 IP 192.168.43.1.46103 > 192.168.45.1.bgp: Flags [P.], seq 456:475, ack 457, win 115, options [nop,nop,TS val 34314738 ecr 1289355315], length 19: BGP
23:14:42.926007 ARP, Request who-has 10.0.3.29 tell 10.0.3.31, length 50
23:14:42.926034 ARP, Reply 10.0.3.29 is-at 50:18:4c:00:15:a4 (oui Unknown), length 28
^C
1249 packets captured
1249 packets received by filter
0 packets dropped by kernel
root@invader45:~#

On invader43, checked whether request is received or not on interface eth-3-1 but did not found any icmp packet.

root@invader43:~# tcpdump -i eth-3-1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth-3-1, link-type EN10MB (Ethernet), capture size 262144 bytes
22:50:31.605716 ARP, Request who-has 10.0.3.31 tell 10.0.3.29, length 50
22:50:31.605742 ARP, Reply 10.0.3.31 is-at 50:18:4c:00:0b:d8 (oui Unknown), length 28
22:50:37.857643 IP 192.168.43.1.46103 > 192.168.45.1.bgp: Flags [.], ack 3893303565, win 115, options [nop,nop,TS val 32874736 ecr 1287915312], length 0
22:50:37.857749 IP 192.168.43.1.46103 > 192.168.45.1.bgp: Flags [P.], seq 0:19, ack 1, win 115, options [nop,nop,TS val 32874736 ecr 1287915312], length 19: BGP
22:50:42.923230 ARP, Request who-has 10.0.3.29 tell 10.0.3.31, length 28
22:50:43.167227 ARP, Reply 10.0.3.29 is-at 50:18:4c:00:15:a4 (oui Unknown), length 50
...
...
23:02:26.647245 ARP, Request who-has 10.0.3.31 tell 10.0.3.29, length 50
23:02:26.647274 ARP, Reply 10.0.3.31 is-at 50:18:4c:00:0b:d8 (oui Unknown), length 28
23:02:37.881381 IP 192.168.43.1.46103 > 192.168.45.1.bgp: Flags [.], ack 229, win 115, options [nop,nop,TS val 33594737 ecr 1288635314], length 0
23:02:37.881501 IP 192.168.43.1.46103 > 192.168.45.1.bgp: Flags [P.], seq 228:247, ack 229, win 115, options [nop,nop,TS val 33594737 ecr 1288635314], length 19: BGP
23:02:43.051221 ARP, Request who-has 10.0.3.29 tell 10.0.3.31, length 28
23:02:43.051695 ARP, Reply 10.0.3.29 is-at 50:18:4c:00:15:a4 (oui Unknown), length 50
^C
96 packets captured
96 packets received by filter
0 packets dropped by kernel
root@invader43:~#

Checked below command output and found that it is showing drop packet on eth-21-1

root@invader43:/home/jignesh# goes hdelta | egrep eth-3-1
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 2 (2/s)
vnet.eth-3-1.rx-bytes: 170 (170/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 2 (2/s)
vnet.eth-3-1.port-rx-good-packets: 2 (2/s)
vnet.eth-3-1.port-rx-unicast-packets: 2 (2/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 2 (2/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 2 (2/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 162 (162/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 2 (2/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.tx-pipe-port-table-packets: 2 (0/s)
vnet.eth-3-1.tx-pipe-port-table-bytes: 151 (14/s)
vnet.eth-3-1.tx-packets: 2 (0/s)
vnet.eth-3-1.tx-bytes: 159 (14/s)
vnet.eth-3-1.port-tx-65-to-127-byte-packets: 2 (0/s)
vnet.eth-3-1.port-tx-good-packets: 2 (0/s)
vnet.eth-3-1.port-tx-unicast-packets: 2 (0/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 2 (2/s)
vnet.eth-3-1.rx-bytes: 170 (170/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 2 (2/s)
vnet.eth-3-1.port-rx-good-packets: 2 (2/s)
vnet.eth-3-1.port-rx-unicast-packets: 2 (2/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 2 (2/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 2 (2/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 162 (162/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 2 (2/s)
vnet.eth-3-1.tx-pipe-unicast-queue-cos0-packets: 1 (0/s)
vnet.eth-3-1.tx-pipe-unicast-queue-cos0-bytes: 64 (13/s)
vnet.eth-3-1.tx-pipe-port-table-packets: 1 (0/s)
vnet.eth-3-1.tx-pipe-port-table-bytes: 64 (13/s)
vnet.eth-3-1.tx-packets: 1 (0/s)
vnet.eth-3-1.tx-bytes: 68 (14/s)
vnet.eth-3-1.port-tx-65-to-127-byte-packets: 1 (0/s)
vnet.eth-3-1.port-tx-good-packets: 1 (0/s)
vnet.eth-3-1.port-tx-unicast-packets: 1 (0/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
vnet.eth-3-1.rx-packets: 1 (1/s)
vnet.eth-3-1.rx-bytes: 102 (102/s)
vnet.eth-3-1.port-rx-65-to-127-byte-packets: 1 (1/s)
vnet.eth-3-1.port-rx-good-packets: 1 (1/s)
vnet.eth-3-1.port-rx-unicast-packets: 1 (1/s)
vnet.eth-3-1.port-rx-promiscuous-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-port-table-bytes: 98 (98/s)
vnet.eth-3-1.rx-pipe-ip4-l3-packets: 1 (1/s)
vnet.eth-3-1.rx-pipe-unicast-packets: 1 (1/s)
^C
root@invader43:/home/jignesh# goes hdelta | egrep drop
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (0/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
vnet.eth-21-1.rx-pipe-zero-port-bitmap-drops: 1 (1/s)
^C
root@invader43:/home/jignesh#

Do not know why it is not showing icmp packet which is receiving on interface eth-3-1 in invader43 & why it is trying (?) to forward the icmp request through eth-21-1(route command output shows the exit path for 192.168.44.1 is eth-19-1 which is UP) & showing dropped packet on eth-21-1 in invader43.

Attachment: Network Diagram & output of test case execution on all invaders invader43_quagga_bgp_peering_redundancy_20180719_224719.log invader42_quagga_bgp_peering_redundancy_20180719_224719.log invader45_quagga_bgp_peering_redundancy_20180719_224719.log invader44_quagga_bgp_peering_redundancy_20180719_224719.log

fszyang commented 6 years ago

Please retest with commit 39621febc05e0e0387d5f79ac7822b51db95ed4d If still issue please leave a setup in that state so I can debug

jignesh045 commented 6 years ago

Issue verification status: Verified

Issue status: Passed (need to debug certain points)

root@invader45:~# goes version
github.com/platinasystems/go: v0.41-313-g96fcd2e1
github.com/platinasystems/fe1: v0.2-526-gaa6b634
github.com/platinasystems/firmware-fe1a: v0.2
root@invader45:~#

root@invader45:~# goes hget platina-mk1 packages | grep version:
    version: 96fcd2e13b9d751a95eda8faa60e4974e8143765
    version: aa6b634a0548a66703fd379e26677c12263f752a
    version: 60f39141fbbf78ddb2260dba74c68f2789374f18
root@invader45:~#

root@invader45:~# dpkg --list | grep kernel
ii  kmod                           18-3                             amd64        tools for managing Linux kernel modules
ii  libdrm2:amd64                  2.4.58-2                         amd64        Userspace interface to kernel DRM services -- runtime
ii  linux-image-4.13.0-platina-mk1 4.13-129-g34f08d37ef9c           amd64        Linux kernel, version 4.13.0-platina-mk1
ii  rsyslog                        8.4.2-1+deb8u2                   amd64        reliable system and kernel logging daemon
root@invader45:~#

Steps:

  1. Executed below playbook. 
abhishek@invader28:~/latest/go/ansible/regtest$ ansible-playbook -i hosts_testbed2 playbooks/bgp/quagga_bgp_peering_redundancy.yml -K -e reset_config=False
SUDO password:
[DEPRECATION WARNING]: 'include' for playbook includes. You should use 'import_playbook' instead. This feature will be removed in version 2.8. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.

...
...

TASK [Print test case result] ***************************************************************************************************************************************************************************
ok: [server_emulator] => {
    "msg": "Test case result: Passed"
}

TASK [Print failure summary] ****************************************************************************************************************************************************************************
ok: [server_emulator] => {
    "msg": "Failure summary: None"
}

...
...

abhishek@invader28:~/latest/go/ansible/regtest$
  1. Playbook execution was successful. Observe the 'route' command output. 
    
root@invader44:/# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
...
...
192.168.42.1    10.0.3.32       255.255.255.255 UGH   0      0        0 eth-3-1
192.168.43.1    10.0.19.31      255.255.255.255 UGH   0      0        0 eth-19-1
192.168.45.1    10.0.3.32       255.255.255.255 UGH   0      0        0 eth-3-1
root@invader44:/#

root@invader45:/# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 172.17.2.1 0.0.0.0 UG 0 0 0 eth0 ... ... 192.168.42.1 10.0.19.32 255.255.255.255 UGH 0 0 0 eth-19-1 192.168.43.1 10.0.3.31 255.255.255.255 UGH 0 0 0 eth-3-1 192.168.44.1 10.0.19.32 255.255.255.255 UGH 0 0 0 eth-19-1 root@invader45:/#

root@invader42:/# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 172.17.2.1 0.0.0.0 UG 0 0 0 eth0 ... ... 192.168.43.1 10.0.3.30 255.255.255.255 UGH 0 0 0 eth-3-1 192.168.44.1 10.0.3.30 255.255.255.255 UGH 0 0 0 eth-3-1 192.168.45.1 10.0.19.29 255.255.255.255 UGH 0 0 0 eth-19-1 root@invader42:/#

root@invader43:/# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 172.17.2.1 0.0.0.0 UG 0 0 0 eth0 ... ... 192.168.42.1 10.0.19.30 255.255.255.255 UGH 0 0 0 eth-19-1 192.168.44.1 10.0.19.30 255.255.255.255 UGH 0 0 0 eth-19-1 192.168.45.1 10.0.3.29 255.255.255.255 UGH 0 0 0 eth-3-1 root@invader43:/#

3. This playbook has already bring down & bring up interfaces in invader44 & invader45 to check the redundancy but consider the following manual steps and debug points. Start ping to 192.168.44.1. Here the packet was going out from eth-27-1 but 'route' command output shows the exit path from eth-19-1 though the ping was successful.

root@invader45:~# ping 192.168.44.1 -I 192.168.45.1 PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data. 64 bytes from 192.168.44.1: icmp_seq=1 ttl=63 time=0.221 ms 64 bytes from 192.168.44.1: icmp_seq=2 ttl=63 time=0.242 ms 64 bytes from 192.168.44.1: icmp_seq=3 ttl=63 time=0.217 ms ... ... 64 bytes from 192.168.44.1: icmp_seq=58 ttl=63 time=0.237 ms ^C --- 192.168.44.1 ping statistics --- 58 packets transmitted, 58 received, 0% packet loss, time 58372ms rtt min/avg/max/mdev = 0.125/0.186/0.242/0.031 ms root@invader45:~#

root@invader45:~# tcpdump -i eth-19-1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth-19-1, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@invader45:~# root@invader45:~# tcpdump -i eth-3-1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth-3-1, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@invader45:~# root@invader45:~# tcpdump -i eth-5-1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth-5-1, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@invader45:~# root@invader45:~# tcpdump -i eth-27-1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth-27-1, link-type EN10MB (Ethernet), capture size 262144 bytes 04:37:56.172610 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28504, seq 50, length 64 04:37:56.172769 IP 192.168.44.1 > 192.168.45.1: ICMP echo reply, id 28504, seq 50, length 64 04:37:57.196566 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28504, seq 51, length 64 04:37:57.196710 IP 192.168.44.1 > 192.168.45.1: ICMP echo reply, id 28504, seq 51, length 64 04:37:58.220615 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28504, seq 52, length 64 04:37:58.220728 IP 192.168.44.1 > 192.168.45.1: ICMP echo reply, id 28504, seq 52, length 64 04:37:59.244608 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28504, seq 53, length 64 04:37:59.244811 IP 192.168.44.1 > 192.168.45.1: ICMP echo reply, id 28504, seq 53, length 64 04:38:00.268600 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28504, seq 54, length 64 04:38:00.268778 IP 192.168.44.1 > 192.168.45.1: ICMP echo reply, id 28504, seq 54, length 64 ^C 10 packets captured 10 packets received by filter 0 packets dropped by kernel root@invader45:~#

4. Bringing down interface eth-27-1

root@invader45:~# ifconfig eth-27-1 down root@invader45:~#

root@invader45:~# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 172.17.2.1 0.0.0.0 UG 0 0 0 eth0 ... ... 192.168.42.1 10.0.19.32 255.255.255.255 UGH 0 0 0 eth-19-1 192.168.43.1 10.0.3.31 255.255.255.255 UGH 0 0 0 eth-3-1 192.168.44.1 10.0.19.32 255.255.255.255 UGH 0 0 0 eth-19-1 root@invader45:~#

5. Observe whether the ping is working or not. There is not response in this case until the network converge to make the ping work.

root@invader45:/# ping 192.168.44.1 -I 192.168.45.1 PING 192.168.44.1 (192.168.44.1) from 192.168.45.1 : 56(84) bytes of data. ^C --- 192.168.44.1 ping statistics --- 30 packets transmitted, 0 received, 100% packet loss, time 29674ms

root@invader45:/#

root@invader45:/# tcpdump -i eth-19-1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth-19-1, link-type EN10MB (Ethernet), capture size 262144 bytes 06:24:02.540618 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28546, seq 11, length 64 06:24:03.564570 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28546, seq 12, length 64 06:24:04.592623 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28546, seq 13, length 64 06:24:05.612604 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28546, seq 14, length 64 06:24:06.636603 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28546, seq 15, length 64 06:24:07.660604 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28546, seq 16, length 64 ... ... 06:24:21.996611 IP 192.168.45.1 > 192.168.44.1: ICMP echo request, id 28546, seq 30, length 64 06:24:23.312570 IP 192.168.45.1.bgp > 192.168.42.1.38025: Flags [P.], seq 4126797210:4126797229, ack 1880051528, win 114, options [nop,nop,TS val 3050149144 ecr 3247164448], length 19: BGP ^C 21 packets captured 21 packets received by filter 0 packets dropped by kernel root@invader45:/#

jignesh045 commented 6 years ago

After adding above comment https://github.com/platinasystems/go/issues/120#issuecomment-409234963 me & Frank had a discussion over email on few points that needs to be debug. Below are the summary of discussion.

Point 1:

Jignesh: In this case, what i was expecting is - Should not the inavder45 choose the exit path which is displaying in 'route' output to reach to invader44 (Not sure about the GoES FIB behaviour here)? I think the actual exit interface used in ping request and the exit interface displayed in 'route' command should match.

Frank: The “route” command in Linux does NOT tell you which of the available multi-paths a packet will take. The command just picks one of the available interface, probably the first, one and displays it. The actual path that the packet takes is done via ecmp hash in Linux which the “route” command has no idea because it is packet dependent. In short, don’t use the “route” command to determine which path Linux takes in the case of multi-path.

Point 2:

Jignesh: I was expecting here is - All invaders have dual link connected to each other so in redundancy step execution i have one interface always in UP state so i should not get any ping failure (Don't know who is not converging this BGP topology during interface up/down case? may be BGP, GOES FIB, Linux?). Look at the last step of issue-120 comment when i bring down eth-27-1 interface of inv45 and it started sending ping request from eth-19-1 but did not received any packet.

Frank: On point 2, how long did you have to wait before ping resumes? There is a finite period of time necessary for BGP to propagate to each other with path is reachable and which path is not. It should be very fast however, particularly in such a small setup as ours.

Jignesh: I have waited approximately 30 - 45 sec for ping to continue working. I was expecting the ping response (issue-120 comment, last step) from any one interface of Invader42 or invader43 without BGP convergence because they have two directly connected interfaces with Invader44 (not sure about the Linux behavior of choosing path in redundancy case). I agree about BGP convergence to take place when we have only one link connected between invaders then bringing down and bringing up any interface requires BGP convergence.

Frank: That’s way to long even for convergence. For a small cluster it should converge in under a second. Needs to be debugged.

jignesh045 commented 6 years ago

I have applied the below two commands in all invaders and applied 'systemctl daemon-reload' command & then executed playbook to configure setup2 and found that some cases are failing.

sysctl -w net.ipv4.fib_multipath_hash_policy=1 sysctl -w net.ipv4.fib_multipath_use_neigh=1

Below are the steps that i have executed in inv45.

1) Initially observed the following command output (to get the idea of how network is converged) in all invaders: route, vtysh -c 'sh ip route bgp', vtysh -c 'sh ip route' 2) Inv45 has two interfaces to reach inv44 (Dummy interface IP: 192.168.44.1) which are eth-3-1 & eth-5-1 (via inv43) 3) Initially the ping (initiated from inv45) to 192.168.44.1 from 192.168.45.1 (dummy interface of inv45) was working using eth-3-1 interface 4) I have bring down interface eth-3-1 and check that the ping was working without any loss using eth-5-1 interface 5) I have bring up the interface eth-3-1 and observed that the ping was working using same eth-5-1 interface 6) I have bring down interface eth-5-1 and observed that there is no ping response/reply received. Further debugged that inv44 was replying via eth-19-1 interface but inv43 has not forwarded the ping reply back to inv45. 7) waited around 150-160 seconds and observed that there was a BGP convergence happened and the ping started working again.

RohanK-Calsoft commented 5 years ago

Build version:-

root@invader42:/home/rohan# goes vnetd -version fe1: v1.2.1 fe1a: v1.1.0 vnet-platina-mk1: v1.1.0-rc.1

root@invader42:/home/rohan# goes version v1.2.0-rc0

BuildID[sha1]=d32beefe6b8ba9ac8906ef947a18e5adb66bd938

Verified this on the latest-build and this issue is still reproducible in some of the test-cases The test-cases related to this issue is

1) bird_bgp_peering_redundancy 2) gobgp_bgp_peering_redundancy