platinasystems / go

Other
9 stars 68 forks source link

netlink doesn't support interface routes (i.e. no gateway) #88

Open jignesh045 opened 7 years ago

jignesh045 commented 7 years ago

Issue Description: qsfp & vnetd daemon is not working

Environment: Testbed setup2 (invader42, Invader43, Invader44, Invader45)

GOES version: github.com/platinasystems/go: v0.4-625-g4124346 github.com/platinasystems/fe1: v0.3-236-g581c0a5 github.com/platinasystems/firmware-fe1a: v0.2

Steps:

  1. Assigned loop back IP address in all invaders starting with 192.168.xx.xx/24 to lo interface.
  2. Assigned 192.168.xx.xx/24 IP address to two ports of each invader where xx.xx is port.invader_IP(in this scenario IP address of two interfaces are changed from 10.0.xx.xx to 192.168.xx.xx to avoid confusion with other available interface of 10.0.xx.xx IP)
  3. Configured test bed 2 for ECMP (equal cost multipath routing) test case with static routing.
  4. Configured Leaf/Spine invaders with two different static path two reach to other Leaf/Spine invader using two possible way (as shown in output of static route in show ip route command for invader42).
  5. After configuration restarted quagga service. Later find that goes status in not OK so got below logs in all four invaders.

Nov 8 21:56:32 invader43 systemd[1]: Started User Manager for UID 1013. Nov 8 21:57:57 invader43 goes.qsfp[13443]: dialing:dial tcp 127.0.0.1:1233: getsockopt: connection timed out Nov 8 21:57:57 invader43 goes.qsfp[13443]: panic: runtime error: invalid memory address or nil pointer dereference Nov 8 21:57:57 invader43 goes.qsfp[13443]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x70d046] Nov 8 21:57:57 invader43 goes.qsfp[13443]: Nov 8 21:57:57 invader43 goes.qsfp[13443]: goroutine 1 [running]: Nov 8 21:57:57 invader43 goes.qsfp[13443]: net/rpc.(Client).send(0x0, 0xc4201fa230) Nov 8 21:57:57 invader43 goes.qsfp[13443]: /usr/local/go/src/net/rpc/client.go:72 +0x26 Nov 8 21:57:57 invader43 goes.qsfp[13443]: net/rpc.(Client).Go(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xc42023a060, 0xb7aea0) Nov 8 21:57:57 invader43 goes.qsfp[13443]: /usr/local/go/src/net/rpc/client.go:311 +0xe7 Nov 8 21:57:57 invader43 goes.qsfp[13443]: net/rpc.(Client).Call(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xe, 0x14a) Nov 8 21:57:57 invader43 goes.qsfp[13443]: /usr/local/go/src/net/rpc/client.go:317 +0x97 Nov 8 21:57:57 invader43 goes.qsfp[13443]: github.com/platinasystems/go/goes/cmd/qsfp.DoI2cRpc() Nov 8 21:57:57 invader43 goes.qsfp[13443]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:188 +0x94 Nov 8 21:57:57 invader43 goes.qsfp[13443]: github.com/platinasystems/go/goes/cmd/qsfp.readStopped(0xc71820) Nov 8 21:57:57 invader43 goes.qsfp[13443]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:175 +0xe6 Nov 8 21:57:57 invader43 goes.qsfp[13443]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).updatePresence(0xc420139680, 0xc42020e180, 0x0) Nov 8 21:57:57 invader43 goes.qsfp[13443]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:134 +0x4d Nov 8 21:57:57 invader43 goes.qsfp[13443]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).Main(0xc420139680, 0xc4200101d0, 0x0, 0x0, 0x0, 0x0) Nov 8 21:57:57 invader43 goes.qsfp[13443]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:120 +0x517 Nov 8 21:57:57 invader43 goes.qsfp[13443]: github.com/platinasystems/go/goes.(Goes).Main(0xc420087550, 0xc4200101d0, 0x1, 0x1, 0x0, 0x0) Nov 8 21:57:57 invader43 goes.qsfp[13443]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/goes.go:339 +0xd46 Nov 8 21:57:57 invader43 goes.qsfp[13443]: main.main() Nov 8 21:57:57 invader43 goes.qsfp[13443]: /home/jpang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:19 +0x57 Nov 8 21:57:57 invader43 goes.qsfp[13443]: exit status 2

root@invader43:/# goes status GOES status

Mode - TUNTAP PCI - OK Check daemons - Not OK status: qsfp daemon not running root@invader43:/# root@invader43:/# root@invader43:/# date Wed Nov 8 22:03:20 PST 2017 root@invader43:/# root@invader43:/#

Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-9-1: link is not ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-7-1: link becomes ready Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-11-1: link is not ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-1-1: link becomes ready Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-13-1: link is not ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-11-1: link becomes ready Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-15-1: link is not ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-13-1: link becomes ready Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-17-1: link is not ready Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-5-1: link becomes ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-15-1: link becomes ready Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-19-1: link is not ready Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-9-1: link becomes ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-21-1: link is not ready Nov 8 22:34:31 invader43 goes.vnetd[17028]: panic: vnet: RTA_MULTIPATH next-hop without RTA_GATEWAY Nov 8 22:34:31 invader43 goes.vnetd[17028]: goroutine 11 [running]: Nov 8 22:34:31 invader43 goes.vnetd[17028]: runtime/debug.Stack(0xbc40a0, 0xc421250e60, 0xc42176baa0) Nov 8 22:34:31 invader43 goes.vnetd[17028]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79 Nov 8 22:34:31 invader43 goes.vnetd[17028]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler.func1(0xc420222000, 0xc420222558) Nov 8 22:34:31 invader43 goes.vnetd[17028]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:181 +0x159 Nov 8 22:34:31 invader43 goes.vnetd[17028]: panic(0xb7aea0, 0xc421250e30) Nov 8 22:34:31 invader43 goes.vnetd[17028]: /usr/local/go/src/runtime/panic.go:489 +0x2cf Nov 8 22:34:31 invader43 goes.vnetd[17028]: github.com/platinasystems/go/vnet/unix.(net_namespace).parse_ip4_next_hops(0xc42021cb10, 0xc42171c4e0, 0xc42176bd20, 0x201f0ba8c0, 0xc421eaecc5) Nov 8 22:34:31 invader43 goes.vnetd[17028]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:678 +0x4cf Nov 8 22:34:31 invader43 goes.vnetd[17028]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).ip4RouteMsg(0xc420260300, 0xc42171c4e0, 0xc42171c400, 0x1, 0x0) Nov 8 22:34:31 invader43 goes.vnetd[17028]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:706 +0xdf Nov 8 22:34:31 invader43 goes.vnetd[17028]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).EventAction(0xc420260300) Nov 8 22:34:31 invader43 goes.vnetd[17028]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:497 +0x5dd Nov 8 22:34:31 invader43 goes.vnetd[17028]: github.com/platinasystems/go/elib/loop.(nodeEvent).do(0xc420260540) Nov 8 22:34:31 invader43 goes.vnetd[17028]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:152 +0x8f Nov 8 22:34:31 invader43 goes.vnetd[17028]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler(0xc420222000, 0x15dcc60, 0xc420222550) Nov 8 22:34:31 invader43 goes.vnetd[17028]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:195 +0x175 Nov 8 22:34:31 invader43 goes.vnetd[17028]: created by github.com/platinasystems/go/elib/loop.(Node).maybeStartEventHandler Nov 8 22:34:31 invader43 goes.vnetd[17028]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:280 +0x29f Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-23-1: link is not ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-25-1: link is not ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-27-1: link is not ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-29-1: link is not ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-31-1: link is not ready Nov 8 22:34:31 invader43 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:34:31 invader43 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:36:39 invader43 goes.qsfp[17005]: dialing:dial tcp 127.0.0.1:1233: getsockopt: connection timed out Nov 8 22:36:39 invader43 goes.qsfp[17005]: dialing:dial tcp 127.0.0.1:1233: getsockopt: connection timed out Nov 8 22:36:39 invader43 goes.qsfp[17005]: panic: runtime error: invalid memory address or nil pointer dereference Nov 8 22:36:39 invader43 goes.qsfp[17005]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x70d046] Nov 8 22:36:39 invader43 goes.qsfp[17005]: Nov 8 22:36:39 invader43 goes.qsfp[17005]: goroutine 1 [running]: Nov 8 22:36:39 invader43 goes.qsfp[17005]: net/rpc.(Client).send(0x0, 0xc4202381e0) Nov 8 22:36:39 invader43 goes.qsfp[17005]: /usr/local/go/src/net/rpc/client.go:72 +0x26 Nov 8 22:36:39 invader43 goes.qsfp[17005]: net/rpc.(Client).Go(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xc42024c060, 0xb7aea0) Nov 8 22:36:39 invader43 goes.qsfp[17005]: /usr/local/go/src/net/rpc/client.go:311 +0xe7 Nov 8 22:36:39 invader43 goes.qsfp[17005]: net/rpc.(Client).Call(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xe, 0x14a) Nov 8 22:36:39 invader43 goes.qsfp[17005]: /usr/local/go/src/net/rpc/client.go:317 +0x97 Nov 8 22:36:39 invader43 goes.qsfp[17005]: github.com/platinasystems/go/goes/cmd/qsfp.DoI2cRpc() Nov 8 22:36:39 invader43 goes.qsfp[17005]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:188 +0x94 Nov 8 22:36:39 invader43 goes.qsfp[17005]: github.com/platinasystems/go/goes/cmd/qsfp.readStopped(0xc71820) Nov 8 22:36:39 invader43 goes.qsfp[17005]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:175 +0xe6 Nov 8 22:36:39 invader43 goes.qsfp[17005]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).updatePresence(0xc42012b680, 0xc420228180, 0x0) Nov 8 22:36:39 invader43 goes.qsfp[17005]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:134 +0x4d Nov 8 22:36:39 invader43 goes.qsfp[17005]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).Main(0xc42012b680, 0xc4200101d0, 0x0, 0x0, 0x0, 0x0) Nov 8 22:36:39 invader43 goes.qsfp[17005]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:120 +0x517 Nov 8 22:36:39 invader43 goes.qsfp[17005]: github.com/platinasystems/go/goes.(*Goes).Main(0xc420079340, 0xc4200101d0, 0x1, 0x1, 0x0, 0x0) Nov 8 22:36:39 invader43 goes.qsfp[17005]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/goes.go:339 +0xd46 Nov 8 22:36:39 invader43 goes.qsfp[17005]: main.main() Nov 8 22:36:39 invader43 goes.qsfp[17005]: /home/jpang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:19 +0x57 Nov 8 22:36:39 invader43 goes.qsfp[17005]: exit status 2

root@invader43:/# date Wed Nov 8 22:39:18 PST 2017 root@invader43:/#


Aug 15 04:12:03 invader42 systemd[1]: Started User Manager for UID 1018. Aug 15 04:17:01 invader42 CRON[31554]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 15 04:31:25 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-3-1: link becomes ready Aug 15 04:31:25 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-5-1: link becomes ready Aug 15 04:31:25 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-7-1: link becomes ready Aug 15 04:31:25 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-11-1: link becomes ready Aug 15 04:31:25 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-15-1: link becomes ready Aug 15 04:36:26 invader42 goes.vnetd[25370]: panic: vnet: RTA_MULTIPATH next-hop without RTA_GATEWAY Aug 15 04:36:26 invader42 goes.vnetd[25370]: goroutine 15 [running]: Aug 15 04:36:26 invader42 goes.vnetd[25370]: runtime/debug.Stack(0xbc40a0, 0xc422268c90, 0xc421f81aa0) Aug 15 04:36:26 invader42 goes.vnetd[25370]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79 Aug 15 04:36:26 invader42 goes.vnetd[25370]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler.func1(0xc420222000, 0xc420222558) Aug 15 04:36:26 invader42 goes.vnetd[25370]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:181 +0x159 Aug 15 04:36:26 invader42 goes.vnetd[25370]: panic(0xb7aea0, 0xc422268c60) Aug 15 04:36:26 invader42 goes.vnetd[25370]: /usr/local/go/src/runtime/panic.go:489 +0x2cf Aug 15 04:36:26 invader42 goes.vnetd[25370]: github.com/platinasystems/go/vnet/unix.(net_namespace).parse_ip4_next_hops(0xc4202a4090, 0xc421683860, 0xc421cc2720, 0x202005a8c0, 0x0) Aug 15 04:36:26 invader42 goes.vnetd[25370]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:678 +0x4cf Aug 15 04:36:26 invader42 goes.vnetd[25370]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).ip4RouteMsg(0xc423381b40, 0xc421683860, 0xc421683801, 0x1, 0x0) Aug 15 04:36:26 invader42 goes.vnetd[25370]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:706 +0xdf Aug 15 04:36:26 invader42 goes.vnetd[25370]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).EventAction(0xc423381b40) Aug 15 04:36:26 invader42 goes.vnetd[25370]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:497 +0x5dd Aug 15 04:36:26 invader42 goes.vnetd[25370]: github.com/platinasystems/go/elib/loop.(nodeEvent).do(0xc4213c96c0) Aug 15 04:36:26 invader42 goes.vnetd[25370]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:152 +0x8f Aug 15 04:36:26 invader42 goes.vnetd[25370]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler(0xc420222000, 0x15dcc60, 0xc420222550) Aug 15 04:36:26 invader42 goes.vnetd[25370]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:195 +0x175 Aug 15 04:36:26 invader42 goes.vnetd[25370]: created by github.com/platinasystems/go/elib/loop.(*Node).maybeStartEventHandler Aug 15 04:36:26 invader42 goes.vnetd[25370]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:280 +0x29f Aug 15 05:09:44 invader42 systemd[1]: Starting user-1013.slice. Aug 15 05:09:44 invader42 systemd[1]: Created slice user-1013.slice. Aug 15 05:09:44 invader42 systemd[1]: Starting User Manager for UID 1013... Aug 15 05:09:44 invader42 systemd[1]: Starting Session c25 of user shreyas. Aug 15 05:09:44 invader42 systemd[1]: Started Session c25 of user shreyas. Aug 15 05:09:44 invader42 systemd[31679]: Starting Paths. Aug 15 05:09:44 invader42 systemd[31679]: Reached target Paths. Aug 15 05:09:44 invader42 systemd[31679]: Starting Timers. Aug 15 05:09:44 invader42 systemd[31679]: Reached target Timers. Aug 15 05:09:44 invader42 systemd[31679]: Starting Sockets. Aug 15 05:09:44 invader42 systemd[31679]: Reached target Sockets. Aug 15 05:09:44 invader42 systemd[31679]: Starting Basic System. Aug 15 05:09:44 invader42 systemd[31679]: Reached target Basic System. Aug 15 05:09:44 invader42 systemd[31679]: Starting Default. Aug 15 05:09:44 invader42 systemd[31679]: Reached target Default. Aug 15 05:09:44 invader42 systemd[31679]: Startup finished in 23ms. Aug 15 05:09:44 invader42 systemd[1]: Started User Manager for UID 1013.

root@invader42:/# goes status GOES status

Mode - TUNTAP PCI - OK Check daemons - Not OK status: qsfp daemon not running

root@invader42:/#

root@invader42:/# vtysh -c "sh ip route" Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, A - Babel,

  • selected route, * - FIB route

K> 0.0.0.0/0 via 172.17.2.1, eth0 C> 127.0.0.0/8 is directly connected, lo C> 172.17.2.0/23 is directly connected, eth0 C> 172.18.0.0/16 is directly connected, docker0 S 192.168.5.27/32 [1/0] is directly connected, eth-27-1 inactive S 192.168.5.30/32 [1/0] is directly connected, eth-27-1 inactive S 192.168.5.32/32 [1/0] is directly connected, eth-27-1 inactive is directly connected, eth-11-1 inactive S 192.168.11.29/32 [1/0] is directly connected, eth-11-1 inactive S 192.168.21.29/32 [1/0] is directly connected, eth-11-1 inactive S 192.168.21.32/32 [1/0] is directly connected, eth-27-1 inactive is directly connected, eth-11-1 inactive root@invader42:/#

root@invader42:/# root@invader42:/# date Tue Aug 15 05:15:49 PDT 2017 root@invader42:/# root@invader42:/#

root@invader42:/# goes restart root@invader42:/# root@invader42:/# date Tue Aug 15 05:45:19 PDT 2017 root@invader42:/# root@invader42:/# root@invader42:/# goes status GOES status

Mode - TUNTAP PCI - OK Check daemons - OK Check Redis - OK Check vnet - Not OK status: vnetd daemon not responding root@invader42:/# root@invader42:/# date Tue Aug 15 05:45:41 PDT 2017 root@invader42:/# root@invader42:/# vi /var/log/syslog

Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-13-1: link is not ready Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-15-1: link is not ready Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-17-1: link is not ready Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-1-1: link becomes ready Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-19-1: link is not ready Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-9-1: link becomes ready Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-21-1: link is not ready Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-19-1: link becomes ready Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-23-1: link is not ready Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-25-1: link is not ready Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-23-1: link becomes ready Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-17-1: link becomes ready Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-27-1: link is not ready Aug 15 05:45:17 invader42 goes.vnetd[32054]: panic: vnet: RTA_MULTIPATH next-hop without RTA_GATEWAY Aug 15 05:45:17 invader42 goes.vnetd[32054]: goroutine 11 [running]: Aug 15 05:45:17 invader42 goes.vnetd[32054]: runtime/debug.Stack(0xbc40a0, 0xc42013d020, 0xc421557aa0) Aug 15 05:45:17 invader42 goes.vnetd[32054]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79 Aug 15 05:45:17 invader42 goes.vnetd[32054]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler.func1(0xc4201dc000, 0xc4201dc558) Aug 15 05:45:17 invader42 goes.vnetd[32054]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:181 +0x159 Aug 15 05:45:17 invader42 goes.vnetd[32054]: panic(0xb7aea0, 0xc42013cfe0) Aug 15 05:45:17 invader42 goes.vnetd[32054]: /usr/local/go/src/runtime/panic.go:489 +0x2cf Aug 15 05:45:17 invader42 goes.vnetd[32054]: github.com/platinasystems/go/vnet/unix.(net_namespace).parse_ip4_next_hops(0xc420274090, 0xc421be8d00, 0xc421557d20, 0x202005a8c0, 0xc421d39985) Aug 15 05:45:17 invader42 goes.vnetd[32054]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:678 +0x4cf Aug 15 05:45:17 invader42 goes.vnetd[32054]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).ip4RouteMsg(0xc42023c0c0, 0xc421be8d00, 0xc421be8d00, 0x1, 0x0) Aug 15 05:45:17 invader42 goes.vnetd[32054]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:706 +0xdf Aug 15 05:45:17 invader42 goes.vnetd[32054]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).EventAction(0xc42023c0c0) Aug 15 05:45:17 invader42 goes.vnetd[32054]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:497 +0x5dd Aug 15 05:45:17 invader42 goes.vnetd[32054]: github.com/platinasystems/go/elib/loop.(nodeEvent).do(0xc42012e000) Aug 15 05:45:17 invader42 goes.vnetd[32054]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:152 +0x8f Aug 15 05:45:17 invader42 goes.vnetd[32054]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler(0xc4201dc000, 0x15dcc60, 0xc4201dc550) Aug 15 05:45:17 invader42 goes.vnetd[32054]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:195 +0x175 Aug 15 05:45:17 invader42 goes.vnetd[32054]: created by github.com/platinasystems/go/elib/loop.(*Node).maybeStartEventHandler Aug 15 05:45:17 invader42 goes.vnetd[32054]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:280 +0x29f Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-29-1: link is not ready Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-31-1: link is not ready Aug 15 05:45:17 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:45:17 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server.

root@invader42:/#

Aug 15 05:49:34 invader42 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-23-1: link becomes ready Aug 15 05:49:34 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-27-1: link is not ready Aug 15 05:49:34 invader42 goes.vnetd[6308]: panic: vnet: RTA_MULTIPATH next-hop without RTA_GATEWAY Aug 15 05:49:34 invader42 goes.vnetd[6308]: goroutine 35 [running]: Aug 15 05:49:34 invader42 goes.vnetd[6308]: runtime/debug.Stack(0xbc40a0, 0xc4213a8fa0, 0xc421cb3aa0) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79 Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler.func1(0xc4201d0000, 0xc4201d0558) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:181 +0x159 Aug 15 05:49:34 invader42 goes.vnetd[6308]: panic(0xb7aea0, 0xc4213a8f50) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /usr/local/go/src/runtime/panic.go:489 +0x2cf Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/vnet/unix.(net_namespace).parse_ip4_next_hops(0xc420288090, 0xc4217ccd00, 0xc421cb3d20, 0x202005a8c0, 0xc421e2bf84) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:678 +0x4cf Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).ip4RouteMsg(0xc420051b80, 0xc4217ccd00, 0xc4217ccd00, 0x1, 0x0) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:706 +0xdf Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).EventAction(0xc420051b80) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:497 +0x5dd Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/elib/loop.(nodeEvent).do(0xc420051bc0) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:152 +0x8f Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler(0xc4201d0000, 0x15dcc60, 0xc4201d0550) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:195 +0x175 Aug 15 05:49:34 invader42 goes.vnetd[6308]: created by github.com/platinasystems/go/elib/loop.(Node).maybeStartEventHandler Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:280 +0x29f Aug 15 05:49:34 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-29-1: link is not ready Aug 15 05:49:34 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-31-1: link is not ready Aug 15 05:49:34 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:51:41 invader42 goes.qsfp[6282]: dialing:dial tcp 127.0.0.1:1233: getsockopt: connection timed out Aug 15 05:51:41 invader42 goes.qsfp[6282]: panic: runtime error: invalid memory address or nil pointer dereference Aug 15 05:51:41 invader42 goes.qsfp[6282]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x70d046] Aug 15 05:51:41 invader42 goes.qsfp[6282]: Aug 15 05:51:41 invader42 goes.qsfp[6282]: goroutine 1 [running]: Aug 15 05:51:41 invader42 goes.qsfp[6282]: net/rpc.(Client).send(0x0, 0xc42005c6e0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /usr/local/go/src/net/rpc/client.go:72 +0x26 Aug 15 05:51:41 invader42 goes.qsfp[6282]: net/rpc.(Client).Go(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xc42006c4e0, 0xb7aea0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /usr/local/go/src/net/rpc/client.go:311 +0xe7 Aug 15 05:51:41 invader42 goes.qsfp[6282]: net/rpc.(Client).Call(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xe, 0x14a) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /usr/local/go/src/net/rpc/client.go:317 +0x97 Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes/cmd/qsfp.DoI2cRpc() Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:188 +0x94 Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes/cmd/qsfp.readStopped(0xc71820) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:175 +0xe6 Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).updatePresence(0xc4201659b0, 0xc42006c3c0, 0x0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:134 +0x4d Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).Main(0xc4201659b0, 0xc4200101d0, 0x0, 0x0, 0x0, 0x0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:120 +0x517 Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes.(*Goes).Main(0xc42013a370, 0xc4200101d0, 0x1, 0x1, 0x0, 0x0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/goes.go:339 +0xd46 Aug 15 05:51:41 invader42 goes.qsfp[6282]: main.main() Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:19 +0x57 Aug 15 05:51:41 invader42 goes.qsfp[6282]: exit status 2

root@invader42:/# date Tue Aug 15 05:55:08 PDT 2017 root@invader42:/# root@invader42:/# vi /var/log/daemon.log

Aug 15 05:49:34 invader42 goes.vnetd[6308]: panic: vnet: RTA_MULTIPATH next-hop without RTA_GATEWAY Aug 15 05:49:34 invader42 goes.vnetd[6308]: goroutine 35 [running]: Aug 15 05:49:34 invader42 goes.vnetd[6308]: runtime/debug.Stack(0xbc40a0, 0xc4213a8fa0, 0xc421cb3aa0) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79 Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler.func1(0xc4201d0000, 0xc4201d0558) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:181 +0x159 Aug 15 05:49:34 invader42 goes.vnetd[6308]: panic(0xb7aea0, 0xc4213a8f50) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /usr/local/go/src/runtime/panic.go:489 +0x2cf Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/vnet/unix.(net_namespace).parse_ip4_next_hops(0xc420288090, 0xc4217ccd00, 0xc421cb3d20, 0x202005a8c0, 0xc421e2bf84) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:678 +0x4cf Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).ip4RouteMsg(0xc420051b80, 0xc4217ccd00, 0xc4217ccd00, 0x1, 0x0) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:706 +0xdf Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).EventAction(0xc420051b80) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:497 +0x5dd Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/elib/loop.(nodeEvent).do(0xc420051bc0) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:152 +0x8f Aug 15 05:49:34 invader42 goes.vnetd[6308]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler(0xc4201d0000, 0x15dcc60, 0xc4201d0550) Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:195 +0x175 Aug 15 05:49:34 invader42 goes.vnetd[6308]: created by github.com/platinasystems/go/elib/loop.(Node).maybeStartEventHandler Aug 15 05:49:34 invader42 goes.vnetd[6308]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:280 +0x29f Aug 15 05:49:34 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 systemd[1]: Reloading OpenBSD Secure Shell server. Aug 15 05:49:34 invader42 systemd[1]: Reloaded OpenBSD Secure Shell server. Aug 15 05:51:41 invader42 goes.qsfp[6282]: panic: runtime error: invalid memory address or nil pointer dereference Aug 15 05:51:41 invader42 goes.qsfp[6282]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x70d046] Aug 15 05:51:41 invader42 goes.qsfp[6282]: Aug 15 05:51:41 invader42 goes.qsfp[6282]: goroutine 1 [running]: Aug 15 05:51:41 invader42 goes.qsfp[6282]: net/rpc.(Client).send(0x0, 0xc42005c6e0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /usr/local/go/src/net/rpc/client.go:72 +0x26 Aug 15 05:51:41 invader42 goes.qsfp[6282]: net/rpc.(Client).Go(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xc42006c4e0, 0xb7aea0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /usr/local/go/src/net/rpc/client.go:311 +0xe7 Aug 15 05:51:41 invader42 goes.qsfp[6282]: net/rpc.(Client).Call(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xe, 0x14a) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /usr/local/go/src/net/rpc/client.go:317 +0x97 Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes/cmd/qsfp.DoI2cRpc() Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:188 +0x94 Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes/cmd/qsfp.readStopped(0xc71820) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:175 +0xe6 Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).updatePresence(0xc4201659b0, 0xc42006c3c0, 0x0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:134 +0x4d Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).Main(0xc4201659b0, 0xc4200101d0, 0x0, 0x0, 0x0, 0x0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:120 +0x517 Aug 15 05:51:41 invader42 goes.qsfp[6282]: github.com/platinasystems/go/goes.(*Goes).Main(0xc42013a370, 0xc4200101d0, 0x1, 0x1, 0x0, 0x0) Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/goes.go:339 +0xd46 Aug 15 05:51:41 invader42 goes.qsfp[6282]: main.main() Aug 15 05:51:41 invader42 goes.qsfp[6282]: /home/jpang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:19 +0x57 Aug 15 05:51:41 invader42 goes.qsfp[6282]: exit status 2 Aug 15 06:07:06 invader42 systemd[1]: Starting Synchronise Hardware Clock to System Clock... Aug 15 06:07:06 invader42 systemd[1]: Stopping Session c27 of user shreyas.


Nov 8 22:04:36 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:36 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:37 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:37 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:37 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:37 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:37 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:37 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:38 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:38 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:38 invader45 goes.qsfp[408]: i2cReq error:connection is shut down Nov 8 22:04:38 invader45 goes.qsfp[408]: i2cReq error:connection is shut down

root@invader45:/# goes status GOES status

Mode - TUNTAP PCI - OK Check daemons - OK Check Redis - OK Check vnet - Not OK status: vnetd daemon not responding root@invader45:/# root@invader45:/# date Wed Nov 8 22:09:20 PST 2017 root@invader45:/#


Nov 8 22:03:37 invader44 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:03:37 invader44 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:03:37 invader44 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-27-1: link is not ready Nov 8 22:03:37 invader44 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:03:37 invader44 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:03:37 invader44 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-29-1: link is not ready Nov 8 22:03:37 invader44 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:03:37 invader44 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:03:37 invader44 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-31-1: link is not ready Nov 8 22:03:37 invader44 systemd[1]: Reloading OpenBSD Secure Shell server. Nov 8 22:03:37 invader44 systemd[1]: Reloaded OpenBSD Secure Shell server. Nov 8 22:03:46 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 13 Nov 8 22:03:46 invader44 dhclient: send_packet: No such device Nov 8 22:03:46 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:03:59 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 14 Nov 8 22:03:59 invader44 dhclient: send_packet: No such device Nov 8 22:03:59 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:04:13 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 2 Nov 8 22:04:13 invader44 dhclient: send_packet: No such device Nov 8 22:04:13 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:04:15 invader44 dhclient: No DHCPOFFERS received. Nov 8 22:04:15 invader44 dhclient: No working leases in persistent database - sleeping. Nov 8 22:05:44 invader44 goes.qsfp[6147]: dialing:dial tcp 127.0.0.1:1233: getsockopt: connection timed out Nov 8 22:05:44 invader44 goes.qsfp[6147]: dialing:dial tcp 127.0.0.1:1233: getsockopt: connection timed out Nov 8 22:05:44 invader44 goes.qsfp[6147]: panic: runtime error: invalid memory address or nil pointer dereference Nov 8 22:05:44 invader44 goes.qsfp[6147]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x70d046] Nov 8 22:05:44 invader44 goes.qsfp[6147]: Nov 8 22:05:44 invader44 goes.qsfp[6147]: goroutine 10 [running]: Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).send(0x0, 0xc420054eb0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:72 +0x26 Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).Go(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1648480, 0xb39bc0, 0x1622bc0, 0xc42005f020, 0xb7aea0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:311 +0xe7 Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).Call(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1648480, 0xb39bc0, 0x1622bc0, 0x0, 0xffffffffffffffff) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:317 +0x97 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.DoI2cRpcio() Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfpio_rw.go:131 +0x94 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.readStoppedio(0xc420035fb0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfpio_rw.go:118 +0xe6 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).updateio(0xc42012b680, 0xc42005ef60, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfpio.go:47 +0x37 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.qsfpioTicker(0xc42012b680, 0x0, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfpio.go:38 +0x143 Nov 8 22:05:44 invader44 goes.qsfp[6147]: created by github.com/platinasystems/go/goes/cmd/qsfp.(Command).Main Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:105 +0x2eb Nov 8 22:05:44 invader44 goes.qsfp[6147]: panic: runtime error: invalid memory address or nil pointer dereference Nov 8 22:05:44 invader44 goes.qsfp[6147]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x70d046] Nov 8 22:05:44 invader44 goes.qsfp[6147]: Nov 8 22:05:44 invader44 goes.qsfp[6147]: goroutine 1 [running]: Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).send(0x0, 0xc42023a000) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:72 +0x26 Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).Go(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xc420236000, 0xb7aea0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:311 +0xe7 Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).Call(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xe, 0x14a) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:317 +0x97 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.DoI2cRpc() Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:188 +0x94 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.readStopped(0xc71820) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:175 +0xe6 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).updatePresence(0xc42012b680, 0xc42005ef00, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:134 +0x4d Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).Main(0xc42012b680, 0xc4200101d0, 0x0, 0x0, 0x0, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:120 +0x517 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes.(*Goes).Main(0xc4200794a0, 0xc4200101d0, 0x1, 0x1, 0x0, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/goes.go:339 +0xd46 Nov 8 22:05:44 invader44 goes.qsfp[6147]: main.main() Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:19 +0x57 Nov 8 22:05:44 invader44 goes.qsfp[6147]: exit status 2 Nov 8 22:08:12 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 7 Nov 8 22:08:12 invader44 dhclient: send_packet: No such device Nov 8 22:08:12 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:08:19 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 12 Nov 8 22:08:19 invader44 dhclient: send_packet: No such device Nov 8 22:08:19 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:08:31 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 15 Nov 8 22:08:31 invader44 dhclient: send_packet: No such device Nov 8 22:08:31 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:08:46 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 15 Nov 8 22:08:46 invader44 dhclient: send_packet: No such device Nov 8 22:08:46 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:09:01 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 11 Nov 8 22:09:01 invader44 dhclient: send_packet: No such device Nov 8 22:09:01 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:09:05 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 8 Nov 8 22:09:05 invader44 dhclient: send_packet: No such device Nov 8 22:09:05 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:09:12 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 1 Nov 8 22:09:12 invader44 dhclient: send_packet: No such device Nov 8 22:09:12 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:09:13 invader44 dhclient: No DHCPOFFERS received. Nov 8 22:09:13 invader44 dhclient: No working leases in persistent database - sleeping. Nov 8 22:09:13 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 15 Nov 8 22:09:13 invader44 dhclient: send_packet: No such device Nov 8 22:09:13 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:09:28 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 12 Nov 8 22:09:28 invader44 dhclient: send_packet: No such device Nov 8 22:09:28 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:09:40 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 15 Nov 8 22:09:40 invader44 dhclient: send_packet: No such device Nov 8 22:09:40 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:09:55 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 11 Nov 8 22:09:55 invader44 dhclient: send_packet: No such device Nov 8 22:09:55 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:10:06 invader44 dhclient: No DHCPOFFERS received. Nov 8 22:10:06 invader44 dhclient: No working leases in persistent database - sleeping.

root@invader44:/# goes status GOES status

Mode - TUNTAP PCI - OK Check daemons - Not OK status: qsfp daemon not running root@invader44:/# root@invader44:/# root@invader44:/# date Wed Nov 8 22:11:49 PST 2017 root@invader44:/#

root@invader44:/# goes restart Device "eth-17-1" does not exist. Cannot find device "eth-17-1" Device "eth-1-1" does not exist. Cannot find device "eth-1-1" Cannot find device "eth-1-1" Failed to bring up eth-1-1. Cannot find device "eth-17-1" Failed to bring up eth-17-1. root@invader44:/# root@invader44:/# goes status GOES status

Mode - TUNTAP PCI - OK Check daemons - Not OK status: qsfp daemon not running root@invader44:/#

Nov 8 22:05:44 invader44 goes.qsfp[6147]: panic: runtime error: invalid memory address or nil pointer dereference Nov 8 22:05:44 invader44 goes.qsfp[6147]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x70d046] Nov 8 22:05:44 invader44 goes.qsfp[6147]: Nov 8 22:05:44 invader44 goes.qsfp[6147]: goroutine 10 [running]: Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).send(0x0, 0xc420054eb0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:72 +0x26 Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).Go(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1648480, 0xb39bc0, 0x1622bc0, 0xc42005f020, 0xb7aea0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:311 +0xe7 Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).Call(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1648480, 0xb39bc0, 0x1622bc0, 0x0, 0xffffffffffffffff) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:317 +0x97 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.DoI2cRpcio() Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfpio_rw.go:131 +0x94 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.readStoppedio(0xc420035fb0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfpio_rw.go:118 +0xe6 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).updateio(0xc42012b680, 0xc42005ef60, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfpio.go:47 +0x37 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.qsfpioTicker(0xc42012b680, 0x0, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfpio.go:38 +0x143 Nov 8 22:05:44 invader44 goes.qsfp[6147]: created by github.com/platinasystems/go/goes/cmd/qsfp.(Command).Main Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:105 +0x2eb Nov 8 22:05:44 invader44 goes.qsfp[6147]: panic: runtime error: invalid memory address or nil pointer dereference Nov 8 22:05:44 invader44 goes.qsfp[6147]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x70d046] Nov 8 22:05:44 invader44 goes.qsfp[6147]: Nov 8 22:05:44 invader44 goes.qsfp[6147]: goroutine 1 [running]: Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).send(0x0, 0xc42023a000) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:72 +0x26 Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).Go(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xc420236000, 0xb7aea0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:311 +0xe7 Nov 8 22:05:44 invader44 goes.qsfp[6147]: net/rpc.(Client).Call(0x0, 0xd2123a, 0x10, 0xb39b80, 0x1647a20, 0xb39bc0, 0x1622520, 0xe, 0x14a) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /usr/local/go/src/net/rpc/client.go:317 +0x97 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.DoI2cRpc() Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:188 +0x94 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.readStopped(0xc71820) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp_rw.go:175 +0xe6 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).updatePresence(0xc42012b680, 0xc42005ef00, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:134 +0x4d Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes/cmd/qsfp.(Command).Main(0xc42012b680, 0xc4200101d0, 0x0, 0x0, 0x0, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/cmd/qsfp/qsfp.go:120 +0x517 Nov 8 22:05:44 invader44 goes.qsfp[6147]: github.com/platinasystems/go/goes.(*Goes).Main(0xc4200794a0, 0xc4200101d0, 0x1, 0x1, 0x0, 0x0) Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/goes/goes.go:339 +0xd46 Nov 8 22:05:44 invader44 goes.qsfp[6147]: main.main() Nov 8 22:05:44 invader44 goes.qsfp[6147]: /home/jpang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:19 +0x57 Nov 8 22:05:44 invader44 goes.qsfp[6147]: exit status 2 Nov 8 22:08:12 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 7 Nov 8 22:08:12 invader44 dhclient: send_packet: No such device Nov 8 22:08:12 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface. Nov 8 22:08:19 invader44 dhclient: DHCPDISCOVER on eth-7-1 to 255.255.255.255 port 67 interval 12 Nov 8 22:08:19 invader44 dhclient: send_packet: No such device Nov 8 22:08:19 invader44 dhclient: dhclient.c:1966: Failed to send 300 byte long packet over eth-7-1 interface.

jasonlpang commented 7 years ago

Hi Jignesh, We just finished moving all of the qsfp functions into the vnet package. There are quite a few changes. Can you please load the latest build on to the units and see if the panics still occur?

// copy “goes-platina-mk1-installer” from the location below jpang@invader29:~$ ls -l total 14172 -rwxr-xr-x 1 jpang root 9298094 Nov 14 13:05 goes-platina-mk1-installer

// on each box:

/goes-platina-mk1-installer If the upgrade works properly you should get the same output as bellow on each box $ goes status GOES status ====================== Mode - TUNTAP PCI - OK Check daemons - OK Check Redis - OK Check vnet - OK $ goes hget platina packages | grep version: version: b5c22f80d6b981796bdd875839325e9750beb5f0 version: 581c0a529a332cdab0e8f767130c48fd3b8b0f01 version: 60f39141fbbf78ddb2260dba74c68f2789374f18 thanks, Jason
jignesh045 commented 7 years ago

Hi Jason,

I have installed the new version in all four invaders of setup1 & got same error of vnetd daemon not responding:

root@invader29:/# goes hget platina packages | grep version:

version: b5c22f80d6b981796bdd875839325e9750beb5f0
version: 581c0a529a332cdab0e8f767130c48fd3b8b0f01
version: 60f39141fbbf78ddb2260dba74c68f2789374f18

root@invader29:/#

Steps:

  1. Configured interface IP Address as per below command:

root@invader29:/# ifconfig eth-21-1 192.168.21.29 netmask 255.255.255.0 root@invader29:/# ifconfig eth-11-1 192.168.11.29 netmask 255.255.255.0

  1. Configured below static route in Quagga.conf file in invader29.

ip route 192.168.11.31/32 eth-11-1 ip route 192.168.27.31/32 eth-11-1 ip route 192.168.5.32/32 eth-21-1 ip route 192.168.21.32/32 eth-21-1 ip route 192.168.5.30/32 eth-11-1 ip route 192.168.27.30/32 eth-11-1 ip route 192.168.5.30/32 eth-21-1 ip route 192.168.27.30/32 eth-21-1

  1. restart quagga service & observed below command:

root@invader29:/# service quagga restart root@invader29:/# vtysh -c "sh ip route" Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, A - Babel,

  • selected route, - FIB route K> 0.0.0.0/0 via 172.17.2.1, eth0 C> 127.0.0.0/8 is directly connected, lo C> 172.17.0.0/16 is directly connected, docker0 C> 172.17.2.0/23 is directly connected, eth0 C> 172.18.0.0/16 is directly connected, br-a459773cbb9a C>* 172.19.0.0/16 is directly connected, br-5966ece8cb56 S 192.168.5.30/32 [1/0] is directly connected, eth-11-1 inactive is directly connected, eth-21-1 inactive S 192.168.5.32/32 [1/0] is directly connected, eth-21-1 inactive S 192.168.11.31/32 [1/0] is directly connected, eth-11-1 inactive S 192.168.21.32/32 [1/0] is directly connected, eth-21-1 inactive S 192.168.27.30/32 [1/0] is directly connected, eth-11-1 inactive is directly connected, eth-21-1 inactive S 192.168.27.31/32 [1/0] is directly connected, eth-11-1 inactive root@invader29:/# root@invader29:/# root@invader29:/# goes status

GOES status Mode - TUNTAP PCI - OK Check daemons - OK Check Redis - OK Check vnet - Not OK status: vnetd daemon not responding root@invader29:/# root@invader29:/# root@invader29:/# vi /var/log/syslog root@invader29:/# root@invader29:/# root@invader29:/# goes status

GOES status Mode - TUNTAP PCI - OK Check daemons - OK Check Redis - OK Check vnet - Not OK status: vnetd daemon not responding root@invader29:/#

  1. Got below log in daemon.log file:

Nov 15 01:51:31 invader29 systemd[1]: Stopping LSB: start and stop the Quagga routing suite... Nov 15 01:51:31 invader29 quagga[31493]: Stopping Quagga monitor daemon: (watchquagga). Nov 15 01:51:37 invader29 quagga[31493]: Stopping Quagga daemons (prio:0): (waiting) .. bgpd (waiting) .. ospfd (waiting) .. zebra (ripd) (ripngd) (ospf6d) (isisd) (babeld). Nov 15 01:51:37 invader29 quagga[31493]: Removing all routes made by zebra. Nov 15 01:51:37 invader29 systemd[1]: Starting LSB: start and stop the Quagga routing suite... Nov 15 01:51:37 invader29 quagga[31542]: Loading capability module if not yet done. Nov 15 01:51:37 invader29 quagga[31542]: Starting Quagga daemons (prio:10): zebra bgpd ospfd. Nov 15 01:51:37 invader29 systemd[1]: Started LSB: start and stop the Quagga routing suite. Nov 15 01:51:37 invader29 goes.vnetd[26905]: panic: vnet: RTA_MULTIPATH next-hop without RTA_GATEWAY Nov 15 01:51:37 invader29 goes.vnetd[26905]: goroutine 51 [running]: Nov 15 01:51:37 invader29 goes.vnetd[26905]: runtime/debug.Stack(0xba7180, 0xc4210837f0, 0xc4214e5aa0) Nov 15 01:51:37 invader29 goes.vnetd[26905]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79 Nov 15 01:51:37 invader29 goes.vnetd[26905]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler.func1(0xc420212000, 0xc420212558) Nov 15 01:51:37 invader29 goes.vnetd[26905]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:181 +0x159 Nov 15 01:51:37 invader29 goes.vnetd[26905]: panic(0xb5e680, 0xc4210837a0) Nov 15 01:51:37 invader29 goes.vnetd[26905]: /usr/local/go/src/runtime/panic.go:489 +0x2cf Nov 15 01:51:37 invader29 goes.vnetd[26905]: github.com/platinasystems/go/vnet/unix.(net_namespace).parse_ip4_next_hops(0xc420294090, 0xc42040fa00, 0xc421a30220, 0x201e05a8c0, 0x0) Nov 15 01:51:37 invader29 goes.vnetd[26905]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:678 +0x4cf Nov 15 01:51:37 invader29 goes.vnetd[26905]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).ip4RouteMsg(0xc424449fc0, 0xc42040fa00, 0xc42040fa00, 0x1, 0x0) Nov 15 01:51:37 invader29 goes.vnetd[26905]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:706 +0xdf Nov 15 01:51:37 invader29 goes.vnetd[26905]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).EventAction(0xc424449fc0) Nov 15 01:51:37 invader29 goes.vnetd[26905]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:497 +0x5dd Nov 15 01:51:37 invader29 goes.vnetd[26905]: github.com/platinasystems/go/elib/loop.(nodeEvent).do(0xc42010e5c0) Nov 15 01:51:37 invader29 goes.vnetd[26905]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:152 +0x8f Nov 15 01:51:37 invader29 goes.vnetd[26905]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler(0xc420212000, 0x1839840, 0xc420212550) Nov 15 01:51:37 invader29 goes.vnetd[26905]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:195 +0x175 Nov 15 01:51:37 invader29 goes.vnetd[26905]: created by github.com/platinasystems/go/elib/loop.(*Node).maybeStartEventHandler Nov 15 01:51:37 invader29 goes.vnetd[26905]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:280 +0x29f Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #789 eth-22-1, fe80::5218:4cff:fe00:5ec#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #788 eth-11-1, fe80::5218:4cff:fe00:5c8#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #787 eth-25-1, fe80::5218:4cff:fe00:600#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #786 eth-14-1, fe80::5218:4cff:fe00:5cc#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #785 eth-21-1, fe80::5218:4cff:fe00:5f0#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #784 eth-3-1, fe80::5218:4cff:fe00:5a8#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #783 eth-6-1, fe80::5218:4cff:fe00:5ac#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #782 eth-13-1, fe80::5218:4cff:fe00:5d0#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #781 eth-27-1, fe80::5218:4cff:fe00:608#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #780 eth-30-1, fe80::5218:4cff:fe00:60c#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #779 eth-5-1, fe80::5218:4cff:fe00:5b0#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #778 eth-23-1, fe80::5218:4cff:fe00:5f8#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #777 eth-26-1, fe80::5218:4cff:fe00:5fc#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #776 eth-15-1, fe80::5218:4cff:fe00:5d8#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #775 eth-29-1, fe80::5218:4cff:fe00:610#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #774 eth-18-1, fe80::5218:4cff:fe00:5dc#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #773 eth-7-1, fe80::5218:4cff:fe00:5b8#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #772 eth-10-1, fe80::5218:4cff:fe00:5bc#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #771 eth-17-1, fe80::5218:4cff:fe00:5e0#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #770 eth-31-1, fe80::5218:4cff:fe00:618#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #769 eth-9-1, fe80::5218:4cff:fe00:5c0#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #768 eth-1-1, fe80::5218:4cff:fe00:5a0#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #767 eth-19-1, fe80::5218:4cff:fe00:5e8#123, interface stats: received=0, sent=0, dropped=0, active_time=6000 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #736 eth-30-1, 10.0.30.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #735 eth-23-1, 10.0.23.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #734 eth-17-1, 10.0.17.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #733 eth-10-1, 10.0.10.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #732 eth-24-1, 10.0.24.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #731 eth-13-1, 10.0.13.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #730 eth-7-1, 10.0.7.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #729 eth-8-1, 10.0.8.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #728 eth-6-1, 10.0.6.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #727 eth-27-1, 10.0.27.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #726 eth-25-1, 10.0.25.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #725 eth-26-1, 10.0.26.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #724 eth-14-1, 10.0.14.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #723 eth-5-1, 10.0.5.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #722 eth-31-1, 10.0.31.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #721 eth-29-1, 10.0.29.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #720 eth-11-1, 10.0.11.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #719 eth-12-1, 10.0.12.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #718 eth-32-1, 10.0.32.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #717 eth-21-1, 10.0.21.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #716 eth-19-1, 10.0.19.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #715 eth-4-1, 10.0.4.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #714 eth-2-1, 10.0.2.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #713 eth-18-1, 10.0.18.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #712 eth-1-1, 10.0.1.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #711 eth-28-1, 10.0.28.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #710 eth-20-1, 10.0.20.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #709 eth-16-1, 10.0.16.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #708 eth-9-1, 10.0.9.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #707 eth-3-1, 10.0.3.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #706 eth-22-1, 10.0.22.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: Deleting interface #705 eth-15-1, 10.0.15.29#123, interface stats: received=0, sent=0, dropped=0, active_time=16500 secs Nov 15 01:53:20 invader29 ntpd[674]: peers refreshed

jasonlpang commented 7 years ago

Hi Jignesh, In the Quagga.conf file is there a way to specific the default gateway for the routes to be added? For netlink, looks like adding a route needs the gateway , destination address, and the interface.

thanks jason

stigt commented 7 years ago

Typically interface routes are used on point-to-point interfaces since the only destination is the other end. An interface route will work on an ethernet if it's in the same subnet as that interface, but if not it will just get a failed arp. When we add support for tunnels, then we'll need to support interface routes.

On Wed, Nov 15, 2017 at 2:04 PM, jasonlpang notifications@github.com wrote:

Hi Jignesh, In the Quagga.conf file is there a way to specific the default gateway for the routes to be added? For netlink, looks like adding a route needs the gateway , destination address, and the interface.

thanks jason

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/platinasystems/go/issues/88#issuecomment-344743856, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQZoDbzOuszyYonU2hmcpwoMApqnLEaks5s21_6gaJpZM4QdVW7 .

jignesh045 commented 7 years ago

Hi Jason,

In quagga (version: 0.99.23.1), It has way to either configure gateway or interface for static route. Look at below available option when we try to configure static route with gateway/interface through vtysh terminal in quagga

root@invader29:/# vtysh

Hello, this is Quagga (version 0.99.23.1). Copyright 1996-2005 Kunihiro Ishiguro, et al.

invader29# conf t invader29(config)# invader29(config)# ip route A.B.C.D IP destination prefix A.B.C.D/M IP destination prefix (e.g. 10.0.0.0/8) invader29(config)# invader29(config)# ip route 192.168.27.30/32 A.B.C.D IP gateway address INTERFACE IP gateway interface name blackhole Silently discard pkts when matched null0 Null interface reject Emit an ICMP unreachable when matched invader29(config)# ip route 192.168.27.30/32 192.168.11.31

<1-255> Distance value for this route blackhole Silently discard pkts when matched reject Emit an ICMP unreachable when matched invader29(config)# ip route 192.168.27.30/32 eth-11-1 <1-255> Distance value for this route blackhole Silently discard pkts when matched reject Emit an ICMP unreachable when matched invader29(config)# exit invader29# invader29# invader29# exit root@invader29:/#
jasonlpang commented 7 years ago

Discussion with Stig: Jason,

I checked the code and do see it's a deliberate panic in our code:

                    if gw := nh.attrs[netlink.RTA_GATEWAY]; gw != nil {
            set_ip4_next_hop_address(gw, &nh.NextHop)
        } else {
            panic("RTA_MULTIPATH next-hop without RTA_GATEWAY")
        }

It looks like the netlink code doesn't handle interface routes (i.e. no gateway).

stig

jignesh045 commented 6 years ago

I have recreate this issue with the following test case: Regress_GeneralRouting_BGP_FRR_BGP Peering_EBGP_loopback

Test case objective: To check ping response in between all invaders after establishing eBGP neighborship using loopback interface (i have used gateway route instead of interface route)

Below are the detail test case steps:

Test case description: Regress_GeneralRouting_BGP_FRR_BGP Peering_EBGP_loopback

This test case covers eBGP peering through loopback interface i.e. (1) Between Invader45 (Leaf) & Invader43 (Spine) via eth-3-1 (2) Between Invader45 (Leaf) & Invader42 (Spine) via eth-19-1 (3) Between Invader44 (Leaf) & Invader42 (Spine) via eth-3-1 (4) Between Invader44 (Leaf) & Invader43 (Spine) via eth-19-1.

Precondition:

1. BGPD process should be running using frr in all devices
2. router terminal should be accessible using vtysh command in all devices

Steps:

1) login to Invader44 & Backup BGP configuration files "/etc/frr/bgpd.conf" & "/etc/frr/zebra.conf" to replace it after execution of test case

2) Do same step as mentioned in step 1 for Invader45 & Invader42 & Invader43

For updating BGP configuration command in bgpd.conf file (Step 3,4,5,6) in all four devices, insert the BGP command in conf file in such a way so that it looks like (ex. for Invader43)

root@Invader43:/# cat /etc/frr/bgpd.conf ! (output cut) log stdout ! ! router bgp 65243 bgp router-id 172.17.2.43 neighbor 192.168.44.1 remote-as 65244 neighbor 192.168.44.1 ebgp-multihop 2 neighbor 192.168.44.1 update-source dummy0 neighbor 192.168.45.1 remote-as 65245 neighbor 192.168.45.1 ebgp-multihop 2 neighbor 192.168.45.1 update-source dummy0 network 192.168.43.0/24 ! line vty ! root@Invader43:/#

root@Invader43:/# cat /etc/frr/zebra.conf ! (output cut) interface vnet ! ip route 192.168.44.0/24 10.0.19.30 ip route 192.168.45.0/24 10.0.3.29 ! ! ! line vty ! root@Invader43:/#

3) login to Invader44

(a) apply the following command in Invader44

root@invader44:/# ip link add dummy0 type dummy root@invader44:/# ifconfig dummy0 192.168.44.1 netmask 255.255.255.0

(b) append below BGP Configuration in /etc/frr/bgpd.conf file as discussed above.

router bgp 65244 bgp router-id 172.17.2.44 neighbor 192.168.43.1 remote-as 65243 neighbor 192.168.43.1 ebgp-multihop 2 neighbor 192.168.43.1 update-source dummy0 neighbor 192.168.42.1 remote-as 65242 neighbor 192.168.42.1 ebgp-multihop 2 neighbor 192.168.42.1 update-source dummy0 network 192.168.44.0/24 !

(c) append below BGP Configuration in /etc/frr/zebra.conf in between interface vnet & line vty command

ip route 192.168.42.0/24 10.0.3.32 ip route 192.168.43.0/24 10.0.19.31

4) login to Invader45

(a) apply the following command in Invader45

root@invader45:/# ip link add dummy0 type dummy root@invader45:/# ifconfig dummy0 192.168.45.1 netmask 255.255.255.0

(b) append below BGP Configuration in /etc/frr/bgpd.conf file as discussed above.

router bgp 65245 bgp router-id 172.17.2.45 neighbor 192.168.43.1 remote-as 65243 neighbor 192.168.43.1 ebgp-multihop 2 neighbor 192.168.43.1 update-source dummy0 neighbor 192.168.42.1 remote-as 65242 neighbor 192.168.42.1 ebgp-multihop 2 neighbor 192.168.42.1 update-source dummy0 network 192.168.45.0/24

(c) append below BGP Configuration in /etc/frr/zebra.conf in between interface vnet & line vty command

ip route 192.168.42.0/24 10.0.19.32 ip route 192.168.43.0/24 10.0.3.31

5) login to Invader43

(a) apply the following command in Invader43

root@invader43:/# ip link add dummy0 type dummy root@invader43:/# ifconfig dummy0 192.168.43.1 netmask 255.255.255.0

(b) append below BGP Configuration in /etc/frr/bgpd.conf file as discussed above.

router bgp 65243 bgp router-id 172.17.2.43 neighbor 192.168.45.1 remote-as 65245 neighbor 192.168.45.1 ebgp-multihop 2 neighbor 192.168.45.1 update-source dummy0 neighbor 192.168.44.1 remote-as 65244 neighbor 192.168.44.1 ebgp-multihop 2 neighbor 192.168.44.1 update-source dummy0 network 192.168.43.0/24

(c) append below BGP Configuration in /etc/frr/zebra.conf in between interface vnet & line vty command

ip route 192.168.44.0/24 10.0.19.30 ip route 192.168.45.0/24 10.0.3.29

6) login to Invader42

(a) apply the following command in Invader42

root@invader42:/# ip link add dummy0 type dummy root@invader42:/# ifconfig dummy0 192.168.42.1 netmask 255.255.255.0

(b) append below BGP Configuration in /etc/frr/bgpd.conf file as discussed above.

router bgp 65242 bgp router-id 172.17.2.42 neighbor 192.168.45.1 remote-as 65245 neighbor 192.168.45.1 ebgp-multihop 2 neighbor 192.168.45.1 update-source dummy0 neighbor 192.168.44.1 remote-as 65244 neighbor 192.168.44.1 ebgp-multihop 2 neighbor 192.168.44.1 update-source dummy0 network 192.168.42.0/24

(c) append below BGP Configuration in /etc/frr/zebra.conf in between interface vnet & line vty command

ip route 192.168.44.0/24 10.0.3.30 ip route 192.168.45.0/24 10.0.19.29

7) login to Invader45 & restart frr service using following root command & make sure it's restarting without fail as shown below:

root@Invader45:/# service frr restart root@Invader45:/#

root@invader45:/# service frr status ● frr.service - FRRouting Loaded: loaded (/lib/systemd/system/frr.service; enabled) Active: active (running) since Mon 2017-12-11 22:23:34 PST; 8s ago Process: 23953 ExecStop=/usr/lib/frr/frr stop (code=exited, status=0/SUCCESS) Process: 24032 ExecStart=/usr/lib/frr/frr start (code=exited, status=0/SUCCESS) CGroup: /system.slice/frr.service ├─24053 /usr/lib/frr/zebra -s 90000000 --daemon -A 127.0.0.1 ├─24060 /usr/lib/frr/bgpd --daemon -A 127.0.0.1 ├─24067 /usr/lib/frr/ospfd --daemon -A 127.0.0.1 ├─24074 /usr/lib/frr/isisd --daemon -A 127.0.0.1 └─24081 /usr/lib/frr/watchfrr -adz -r /usr/sbin/servicebBfrrbBrestartbB%s -s /usr/sbin/servicebBfrrbBstartbB%s -k /usr/sbin/servicebBfrrbBstopbB%s -b bB -t 30 zebra bgpd ospfd isisd

Dec 11 22:23:33 invader45 frr[24032]: . isisd. Dec 11 22:23:33 invader45 watchfrr[24081]: watchfrr 3.0.2 watching [zebra bgpd ospfd isisd], mode [phased zebra restart] Dec 11 22:23:34 invader45 watchfrr[24081]: isisd state -> up : connect succeeded Dec 11 22:23:34 invader45 watchfrr[24081]: bgpd state -> up : connect succeeded Dec 11 22:23:34 invader45 watchfrr[24081]: ospfd state -> up : connect succeeded Dec 11 22:23:34 invader45 watchfrr[24081]: zebra state -> up : connect succeeded Dec 11 22:23:34 invader45 watchfrr[24081]: Watchfrr: Notifying Systemd we are up and running Dec 11 22:23:34 invader45 frr[24032]: Starting Frr monitor daemon: watchfrr. Dec 11 22:23:34 invader45 frr[24032]: Exiting from the script Dec 11 22:23:34 invader45 systemd[1]: Started FRRouting. root@invader45:/#

8) Do same step as mentioned in step 7 for Invader44, Invader42 & Invader43

9) Wait for 35 second for BGP convergence & then verify expected result. Once expected result is verified, replace all original files which were back up in earlier steps as mentioned in step 1,2 for Invader44, Invader45, Invader43 & Invader42.

10) Repeat Step 7 (restart frr service) for all four devices.

Expected Results: Ping should work in between all invaders

(a) To verify expected result in Invader44

1. login to Invader44
2. In root user, apply the command: vtysh -c "sh ip route" & observe route for all dummy0 interfaces
3. ping all other invaders

root@invader44:/# vtysh -c "sh ip route" Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, P - PIM, N - NHRP, T - Table, v - VNC, V - VNC-Direct,

  • selected route, * - FIB route

K> 0.0.0.0/0 via 172.17.2.1, eth0 C> 10.0.1.0/24 is directly connected, eth-1-1 C> 10.0.2.0/24 is directly connected, eth-2-1 C> 10.0.3.0/24 is directly connected, eth-3-1 C> 10.0.4.0/24 is directly connected, eth-4-1 C> 10.0.5.0/24 is directly connected, eth-5-1 C> 10.0.6.0/24 is directly connected, eth-6-1 C> 10.0.7.0/24 is directly connected, eth-7-1 C> 10.0.8.0/24 is directly connected, eth-8-1 C> 10.0.9.0/24 is directly connected, eth-9-1 C> 10.0.11.0/24 is directly connected, eth-11-1 C> 10.0.12.0/24 is directly connected, eth-12-1 C> 10.0.13.0/24 is directly connected, eth-13-1 C> 10.0.14.0/24 is directly connected, eth-14-1 C> 10.0.15.0/24 is directly connected, eth-15-1 C> 10.0.16.0/24 is directly connected, eth-16-1 C> 10.0.17.0/24 is directly connected, eth-17-1 C> 10.0.18.0/24 is directly connected, eth-18-1 C> 10.0.19.0/24 is directly connected, eth-19-1 C> 10.0.20.0/24 is directly connected, eth-20-1 C> 10.0.21.0/24 is directly connected, eth-21-1 C> 10.0.22.0/24 is directly connected, eth-22-1 C> 10.0.23.0/24 is directly connected, eth-23-1 C> 10.0.24.0/24 is directly connected, eth-24-1 C> 10.0.25.0/24 is directly connected, eth-25-1 C> 10.0.27.0/24 is directly connected, eth-27-1 C> 10.0.29.0/24 is directly connected, eth-29-1 C> 10.0.31.0/24 is directly connected, eth-31-1 C> 172.17.2.0/23 is directly connected, eth0 B 192.168.42.0/24 [20/0] via 192.168.42.1 inactive, 00:30:23 S> 192.168.42.0/24 [1/0] via 10.0.3.32, eth-3-1 B 192.168.43.0/24 [20/0] via 192.168.43.1 inactive, 00:29:14 S> 192.168.43.0/24 [1/0] via 10.0.19.31, eth-19-1 C> 192.168.44.0/24 is directly connected, dummy0 B> 192.168.45.0/24 [20/0] via 192.168.42.1 (recursive), 00:06:22

root@invader44:/# ping 192.168.42.1 PING 192.168.42.1 (192.168.42.1) 56(84) bytes of data. 64 bytes from 192.168.42.1: icmp_seq=1 ttl=64 time=1.19 ms 64 bytes from 192.168.42.1: icmp_seq=2 ttl=64 time=1.09 ms 64 bytes from 192.168.42.1: icmp_seq=3 ttl=64 time=1.11 ms ^C --- 192.168.42.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.098/1.137/1.196/0.050 ms root@invader44:/# root@invader44:/# ping 192.168.43.1 PING 192.168.43.1 (192.168.43.1) 56(84) bytes of data. 64 bytes from 192.168.43.1: icmp_seq=1 ttl=64 time=0.787 ms 64 bytes from 192.168.43.1: icmp_seq=2 ttl=64 time=0.985 ms 64 bytes from 192.168.43.1: icmp_seq=3 ttl=64 time=1.01 ms ^C --- 192.168.43.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2028ms rtt min/avg/max/mdev = 0.787/0.928/1.012/0.100 ms root@invader44:/# root@invader44:/# ping 192.168.45.1 PING 192.168.45.1 (192.168.45.1) 56(84) bytes of data. ^C --- 192.168.45.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2051ms

root@invader44:/#

(b) To verify expected result in Invader45

1. login to Invader45
2. In root user, apply the command: vtysh -c "sh ip route" & observe route for all dummy0 interfaces
3. ping all other invaders

root@invader45:/# vtysh -c "sh ip route" Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, P - PIM, N - NHRP, T - Table, v - VNC, V - VNC-Direct,

  • selected route, * - FIB route

K> 0.0.0.0/0 via 172.17.2.1, eth0 C> 10.0.1.0/24 is directly connected, eth-1-1 C> 10.0.2.0/24 is directly connected, eth-2-1 C> 10.0.3.0/24 is directly connected, eth-3-1 C> 10.0.4.0/24 is directly connected, eth-4-1 C> 10.0.5.0/24 is directly connected, eth-5-1 C> 10.0.6.0/24 is directly connected, eth-6-1 C> 10.0.7.0/24 is directly connected, eth-7-1 C> 10.0.8.0/24 is directly connected, eth-8-1 C> 10.0.9.0/24 is directly connected, eth-9-1 C> 10.0.11.0/24 is directly connected, eth-11-1 C> 10.0.13.0/24 is directly connected, eth-13-1 C> 10.0.15.0/24 is directly connected, eth-15-1 C> 10.0.17.0/24 is directly connected, eth-17-1 C> 10.0.18.0/24 is directly connected, eth-18-1 C> 10.0.19.0/24 is directly connected, eth-19-1 C> 10.0.20.0/24 is directly connected, eth-20-1 C> 10.0.21.0/24 is directly connected, eth-21-1 C> 10.0.22.0/24 is directly connected, eth-22-1 C> 10.0.23.0/24 is directly connected, eth-23-1 C> 10.0.24.0/24 is directly connected, eth-24-1 C> 10.0.25.0/24 is directly connected, eth-25-1 C> 10.0.26.0/24 is directly connected, eth-26-1 C> 10.0.27.0/24 is directly connected, eth-27-1 C> 10.0.28.0/24 is directly connected, eth-28-1 C> 10.0.29.0/24 is directly connected, eth-29-1 C> 10.0.30.0/24 is directly connected, eth-30-1 C> 10.0.31.0/24 is directly connected, eth-31-1 C> 10.0.32.0/24 is directly connected, eth-32-1 C> 172.17.2.0/23 is directly connected, eth0 B 192.168.42.0/24 [20/0] via 192.168.42.1 inactive, 00:06:16 S> 192.168.42.0/24 [1/0] via 10.0.19.32, eth-19-1 B 192.168.43.0/24 [20/0] via 192.168.43.1 inactive, 00:06:16 S> 192.168.43.0/24 [1/0] via 10.0.3.31, eth-3-1 B> 192.168.44.0/24 [20/0] via 192.168.42.1 (recursive), 00:06:16

root@invader45:/# ping 192.168.42.1 PING 192.168.42.1 (192.168.42.1) 56(84) bytes of data. 64 bytes from 192.168.42.1: icmp_seq=1 ttl=64 time=1.40 ms 64 bytes from 192.168.42.1: icmp_seq=2 ttl=64 time=1.21 ms 64 bytes from 192.168.42.1: icmp_seq=3 ttl=64 time=1.17 ms ^C --- 192.168.42.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.178/1.265/1.407/0.108 ms root@invader45:/# root@invader45:/# ping 192.168.43.1 PING 192.168.43.1 (192.168.43.1) 56(84) bytes of data. 64 bytes from 192.168.43.1: icmp_seq=1 ttl=64 time=1.06 ms 64 bytes from 192.168.43.1: icmp_seq=2 ttl=64 time=1.00 ms 64 bytes from 192.168.43.1: icmp_seq=3 ttl=64 time=0.947 ms ^C --- 192.168.43.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.947/1.004/1.063/0.053 ms root@invader45:/# root@invader45:/# ping 192.168.44.1 PING 192.168.44.1 (192.168.44.1) 56(84) bytes of data. ^C --- 192.168.44.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2025ms

root@invader45:/#

(c) To verify expected result in Invader43

1. login to Invader43
2. In root user, apply the command: vtysh -c "sh ip route" & observe route for all dummy0 interfaces
3. ping all other invaders

root@invader43:/# vtysh -c "sh ip route" Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, P - PIM, N - NHRP, T - Table, v - VNC, V - VNC-Direct,

  • selected route, * - FIB route

K> 0.0.0.0/0 via 172.17.2.1, eth0 C> 10.0.1.0/24 is directly connected, eth-1-1 C> 10.0.3.0/24 is directly connected, eth-3-1 C> 10.0.5.0/24 is directly connected, eth-5-1 C> 10.0.7.0/24 is directly connected, eth-7-1 C> 10.0.9.0/24 is directly connected, eth-9-1 C> 10.0.11.0/24 is directly connected, eth-11-1 C> 10.0.13.0/24 is directly connected, eth-13-1 C> 10.0.15.0/24 is directly connected, eth-15-1 C> 10.0.17.0/24 is directly connected, eth-17-1 C> 10.0.19.0/24 is directly connected, eth-19-1 C> 10.0.21.0/24 is directly connected, eth-21-1 C> 10.0.23.0/24 is directly connected, eth-23-1 C> 10.0.25.0/24 is directly connected, eth-25-1 C> 10.0.27.0/24 is directly connected, eth-27-1 C> 10.0.29.0/24 is directly connected, eth-29-1 C> 10.0.31.0/24 is directly connected, eth-31-1 C> 172.17.2.0/23 is directly connected, eth0 B> 192.168.42.0/24 [20/0] via 192.168.44.1 (recursive), 00:34:46

root@invader43:/# root@invader43:/# ping 192.168.44.1 PING 192.168.44.1 (192.168.44.1) 56(84) bytes of data. 64 bytes from 192.168.44.1: icmp_seq=1 ttl=64 time=0.774 ms 64 bytes from 192.168.44.1: icmp_seq=2 ttl=64 time=0.910 ms 64 bytes from 192.168.44.1: icmp_seq=3 ttl=64 time=0.919 ms ^C --- 192.168.44.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2014ms rtt min/avg/max/mdev = 0.774/0.867/0.919/0.074 ms root@invader43:/# root@invader43:/# ping 192.168.45.1 PING 192.168.45.1 (192.168.45.1) 56(84) bytes of data. 64 bytes from 192.168.45.1: icmp_seq=1 ttl=64 time=1.13 ms 64 bytes from 192.168.45.1: icmp_seq=2 ttl=64 time=1.06 ms 64 bytes from 192.168.45.1: icmp_seq=3 ttl=64 time=1.10 ms ^C --- 192.168.45.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 1.065/1.102/1.135/0.039 ms root@invader43:/#

(d) To verify expected result in Invader42

1. login to Invader42
2. In root user, apply the command: vtysh -c "sh ip route" & observe route for all dummy0 interfaces
3. ping all other invaders

root@invader42:/# vtysh -c "sh ip route" Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, P - PIM, N - NHRP, T - Table, v - VNC, V - VNC-Direct,

  • selected route, * - FIB route

K> 0.0.0.0/0 via 172.17.2.1, eth0 C> 10.0.1.0/24 is directly connected, eth-1-1 C> 10.0.3.0/24 is directly connected, eth-3-1 C> 10.0.5.0/24 is directly connected, eth-5-1 C> 10.0.7.0/24 is directly connected, eth-7-1 C> 10.0.9.0/24 is directly connected, eth-9-1 C> 10.0.11.0/24 is directly connected, eth-11-1 C> 10.0.13.0/24 is directly connected, eth-13-1 C> 10.0.15.0/24 is directly connected, eth-15-1 C> 10.0.17.0/24 is directly connected, eth-17-1 C> 10.0.19.0/24 is directly connected, eth-19-1 C> 10.0.21.0/24 is directly connected, eth-21-1 C> 10.0.23.0/24 is directly connected, eth-23-1 C> 10.0.25.0/24 is directly connected, eth-25-1 C> 10.0.27.0/24 is directly connected, eth-27-1 C> 10.0.29.0/24 is directly connected, eth-29-1 C> 10.0.31.0/24 is directly connected, eth-31-1 C> 172.17.2.0/23 is directly connected, eth0 C>* 192.168.42.0/24 is directly connected, dummy0 B> 192.168.43.0/24 [20/0] via 192.168.44.1 (recursive), 00:35:40

root@invader42:/# root@invader42:/# ping 192.168.44.1 PING 192.168.44.1 (192.168.44.1) 56(84) bytes of data. 64 bytes from 192.168.44.1: icmp_seq=1 ttl=64 time=1.20 ms 64 bytes from 192.168.44.1: icmp_seq=2 ttl=64 time=1.04 ms 64 bytes from 192.168.44.1: icmp_seq=3 ttl=64 time=1.07 ms ^C --- 192.168.44.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.046/1.108/1.200/0.066 ms root@invader42:/# root@invader42:/# ping 192.168.45.1 PING 192.168.45.1 (192.168.45.1) 56(84) bytes of data. 64 bytes from 192.168.45.1: icmp_seq=1 ttl=64 time=1.33 ms 64 bytes from 192.168.45.1: icmp_seq=2 ttl=64 time=1.12 ms 64 bytes from 192.168.45.1: icmp_seq=3 ttl=64 time=1.14 ms ^C --- 192.168.45.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.120/1.199/1.337/0.105 ms root@invader42:/#

jignesh045 commented 6 years ago

Issue verification Status: Verified Issue Status: Case 1- Passed, Case 2- Failed

Pre-requisite: FRR package should be installed in all invaders

Issue verify in below build (in all setup2 invaders)

root@invader45:~# goes version
github.com/platinasystems/go: v0.41-244-g49d9e6da
github.com/platinasystems/fe1: v0.2-520-g85494c1
github.com/platinasystems/firmware-fe1a: v0.2
root@invader45:~#

root@invader45:~# goes hget platina-mk1 packages | grep version:
    version: 49d9e6dae2471c7d8f20f88997cd3b4578dee36a
    version: 85494c17b94de997f1d2dac7a0cf21a4e76343f0
    version: 60f39141fbbf78ddb2260dba74c68f2789374f18
root@invader45:~#

root@invader45:~# dpkg --list | grep kernel
ii  kmod                           18-3                             amd64        tools for managing Linux kernel modules
ii  libdrm2:amd64                  2.4.58-2                         amd64        Userspace interface to kernel DRM services -- runtime
ii  linux-image-4.13.0-platina-mk1 4.13-117-g00814e4e1a1e           amd64        Linux kernel, version 4.13.0-platina-mk1
ii  rsyslog                        8.4.2-1+deb8u2                   amd64        reliable system and kernel logging daemon
root@invader45:~#

Steps:

Case 1: Static route added (to established BGP neighborship) via Linux command prompt

  1. Executed below playbook (only displaying required output)
abhishek@invader28:~/latest/go/ansible/regtest$ ansible-playbook -i hosts_testbed2 playbooks/bgp/frr_bgp_peering_ebgp_loopback.yml -K -e reset_config=False
SUDO password:
[DEPRECATION WARNING]: 'include' for playbook includes. You should use 'import_playbook' instead. This feature will be removed in version 2.8. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.

PLAY [leaf:spine] ***************************************************************************************************************************************************************************************
...
...

TASK [Add static routes] ********************************************************************************************************************************************************************************
ok: [invader45]
ok: [invader44]
ok: [invader43]
ok: [invader42]

TASK [Add bgp config in /etc/frr/bgpd.conf] *************************************************************************************************************************************************************
changed: [invader45]
changed: [invader44]
changed: [invader43]
changed: [invader42]

TASK [Restart frr] **************************************************************************************************************************************************************************************
changed: [invader45]
changed: [invader44]
changed: [invader43]
changed: [invader42]

TASK [Wait for 35 secs for bgp convergence] *************************************************************************************************************************************************************
Pausing for 35 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [invader45]

TASK [file] *********************************************************************************************************************************************************************************************
ok: [invader45]
ok: [invader44]
ok: [invader43]
ok: [invader42]

...
...

TASK [Print test case result] ***************************************************************************************************************************************************************************
ok: [server_emulator] => {
    "msg": "Test case result: Passed"
}

TASK [Print failure summary] ****************************************************************************************************************************************************************************
ok: [server_emulator] => {
    "msg": "Failure summary: None"
}

TASK [lineinfile] ***************************************************************************************************************************************************************************************
changed: [server_emulator]

PLAY [leaf:spine] ***************************************************************************************************************************************************************************************

TASK [Get goes status] **********************************************************************************************************************************************************************************
changed: [invader45]
changed: [invader44]
changed: [invader43]
changed: [invader42]

TASK [Print goes status] ********************************************************************************************************************************************************************************
ok: [invader45] => {
    "goes.stdout": "GOES status\n======================\n  Mode            - SRIOV\n  PCI             - OK\n  Check daemons   - OK\n  Check Redis     - OK\n  Check vnet      - OK"
}
ok: [invader44] => {
    "goes.stdout": "GOES status\n======================\n  Mode            - SRIOV\n  PCI             - OK\n  Check daemons   - OK\n  Check Redis     - OK\n  Check vnet      - OK"
}
ok: [invader43] => {
    "goes.stdout": "GOES status\n======================\n  Mode            - SRIOV\n  PCI             - OK\n  Check daemons   - OK\n  Check Redis     - OK\n  Check vnet      - OK"
}
ok: [invader42] => {
    "goes.stdout": "GOES status\n======================\n  Mode            - SRIOV\n  PCI             - OK\n  Check daemons   - OK\n  Check Redis     - OK\n  Check vnet      - OK"
}

PLAY RECAP **********************************************************************************************************************************************************************************************
invader42                  : ok=18   changed=8    unreachable=0    failed=0
invader43                  : ok=18   changed=8    unreachable=0    failed=0
invader44                  : ok=18   changed=8    unreachable=0    failed=0
invader45                  : ok=19   changed=8    unreachable=0    failed=0
server_emulator            : ok=7    changed=2    unreachable=0    failed=0

abhishek@invader28:~/latest/go/ansible/regtest$
  1. Did not observed any GoES failure
  2. Reset the setup configuration via playbook:
abhishek@invader28:~/latest/go/ansible/regtest$ ansible-playbook -i hosts_testbed2 playbooks/bgp/frr_bgp_peering_ebgp_loopback_reset.yml -K
SUDO password:

PLAY [leaf:spine] ***************************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************
ok: [invader42]
ok: [invader44]
ok: [invader43]
ok: [invader45]

TASK [Restore backup of /etc/frr/bgpd.conf] *************************************************************************************************************************************************************
changed: [invader45]
changed: [invader44]
changed: [invader43]
changed: [invader42]

TASK [Remove backup file] *******************************************************************************************************************************************************************************
changed: [invader44]
changed: [invader45]
changed: [invader43]
changed: [invader42]

TASK [Delete static routes] *****************************************************************************************************************************************************************************
ok: [invader44]
ok: [invader45]
ok: [invader43]
ok: [invader42]

TASK [Delete dummy interface] ***************************************************************************************************************************************************************************
changed: [invader45]
changed: [invader44]
changed: [invader43]
changed: [invader42]

TASK [Restart frr] **************************************************************************************************************************************************************************************
changed: [invader44]
changed: [invader45]
changed: [invader43]
changed: [invader42]

PLAY RECAP **********************************************************************************************************************************************************************************************
invader42                  : ok=6    changed=4    unreachable=0    failed=0
invader43                  : ok=6    changed=4    unreachable=0    failed=0
invader44                  : ok=6    changed=4    unreachable=0    failed=0
invader45                  : ok=6    changed=4    unreachable=0    failed=0

abhishek@invader28:~/latest/go/ansible/regtest$

Case 2: Static route added via zebra.conf (/etc/frr/zebra.conf) file (Checked without executing any BGP test case).

  1. Add any static route via VTYSH console

root@invader44:~# vtysh

Hello, this is FRRouting (version 3.0.3). Copyright 1996-2005 Kunihiro Ishiguro, et al.

invader44# conf t
invader44(config)#
invader44(config)# ip route 192.168.11.31/32 eth-11-1
invader44(config)# ip route 192.168.27.31/32 eth-11-1
invader44(config)# ip route 192.168.5.32/32 eth-21-1
invader44(config)# ip route 192.168.21.32/32 eth-21-1
invader44(config)# ip route 192.168.5.30/32 eth-11-1
invader44(config)# ip route 192.168.27.30/32 eth-11-1
invader44(config)# ip route 192.168.5.30/32 eth-21-1
invader44(config)# ip route 192.168.27.30/32 eth-21-1
invader44(config)#
  1. save the configuration
invader44(config)# do wr
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Configuration saved to /etc/frr//zebra.conf
Configuration saved to /etc/frr//ospfd.conf
Configuration saved to /etc/frr//bgpd.conf
Configuration saved to /etc/frr//isisd.conf
invader44(config)#
invader44(config)# exit
invader44#
  1. check the ip route in vtysh terminal
invader44# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, P - PIM, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct,
       > - selected route, * - FIB route

K>* 0.0.0.0/0 via 172.17.2.1, eth0
C>* 172.17.2.0/23 is directly connected, eth0
S   192.168.5.30/32 [1/0] is directly connected, eth-11-1 inactive
                          is directly connected, eth-21-1 inactive
S   192.168.5.32/32 [1/0] is directly connected, eth-21-1 inactive
S   192.168.11.31/32 [1/0] is directly connected, eth-11-1 inactive
S   192.168.21.32/32 [1/0] is directly connected, eth-21-1 inactive
S   192.168.27.30/32 [1/0] is directly connected, eth-11-1 inactive
                           is directly connected, eth-21-1 inactive
S   192.168.27.31/32 [1/0] is directly connected, eth-11-1 inactive
invader44#
  1. Check the configuration stored in zebra.conf file (Only displaying required output)
root@invader44:~# cat /etc/frr/zebra.conf
!
! Zebra configuration saved from vty
!   2018/07/27 03:45:20
!
frr version 3.0.3
frr defaults traditional
!
hostname Router
password zebra
enable password zebra
!
!
...
...
interface lo
!
ip route 192.168.5.30/32 eth-11-1
ip route 192.168.5.30/32 eth-21-1
ip route 192.168.5.32/32 eth-21-1
ip route 192.168.11.31/32 eth-11-1
ip route 192.168.21.32/32 eth-21-1
ip route 192.168.27.30/32 eth-11-1
ip route 192.168.27.30/32 eth-21-1
ip route 192.168.27.31/32 eth-11-1
!
!
!
line vty
!
root@invader44:~#
  1. routes are not displaying in 'route' command output (may be vnetd crashed at that time)
root@invader44:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.2.1      0.0.0.0         UG    0      0        0 eth0
10.0.1.0        *               255.255.255.0   U     0      0        0 eth-1-1
10.0.2.0        *               255.255.255.0   U     0      0        0 eth-2-1
10.0.3.0        *               255.255.255.0   U     0      0        0 eth-3-1
10.0.4.0        *               255.255.255.0   U     0      0        0 eth-4-1
10.0.5.0        *               255.255.255.0   U     0      0        0 eth-5-1
10.0.6.0        *               255.255.255.0   U     0      0        0 eth-6-1
10.0.7.0        *               255.255.255.0   U     0      0        0 eth-7-1
10.0.8.0        *               255.255.255.0   U     0      0        0 eth-8-1
10.0.9.0        *               255.255.255.0   U     0      0        0 eth-9-1
10.0.10.0       *               255.255.255.0   U     0      0        0 eth-10-1
10.0.11.0       *               255.255.255.0   U     0      0        0 eth-11-1
10.0.12.0       *               255.255.255.0   U     0      0        0 eth-12-1
10.0.13.0       *               255.255.255.0   U     0      0        0 eth-13-1
10.0.14.0       *               255.255.255.0   U     0      0        0 eth-14-1
10.0.15.0       *               255.255.255.0   U     0      0        0 eth-15-1
10.0.16.0       *               255.255.255.0   U     0      0        0 eth-16-1
10.0.17.0       *               255.255.255.0   U     0      0        0 eth-17-1
10.0.18.0       *               255.255.255.0   U     0      0        0 eth-18-1
10.0.19.0       *               255.255.255.0   U     0      0        0 eth-19-1
10.0.20.0       *               255.255.255.0   U     0      0        0 eth-20-1
10.0.21.0       *               255.255.255.0   U     0      0        0 eth-21-1
10.0.22.0       *               255.255.255.0   U     0      0        0 eth-22-1
10.0.23.0       *               255.255.255.0   U     0      0        0 eth-23-1
10.0.24.0       *               255.255.255.0   U     0      0        0 eth-24-1
10.0.25.0       *               255.255.255.0   U     0      0        0 eth-25-1
10.0.26.0       *               255.255.255.0   U     0      0        0 eth-26-1
10.0.27.0       *               255.255.255.0   U     0      0        0 eth-27-1
10.0.28.0       *               255.255.255.0   U     0      0        0 eth-28-1
10.0.29.0       *               255.255.255.0   U     0      0        0 eth-29-1
10.0.30.0       *               255.255.255.0   U     0      0        0 eth-30-1
10.0.31.0       *               255.255.255.0   U     0      0        0 eth-31-1
10.0.32.0       *               255.255.255.0   U     0      0        0 eth-32-1
172.17.2.0      *               255.255.254.0   U     0      0        0 eth0
172.18.0.0      *               255.255.0.0     U     0      0        0 docker0
root@invader44:~#
  1. Check FIB entry & GoES status
root@invader44:~# goes vnet show ip fib
vnet: dial unix @platina-mk1/vnet: connect: connection refused
root@invader44:~#
root@invader44:~# goes status
GOES status
======================
  Mode            - SRIOV
  PCI             - OK
  Check daemons   - OK
  Check Redis     - OK
  Check vnet      - Not OK
status: vnetd daemon not responding
root@invader44:~#
  1. . check /var/log/syslog file for panic
Jul 27 03:37:44 debian frr[11777]: Loading capability module if not yet done.
Jul 27 03:37:44 debian goes.vnetd[8143]: eventHandler: panic vnet: RTA_MULTIPATH next-hop without RTA_GATEWAY
Jul 27 03:37:44 debian goes.vnetd[8143]: panic: vnet: RTA_MULTIPATH next-hop without RTA_GATEWAY
Jul 27 03:37:44 debian goes.vnetd[8143]: goroutine 54 [running]:
Jul 27 03:37:44 debian goes.vnetd[8143]: runtime/debug.Stack(0xc1b8a0, 0xc42013d580, 0xc4218b3a48)
Jul 27 03:37:44 debian goes.vnetd[8143]:         /usr/lib/go-1.9/src/runtime/debug/stack.go:24 +0xa7
Jul 27 03:37:44 debian goes.vnetd[8143]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler.func1(0xc420164418, 0xc420164970)
Jul 27 03:37:44 debian goes.vnetd[8143]:         /home/arthur/gopath/src/github.com/platinasystems/go/elib/loop/event.go:189 +0x1c9
Jul 27 03:37:44 debian goes.vnetd[8143]: panic(0xbd1ac0, 0xee08b0)
Jul 27 03:37:44 debian goes.vnetd[8143]:         /usr/lib/go-1.9/src/runtime/panic.go:491 +0x283
Jul 27 03:37:44 debian goes.vnetd[8143]: github.com/platinasystems/go/vnet/unix.(*net_namespace).parse_ip4_next_hops(0xc4200ba090, 0xc4200749c0, 0xc4218b3c20, 0x201e05a8c0, 0xc4200ba090)
Jul 27 03:37:44 debian goes.vnetd[8143]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:669 +0x637
Jul 27 03:37:44 debian goes.vnetd[8143]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).ip4RouteMsg(0xc422c90d00, 0xc4200749c0, 0xc420074900, 0x1, 0x0)
Jul 27 03:37:44 debian goes.vnetd[8143]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:710 +0xfc
Jul 27 03:37:44 debian goes.vnetd[8143]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc422c90d00)
Jul 27 03:37:44 debian goes.vnetd[8143]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:477 +0x58b
Jul 27 03:37:44 debian goes.vnetd[8143]: github.com/platinasystems/go/elib/loop.(*nodeEvent).do(0xc42180c040)
Jul 27 03:37:44 debian goes.vnetd[8143]:         /home/arthur/gopath/src/github.com/platinasystems/go/elib/loop/event.go:158 +0x98
Jul 27 03:37:44 debian goes.vnetd[8143]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc420164418, 0x1970f00, 0xc420164968)
Jul 27 03:37:44 debian goes.vnetd[8143]:         /home/arthur/gopath/src/github.com/platinasystems/go/elib/loop/event.go:228 +0xa3
Jul 27 03:37:44 debian goes.vnetd[8143]: created by github.com/platinasystems/go/elib/loop.(*Node).maybeStartEventHandler.func1
Jul 27 03:37:44 debian goes.vnetd[8143]:         /home/arthur/gopath/src/github.com/platinasystems/go/elib/loop/event.go:389 +0x271
Jul 27 03:37:44 debian frr[11777]: Starting Frr daemons (prio:10):. zebra. bgpd. ospfd2018/07/27 03:37:44 OSPF: MPLS-TE(initialize_linkparams) Could not find corresponding OSPF Interface for docker0
Jul 27 03:37:44 debian frr[11777]: 2018/07/27 03:37:44 OSPF: MPLS-TE(initialize_linkparams) Could not find corresponding OSPF Interface for eth-1-1
Jul 27 03:37:44 debian frr[11777]: 2018/07/27 03:37:44 OSPF: MPLS-TE(initialize_linkparams) Could not find corresponding OSPF Interface for eth-2-1
Jul 27 03:37:44 debian frr[11777]: 2018/07/27 03:37:44 OSPF: MPLS-TE(initialize_linkparams) Could not find corresponding OSPF Interface for eth-3-1