platinasystems / go

Other
9 stars 68 forks source link

vnet daemon not responding #86

Open bshreyas1 opened 6 years ago

bshreyas1 commented 6 years ago

SHA1 of the 3 repositories : version: 412434633cb86f71ef19e9df3ee85de33dab1f84 version: 581c0a529a332cdab0e8f767130c48fd3b8b0f01 version: 60f39141fbbf78ddb2260dba74c68f2789374f18

I have observed vnet daemon issues while running quagga container on Platina switch. I don't remember the exact steps that might have caused this issue.

The terminal gave below output while moving the interface from container back to switch. root@invader43:/home/shreyas# ./docker_move.sh down R2 eth-17-1 status: vnetd daemon not responding goes not happy

Further in the /var/log/daemon.log I found the below suspicious logs: Aug 13 04:38:12 invader43 goes.vnetd[484]: panic: vnet: unknown destination: 10.0.17.29/32 Aug 13 04:38:12 invader43 goes.vnetd[484]: goroutine 25 [running]: Aug 13 04:38:12 invader43 goes.vnetd[484]: runtime/debug.Stack(0xbc40a0, 0xc421e4e0f0, 0xc4222dc6c8) Aug 13 04:38:12 invader43 goes.vnetd[484]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79 Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler.func1(0xc42021e000, 0xc42021e558) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:181 +0x159 Aug 13 04:38:12 invader43 goes.vnetd[484]: panic(0xbcbb40, 0xc421bf20a0) Aug 13 04:38:12 invader43 goes.vnetd[484]: /usr/local/go/src/runtime/panic.go:489 +0x2cf Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/vnet/ip4.(Fib).replaceNextHop(0xc420096a00, 0xc42026c000, 0xc421e4e070, 0xc420096a00, 0x300000005, 0xc41d01000a, 0x15e4720, 0xc420250750, 0xc420750a30, 0x0) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:749 +0x22c Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/vnet/ip4.(mapFibResult).replaceWithLessSpecific(0xc421bf5a10, 0xc42026c000, 0xc420096a00, 0xc421bf5a00) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:387 +0x26f Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/vnet/ip4.(Fib).addDelReachable(0xc420096a00, 0xc42026c000, 0xc421e4e050, 0x100000005) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:452 +0x10a Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/vnet/ip4.(Fib).addDel(0xc420096a00, 0xc42026c000, 0xc421e4e050, 0x2905f80100000005, 0xc4222dcaf8) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:276 +0x413 Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/vnet/ip4.(Main).addDelRoute(0xc42026c000, 0xc421bf2060, 0x500000002, 0x982601, 0xc42026c0b0, 0x7, 0x2) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:635 +0xf4 Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/vnet/ip4.(Main).(github.com/platinasystems/go/vnet/ip4.addDelRoute)-fm(0xc421bf2060, 0x500000002, 0xc400000001, 0xc4216d41b8, 0x1, 0x27b) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/ip4/package.go:28 +0x4d Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/vnet/ethernet.(ipNeighborMain).AddDelIpNeighbor(0xc420153468, 0xc42026c068, 0xc421bf2040, 0x71d010001, 0x1, 0xc4222dcd18, 0x9ed07e) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/ethernet/neighbor.go:91 +0x283 Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).ip4NeighborMsg(0xc420246040, 0xc4203840e0, 0xc4203840e0, 0x1) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:625 +0x27c Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/vnet/unix.(netlinkEvent).EventAction(0xc420246040) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:506 +0x682 Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/elib/loop.(nodeEvent).do(0xc420246100) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:152 +0x8f Aug 13 04:38:12 invader43 goes.vnetd[484]: github.com/platinasystems/go/elib/loop.(Loop).eventHandler(0xc42021e000, 0x15dcc60, 0xc42021e550) Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:195 +0x175 Aug 13 04:38:12 invader43 goes.vnetd[484]: created by github.com/platinasystems/go/elib/loop.(*Node).maybeStartEventHandler Aug 13 04:38:12 invader43 goes.vnetd[484]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:280 +0x29f

jignesh045 commented 6 years ago

Issue verification status: Verified

Issue status: Not reproduced

Issue verified in below build

root@invader44:/# goes version
github.com/platinasystems/go: v0.41-313-g96fcd2e1
github.com/platinasystems/fe1: v0.2-526-gaa6b634
github.com/platinasystems/firmware-fe1a: v0.2
root@invader44:/#

root@invader44:/# goes hget platina-mk1 packages | grep version:
    version: 96fcd2e13b9d751a95eda8faa60e4974e8143765
    version: aa6b634a0548a66703fd379e26677c12263f752a
    version: 60f39141fbbf78ddb2260dba74c68f2789374f18
root@invader44:/#

root@invader44:/# dpkg --list | grep kernel
ii  kmod                           18-3                             amd64        tools for managing Linux kernel modules
ii  libdrm2:amd64                  2.4.58-2                         amd64        Userspace interface to kernel DRM services -- runtime
ii  linux-image-4.13.0-platina-mk1 4.13-129-g34f08d37ef9c           amd64        Linux kernel, version 4.13.0-platina-mk1
ii  rsyslog                        8.4.2-1+deb8u2                   amd64        reliable system and kernel logging daemon
root@invader44:/#

Steps:

  1. Executed below script and observed goes status and syslog but did not found any vnetd daemon failure and panic logs.
root@invader43:/home/jignesh# cat docker_ifmove.sh
#! /bin/bash
for (( i=1; i <=100; i++ ))
        do
                ./docker_move.sh up d6b47753b048 eth-17-1
                ./docker_move.sh down d6b47753b048 eth-17-1
                goes status
                ifconfig eth-17-1
        done

root@invader43:/home/jignesh#
root@invader43:/home/jignesh# ./docker_ifmove.sh >> docker_move.log
root@invader43:/home/jignesh#
root@invader43:/home/jignesh#
root@invader43:/home/jignesh# vi docker_move.log
root@invader43:/home/jignesh#