search

platinasystems / go

Other
9 stars 68 forks source link

vnetd panic when there are 512 or more adjacencies #98

Open fszyang opened 6 years ago

fszyang commented 6 years ago

Can be reproduced by creating 256 interfaces and assigning IP address to them (results in a glean and local entry for each)

jignesh045 commented 6 years ago

Issue Verification Status: Verified in SRIOV GoES version Issue Result Status: Open (Still exists)

Issue verified in the following SRIOV build:

root@invader45:/home/jignesh# goes version
github.com/platinasystems/go: v0.41-244-g49d9e6da
github.com/platinasystems/fe1: v0.2-520-g85494c1
github.com/platinasystems/firmware-fe1a: v0.2
root@invader45:/home/jignesh#
root@invader45:/home/jignesh# goes hget platina-mk1 packages | grep version:
    version: 49d9e6dae2471c7d8f20f88997cd3b4578dee36a
    version: 85494c17b94de997f1d2dac7a0cf21a4e76343f0
    version: 60f39141fbbf78ddb2260dba74c68f2789374f18
root@invader45:/home/jignesh#

root@invader45:/home/jignesh# dpkg --list | grep kernel
ii  kmod                           18-3                             amd64        tools for managing Linux kernel modules
ii  libdrm2:amd64                  2.4.58-2                         amd64        Userspace interface to kernel DRM services -- runtime
ii  linux-image-4.13.0-platina-mk1 4.13-117-g00814e4e1a1e           amd64        Linux kernel, version 4.13.0-platina-mk1
ii  rsyslog                        8.4.2-1+deb8u2                   amd64        reliable system and kernel logging daemon
root@invader45:/home/jignesh#

Steps:

  1. Executed below script
root@invader45:/home/jignesh# cat issue98.sh
#!/bin/bash
goes stop
rmmod platina-mk1
modprobe platina-mk1
for ((i=1; i<=32; i++));
    do
        for ((j=1; j<=4; j++));

        do
                ip link add eth-$i-$j type platina-mk1
                ip link set eth-$i-$j up
                ethtool -s eth-$i-$j speed 10000 autoneg off
                ifconfig eth-$i-$j 10.0.$i.$j/31
                ifconfig eth-$i-$j:0 192.168.$i.$j netmask 255.255.255.255
                ifconfig eth-$i-$j:1 172.16.$i.$j netmask 255.255.255.255
                ifconfig eth-$i-$j:2 172.17.$i.$j netmask 255.255.255.255
                ifconfig eth-$i-$j:3 172.18.$i.$j netmask 255.255.255.255
                ifconfig eth-$i-$j:4 192.178.$i.$j netmask 255.255.255.255
#               ip add add 10.0.$i.$j/32 dev eth-$i-$j
#               sleep 5
        done
    done
goes start
root@invader45:/home/jignesh#
root@invader45:/home/jignesh# ./issue98.sh
start: (platina-mk1,vnet.ready) timeout
root@invader45:/home/jignesh#
  1. Observed logs in /var/log/syslog 
    Jul 25 06:20:03 debian goes.vnetd[22705]: eventHandler: panic vnet: fe1a alloc_slice_rule: exceeded max size 512 of rxf_rule_pool[6]
Jul 25 06:20:03 debian goes.vnetd[22705]: panic: vnet: fe1a alloc_slice_rule: exceeded max size 512 of rxf_rule_pool[6]
Jul 25 06:20:03 debian goes.vnetd[22705]: goroutine 115 [running]:
Jul 25 06:20:03 debian goes.vnetd[22705]: runtime/debug.Stack(0xc1b8a0, 0xc4248fc380, 0xc424c27590)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /usr/lib/go-1.9/src/runtime/debug/stack.go:24 +0xa7
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler.func1(0xc420164418, 0xc420164970)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/elib/loop/event.go:189 +0x1c9
Jul 25 06:20:03 debian goes.vnetd[22705]: panic(0xc1b8a0, 0xc4248fc350)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /usr/lib/go-1.9/src/runtime/panic.go:491 +0x283
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/fe1/internal/fe1a.(*garbage_dump_rxf_pipe).alloc_slice_rule(0xc420a2ded0, 0xfc0, 0xc42339b080)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/fe1/internal/fe1a/dump.go:698 +0x164
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/fe1/internal/fe1a.(*garbage_dump_rxf_main).rxf_ip46_prefix_counter_add_del(0xc420a2ca78, 0xc42366be00, 0xc42058e068, 0xc424c297d8, 0x0, 0x0, 0xc423bf4300, 0xa)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/fe1/internal/fe1a/dump.go:868 +0xba
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/fe1/internal/fe1a.(*adjacency_main).install_adjacency(0xc420a2c518, 0xc420a2c578, 0xc42366be00, 0xc42058e068, 0x0, 0xc424c298f4, 0x7f3500000203, 0x0, 0x20000000001, 0xc42014d880, ...)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/fe1/internal/fe1a/fib.go:285 +0xba8
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/fe1/internal/fe1a.(*ip4_fib_pipe).add_del(0xc420a2fde0, 0xc424c299d0, 0x0, 0xc423bf6a48, 0x203)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/fe1/internal/fe1a/fib.go:683 +0x22e
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/fe1/internal/fe1a.(*fe1a).ip4_fib_add_del(0xc420a14000, 0xc400000000, 0xc423bf6a48, 0x203)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/fe1/internal/fe1a/fib.go:738 +0xec
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/fe1/internal/fe1a.(*fe1a).(github.com/platinasystems/fe1/internal/fe1a.ip4_fib_add_del)-fm(0xc400000000, 0xc423bf6a48, 0x203)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/fe1/internal/fe1a/fib.go:155 +0x4d
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/vnet/ip4.(*fibMain).callFibAddDelHooks(0xc42058e518, 0xc400000000, 0xc423bf6a48, 0x203)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:580 +0x77
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/vnet/ip4.(*Fib).addDel(0xc42014ea00, 0xc42058e000, 0xc423bf6a48, 0x203, 0x203)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:315 +0x206
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/vnet/ip4.(*Fib).addDelReplace(0xc42014ea00, 0xc42058e000, 0xc423bf6a48, 0x203)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:793 +0x50
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/vnet/ip4.(*Main).addDelInterfaceAddressRoutes(0xc42058e000, 0x1c7)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:835 +0x1e9
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/vnet/ip4.(*Main).AddDelInterfaceAddress(0xc42058e000, 0x50, 0xc423bf69e8, 0x19e2800, 0xc420210b00, 0xc400000386)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:884 +0x283
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).ip4IfaddrMsg(0xc4207ffb40, 0xc424afc370, 0xc424afc370, 0x1)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:574 +0x3a9
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc4207ffb40)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:468 +0x47b
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/elib/loop.(*nodeEvent).do(0xc423fd6040)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/elib/loop/event.go:158 +0x98
Jul 25 06:20:03 debian goes.vnetd[22705]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc420164418, 0x1970f00, 0xc420164968)
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/elib/loop/event.go:228 +0xa3
Jul 25 06:20:03 debian goes.vnetd[22705]: created by github.com/platinasystems/go/elib/loop.(*Node).maybeStartEventHandler.func1
Jul 25 06:20:03 debian goes.vnetd[22705]:         /home/arthur/gopath/src/github.com/platinasystems/go/elib/loop/event.go:389 +0x271