Open Tobbeman opened 3 years ago
Can you run agentctl config get
and agentctl config retrieve
before and after restart? Also, please include logs for both runs. Thanks.
Will do! Our env is currently broken, but once I can repeat the setup I will add the info here.
Ok, done some more investigating
I have been using the rest api instead for these test. This flow will cause the issues:
At this point, using retreive will cause a crash
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1409039]
goroutine 137 [running]:
go.ligato.io/vpp-agent/v3/plugins/vpp/ifplugin/vppcalls/vpp2101.(*InterfaceVppHandler).dumpInterfaces(0xc000457550, 0x0, 0x0, 0x0, 0x203000, 0x4, 0x203000)
go.ligato.io/vpp-agent/v3/plugins/vpp/ifplugin/vppcalls/vpp2101/dump_interface_vppcalls.go:142 +0x5d9
go.ligato.io/vpp-agent/v3/plugins/vpp/ifplugin/vppcalls/vpp2101.(*InterfaceVppHandler).DumpInterfaces(0xc000457550, 0x1ff4dc0, 0xc0011966c0, 0x1bfbb80, 0xc000af7901, 0x14e690d)
go.ligato.io/vpp-agent/v3/plugins/vpp/ifplugin/vppcalls/vpp2101/dump_interface_vppcalls.go:175 +0x54
go.ligato.io/vpp-agent/v3/plugins/configurator.(*dumpService).DumpInterfaces(0x304adb8, 0x1ff4dc0, 0xc0011966c0, 0x0, 0x0, 0x0, 0x3, 0x0)
go.ligato.io/vpp-agent/v3/plugins/configurator/dump.go:204 +0x6d
go.ligato.io/vpp-agent/v3/plugins/configurator.(*dumpService).Dump(0x304adb8, 0x1ff4dc0, 0xc0011966c0, 0xc0011966f0, 0x0, 0x0, 0x0)
go.ligato.io/vpp-agent/v3/plugins/configurator/dump.go:81 +0x19d
go.ligato.io/vpp-agent/v3/plugins/configurator.(*configuratorServer).Dump(0x304adb8, 0x1ff4dc0, 0xc0011966c0, 0xc0011966f0, 0x304adb8, 0x2000be0, 0xc000d837c0)
go.ligato.io/vpp-agent/v3/plugins/configurator/configurator.go:56 +0x4b
go.ligato.io/vpp-agent/v3/proto/ligato/configurator._ConfiguratorService_Dump_Handler.func1(0x1ff4dc0, 0xc0011966c0, 0x1b0fb20, 0xc0011966f0, 0x2d, 0xc00120f310, 0x0, 0xc000af7b30)
go.ligato.io/vpp-agent/v3/proto/ligato/configurator/configurator_grpc.pb.go:217 +0x89
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1(0x1ff4dc0, 0xc0011966c0, 0x1b0fb20, 0xc0011966f0, 0xc000ce3700, 0xc000ce3720, 0xc000c69ba0, 0x5cc246, 0x1c1f360, 0xc0011966c0)
github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0xad
go.ligato.io/vpp-agent/v3/proto/ligato/configurator._ConfiguratorService_Dump_Handler(0x1d05ba0, 0x304adb8, 0x1ff4dc0, 0xc0011966c0, 0xc000ce4fc0, 0xc000a6d3b0, 0x1ff4dc0, 0xc0011966c0, 0x0, 0x0)
go.ligato.io/vpp-agent/v3/proto/ligato/configurator/configurator_grpc.pb.go:219 +0x150
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000e8000, 0x2009a60, 0xc0011de480, 0xc000b86d00, 0xc0007150b0, 0x2f609a8, 0x0, 0x0, 0x0)
google.golang.org/grpc@v1.29.1/server.go:1082 +0x522
google.golang.org/grpc.(*Server).handleStream(0xc0000e8000, 0x2009a60, 0xc0011de480, 0xc000b86d00, 0x0)
google.golang.org/grpc@v1.29.1/server.go:1405 +0xcc5
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000396cb0, 0xc0000e8000, 0x2009a60, 0xc0011de480, 0xc000b86d00)
google.golang.org/grpc@v1.29.1/server.go:746 +0xa5
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
config get will give this:
netallocConfig: {}
linuxConfig: {}
vppConfig:
interfaces:
- name: NCIC-0.4090
type: SUB_INTERFACE
enabled: true
ipAddresses:
- 10.10.22.25/29
vrf: 4090
sub:
parentName: NCIC-0
subId: 4090
- name: host-Vpp2Host.4091
type: SUB_INTERFACE
enabled: true
ipAddresses:
- 10.10.22.33/29
vrf: 4091
sub:
parentName: host-Vpp2Host
subId: 4091
- name: NCIC-0.1
type: SUB_INTERFACE
enabled: true
ipAddresses:
- 169.254.1.1/29
vrf: 10001
sub:
parentName: NCIC-0
subId: 1
- name: NCIC-0.4091
type: SUB_INTERFACE
enabled: true
ipAddresses:
- 10.10.22.9/29
vrf: 4091
sub:
parentName: NCIC-0
subId: 4091
- name: NCIC-0
type: DPDK
enabled: true
physAddress: 0e:92:1a:bc:2c:37
mtu: 9206
rxModes:
- mode: POLLING
- name: host-Vpp2Host
type: AF_PACKET
enabled: true
physAddress: 9a:dd:37:9b:ed:ad
rxModes:
- mode: INTERRUPT
rxPlacements:
- worker: 1
afpacket:
hostIfName: Vpp2Host
- name: eth0
type: DPDK
enabled: true
physAddress: e4:43:4b:e5:62:10
ipAddresses:
- 10.10.12.50/24
vrf: 4091
mtu: 9206
rxModes:
- mode: POLLING
- name: Ext-0.2
type: SUB_INTERFACE
enabled: true
ipAddresses:
- 10.88.99.1/29
vrf: 10002
sub:
parentName: Ext-0
subId: 2
- name: Ext-0
type: DPDK
enabled: true
physAddress: e4:43:4b:e5:62:11
ipAddresses:
- 192.168.23.151/21
vrf: 10001
mtu: 9206
rxModes:
- mode: POLLING
rxPlacements:
- worker: 1
- name: NCIC-0.2
type: SUB_INTERFACE
enabled: true
ipAddresses:
- 169.254.2.1/29
vrf: 10002
sub:
parentName: NCIC-0
subId: 2
- name: host-Vpp2Host.4090
type: SUB_INTERFACE
enabled: true
ipAddresses:
- 10.10.22.1/29
vrf: 4090
sub:
parentName: host-Vpp2Host
subId: 4090
routes:
- vrfId: 4091
dstNetwork: 10.10.25.197/32
nextHopAddr: 10.10.22.34
outgoingInterface: host-Vpp2Host.4091
- vrfId: 4091
dstNetwork: 10.10.25.210/32
nextHopAddr: 10.10.22.10
outgoingInterface: NCIC-0.4091
- vrfId: 4091
dstNetwork: 10.10.25.194/32
nextHopAddr: 10.10.22.34
outgoingInterface: host-Vpp2Host.4091
- vrfId: 4090
dstNetwork: 0.0.0.0/0
nextHopAddr: 10.10.22.3
outgoingInterface: host-Vpp2Host.4090
- vrfId: 4090
dstNetwork: 10.10.23.193/32
nextHopAddr: 10.10.22.26
outgoingInterface: NCIC-0.4090
- vrfId: 4090
dstNetwork: 10.10.22.200/32
nextHopAddr: 10.10.22.2
outgoingInterface: host-Vpp2Host.4090
- vrfId: 4091
dstNetwork: 10.10.25.193/32
nextHopAddr: 10.10.22.34
outgoingInterface: host-Vpp2Host.4091
- vrfId: 4090
dstNetwork: 10.10.23.194/32
nextHopAddr: 10.10.22.26
outgoingInterface: NCIC-0.4090
- vrfId: 4090
dstNetwork: 10.10.22.198/32
nextHopAddr: 10.10.22.2
outgoingInterface: host-Vpp2Host.4090
- vrfId: 4090
dstNetwork: 10.10.22.199/32
nextHopAddr: 10.10.22.2
outgoingInterface: host-Vpp2Host.4090
- vrfId: 10002
dstNetwork: 0.0.0.0/0
nextHopAddr: 10.88.99.2
outgoingInterface: Ext-0.2
- vrfId: 4091
dstNetwork: 10.10.25.198/32
nextHopAddr: 10.10.22.34
outgoingInterface: host-Vpp2Host.4091
- vrfId: 10001
dstNetwork: 0.0.0.0/0
nextHopAddr: 192.168.23.211
outgoingInterface: Ext-0
- vrfId: 4090
dstNetwork: 10.10.22.202/32
nextHopAddr: 10.10.22.26
outgoingInterface: NCIC-0.4090
- vrfId: 4091
dstNetwork: 0.0.0.0/0
nextHopAddr: 10.10.12.100
outgoingInterface: eth0
vrfs:
- id: 4090
label: "4090"
- id: 4091
label: "4091"
- id: 10001
label: "10001"
- id: 10002
label: "10002"
Worth mentioning, retrieve never works for us:
ERROR: rpc error: code = Unknown desc = l2_xconnect_dump crashes in VPP 21.01, dump will be skipped
Another thing worth mentioning, we run vpp on the host, mounting the sockets into the vpp-agent container. Could it be compability issues?
Hello,
We noticed that for some reason ligato will add breaking config once restarted. On the first apply of config, via etcd, it will work correctly. But if we restart ligato, additional entries will be added that break our env.
config, which is applied via etcd using a script:
Route entry on first apply:
Route entry on ligato restart (docker restart ligato):
Note the added arp entry
Using goVPP directly and adding the route twice seems to work, indicating it is something wrong in ligato when constructing models.
Any idea where to look for the issue?