search

FRRouting / frr

The FRRouting Protocol Suite
https://frrouting.org/
Other
3.33k stars 1.25k forks source link

critical: BGP: Received signal 11 at 1611466616 (si_addr 0x0, PC 0x55aaae6f9802); aborting... #7921

Closed Napsterbater closed 3 years ago

Napsterbater commented 3 years ago

Describe the bug

Upon adding/activating a particular IPv6 Multiprotocol Link-Local peer FRR completely crashes and restarts with the following in the logs (set to debug.

Note FRR already running normal with many peers connected when I then set the logging to debug and configured the peer,, this starts at the very first message generated, wichch are the basic and "normal" "unrecognized capability code". 

2021/01/24 00:36:56 warnings: BGP: [EC 33554503] fe80::925 unrecognized capability code: 70 - ignored
2021/01/24 00:36:56 warnings: BGP: [EC 33554503] fe80::925 unrecognized capability code: 71 - ignored
critical: BGP: Received signal 11 at 1611466616 (si_addr 0x0, PC 0x55aaae6f9802); aborting...
critical: BGP: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x5b) [0x7fe0307aa6eb]
critical: BGP: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xe3) [0x7fe0307aa8e3]
critical: BGP: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x7c2d9) [0x7fe0307d22d9]
critical: BGP: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fe0305d43c0]
critical: BGP: /usr/lib/frr/bgpd(+0xd8802) [0x55aaae6f9802]
critical: BGP: /usr/lib/frr/bgpd(bgp_zebra_announce+0x4ca) [0x55aaae6fbb6a]
critical: BGP: /usr/lib/frr/bgpd(+0x8e5a2) [0x55aaae6af5a2]
critical: BGP: /usr/lib/frr/bgpd(+0x8e6ee) [0x55aaae6af6ee]
critical: BGP: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(work_queue_run+0x128) [0x7fe0307ebd08]
critical: BGP: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x5a) [0x7fe0307e1c1a]
critical: BGP: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7fe0307a9298]
critical: BGP: /usr/lib/frr/bgpd(main+0x2c9) [0x55aaae65ed89]
critical: BGP: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fe0303f40b3]
critical: BGP: /usr/lib/frr/bgpd(_start+0x2e) [0x55aaae660fbe]
critical: BGP: in thread work_queue_run scheduled from lib/workqueue.c:139
2021/01/24 00:36:56 warnings: ZEBRA: [EC 4043309122] Client 'vnc' encountered an error and is shutting down.
2021/01/24 00:36:56 warnings: ZEBRA: [EC 4043309122] Client 'bgp' encountered an error and is shutting down.
2021/01/24 00:36:56 debugging: ZEBRA: release_daemon_table_chunks: Released 0 table chunks
2021/01/24 00:36:56 notifications: ZEBRA: client 30 disconnected 0 vnc routes removed from the rib
2021/01/24 00:36:56 debugging: ZEBRA: release_daemon_table_chunks: Released 0 table chunks
2021/01/24 00:36:56 errors: ZEBRA: zebra/zebra_ptm.c:1348 failed to find process pid registration
2021/01/24 00:36:56 notifications: ZEBRA: client 16 disconnected 1729 bgp routes removed from the rib
2021/01/24 00:37:01 notifications: ZEBRA: client 30 says hello and bids fair to announce only vnc routes vrf=0
2021/01/24 00:37:01 notifications: ZEBRA: client 16 says hello and bids fair to announce only bgp routes vrf=0

[x] Did you check if this is a duplicate issue? Did find one, unless I just overlooked it. [ ] Did you test it on the latest FRRouting/frr master branch? Latest via APT/Repo

To Reproduce

conf term
router bgp 4200000001 vrf dn42
neighbor fe80::925 remote-as 4242420925
neighbor fe80::925 peer-group DN42fullpeer
neighbor fe80::925 description YuetAu-dn42-CA-YKF
neighbor fe80::925 update-source wg0925
neighbor fe80::925 interface wg0925
address-family ipv4 unicast
neighbor fe80::925 activate
exit
address-family ipv6 unicast
neighbor fe80::925 activate
exit
exit
exit

Expected behavior

Not to crash?

Versions

Additional context

PCAP of the exchange between the peers if it helps: https://drive.google.com/file/d/1rd6_W4qX2RC3ZQ5WjVVwyLHziRpQA2Fd/view?usp=sharing

Full Config:

frr version 7.5
frr defaults traditional
hostname ATL1-US.napshome.dn42
log record-priority
rpki
  rpki polling_period 3600
  rpki cache ::1 8082 preference 1
  rpki cache fda9:26a9:1c47:1d42:42:42:42:1050 8082 preference 2
  rpki cache fda9:26a9:1c47:2d42:42:42:42:1050 8082 preference 3
  exit
service integrated-vtysh-config
!
enable password ********************
!
ip route 172.22.169.128/28 dn42 nexthop-vrf dn42
ipv6 route fda9:26a9:1c47:d42::/64 dn42 nexthop-vrf dn42
!
vrf dn42
 ip route 10.0.15.0/24 wg0 nexthop-vrf default
 ipv6 route fd42:4242:2601:ac12::1/128 wg2601
 ipv6 route fd86:bad:11b7:9::1/128 wg1080
 exit-vrf
!
router bgp 4200000001
 bgp router-id 10.0.0.42
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp confederation identifier 4242421050
 bgp confederation peers 4200000002 4200000003 4200000010 4200000020
 no bgp network import-check
 neighbor 10.0.0.41 remote-as 4200000002
 neighbor 10.0.0.41 description CAR1
 neighbor 10.0.0.41 timers connect 2
 neighbor fda9:26a9:1c47::41 remote-as 4200000002
 neighbor fda9:26a9:1c47::41 description CAR1
 neighbor fda9:26a9:1c47::41 timers connect 2
 !
 address-family ipv4 unicast
  network 10.0.15.0/24 route-map rm-no-export
  network 10.64.0.0/16 route-map rm-no-export
  network 172.22.169.128/28
  neighbor fda9:26a9:1c47::41 activate
  neighbor fda9:26a9:1c47::41 next-hop-self
  neighbor fda9:26a9:1c47::41 soft-reconfiguration inbound
  neighbor fda9:26a9:1c47::41 prefix-list localpeer in
  import vrf dn42
 exit-address-family
 !
 address-family ipv6 unicast
  network ::/0 route-map rm-no-export
  network 64:ff9b::/96 route-map rm-no-export
  network 2001:470:****:1015::/64 route-map rm-no-export
  network fda9:26a9:1c47::/48
  neighbor fda9:26a9:1c47::41 activate
  neighbor fda9:26a9:1c47::41 next-hop-self
  neighbor fda9:26a9:1c47::41 soft-reconfiguration inbound
  neighbor fda9:26a9:1c47::41 prefix-list localpeer in
  import vrf dn42
 exit-address-family
!
router bgp 4200000001 vrf dn42
 bgp router-id 172.22.169.128
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp confederation identifier 4242421050
 bgp confederation peers 4200000002 4200000003 4200000010 4200000020
 bgp deterministic-med
 update-delay 300 120
 no bgp network import-check
 neighbor Burble-GRC peer-group
 neighbor DN42fullpeer peer-group
 neighbor DN42fullpeer timers connect 30
 neighbor DN42selfpeer peer-group
 neighbor DN42selfpeer timers 15 45
 neighbor DN42selfpeer timers connect 5
 neighbor fd42:4242:2601:ac12::1 remote-as 4242422602
 neighbor fd42:4242:2601:ac12::1 peer-group Burble-GRC
 neighbor fd42:4242:2601:ac12::1 description Burble-GRC
 neighbor fd42:4242:2601:ac12::1 ebgp-multihop 255
 neighbor fd42:4242:2601:ac12::1 update-source wg2601
 neighbor fd42:4242:2601:ac12::1 timers connect 30
 neighbor 172.20.6.249 remote-as 4242421114
 neighbor 172.20.6.249 peer-group DN42fullpeer
 neighbor 172.20.6.249 description GRGR-dn42-US-CHI
 neighbor 172.20.6.249 update-source wg1114
 neighbor 172.20.16.139 remote-as 4242421588
 neighbor 172.20.16.139 peer-group DN42fullpeer
 neighbor 172.20.16.139 description chrismoos-dn42-us-CHI1
 neighbor 172.20.16.139 update-source wg1588
 neighbor 172.20.53.98 remote-as 4242423914
 neighbor 172.20.53.98 peer-group DN42fullpeer
 neighbor 172.20.53.98 description Kioubit-dn42-US-CHI
 neighbor 172.20.53.98 update-source wg3914
 neighbor 172.21.99.133 remote-as 4242421099
 neighbor 172.21.99.133 peer-group DN42fullpeer
 neighbor 172.21.99.133 description OWENSRESEARCH-dn42-us-atl
 neighbor 172.21.99.133 interface wg1099
 neighbor 172.21.99.133 update-source wg1099
 neighbor 172.21.99.133 timers connect 30
 neighbor 172.21.100.194 remote-as 4242423088
 neighbor 172.21.100.194 peer-group DN42fullpeer
 neighbor 172.21.100.194 description sunnut-dn42-us-buff1
 neighbor 172.21.100.194 update-source wg3088
 neighbor 172.22.159.60 remote-as 4242422904
 neighbor 172.22.159.60 peer-group DN42fullpeer
 neighbor 172.22.159.60 description doxz-atl-v4
 neighbor 172.22.159.60 update-source wg2904
 neighbor 172.23.109.169 remote-as 4242423856
 neighbor 172.23.109.169 peer-group DN42fullpeer
 neighbor 172.23.109.169 description scooter-dn42-US-TX
 neighbor 172.23.109.169 update-source wg3856
 neighbor fd42:8ffc:36a:17::1 remote-as 4242420880
 neighbor fd42:8ffc:36a:17::1 peer-group DN42fullpeer
 neighbor fd42:8ffc:36a:17::1 description LIYUMI-dn42-US-LA
 neighbor fd42:8ffc:36a:17::1 disable-connected-check
 neighbor fd42:8ffc:36a:17::1 update-source wg0880
 neighbor fd86:bad:11b7:9::1 remote-as 4242421080
 neighbor fd86:bad:11b7:9::1 peer-group DN42fullpeer
 neighbor fd86:bad:11b7:9::1 description jlu5-dn42-ATL
 neighbor fd86:bad:11b7:9::1 disable-connected-check
 neighbor fd86:bad:11b7:9::1 update-source wg1080
 neighbor fda9:26a9:1c47:d42:42:42:1114:1051 remote-as 4242421114
 neighbor fda9:26a9:1c47:d42:42:42:1114:1051 peer-group DN42fullpeer
 neighbor fda9:26a9:1c47:d42:42:42:1114:1051 description GRGR-dn42-US-CHI
 neighbor fda9:26a9:1c47:d42:42:42:1114:1051 update-source wg1114
 neighbor fda9:26a9:1c47:d42:42:42:2237:1051 remote-as 4242422237
 neighbor fda9:26a9:1c47:d42:42:42:2237:1051 peer-group DN42fullpeer
 neighbor fda9:26a9:1c47:d42:42:42:2237:1051 description munsternet-dn42-us-mks01
 neighbor fda9:26a9:1c47:d42:42:42:2237:1051 update-source wg2237
 neighbor fda9:26a9:1c47:d42:42:42:2502:1051 remote-as 4242422502
 neighbor fda9:26a9:1c47:d42:42:42:2502:1051 peer-group DN42fullpeer
 neighbor fda9:26a9:1c47:d42:42:42:2502:1051 description empty_string-dn42-us-atl
 neighbor fda9:26a9:1c47:d42:42:42:2502:1051 ebgp-multihop 3
 neighbor fda9:26a9:1c47:d42:42:42:2502:1051 update-source fda9:26a9:1c47:d42:42:42:2502:1050
 neighbor fda9:26a9:1c47:d42:42:42:2575:1051 remote-as 4242422575
 neighbor fda9:26a9:1c47:d42:42:42:2575:1051 peer-group DN42fullpeer
 neighbor fda9:26a9:1c47:d42:42:42:2575:1051 description androw-dn42-US-SC
 neighbor fda9:26a9:1c47:d42:42:42:2575:1051 update-source wg2575
 neighbor fda9:26a9:1c47:d42:42:42:2601:1051 remote-as 4242422601
 neighbor fda9:26a9:1c47:d42:42:42:2601:1051 peer-group DN42fullpeer
 neighbor fda9:26a9:1c47:d42:42:42:2601:1051 description Burble-dn42-us-CHI1
 neighbor fda9:26a9:1c47:d42:42:42:2601:1051 update-source wg2601
 neighbor fda9:26a9:1c47:d42:42:42:2904:1051 remote-as 4242422904
 neighbor fda9:26a9:1c47:d42:42:42:2904:1051 peer-group DN42fullpeer
 neighbor fda9:26a9:1c47:d42:42:42:2904:1051 description doxz-atl-v6
 neighbor fda9:26a9:1c47:d42:42:42:2904:1051 update-source wg2904
 neighbor fda9:26a9:1c47:d42:42:42:3914:1051 remote-as 4242423914
 neighbor fda9:26a9:1c47:d42:42:42:3914:1051 peer-group DN42fullpeer
 neighbor fda9:26a9:1c47:d42:42:42:3914:1051 description Kioubit-dn42-US-CHI
 neighbor fda9:26a9:1c47:d42:42:42:3914:1051 update-source wg3914
 neighbor fdc8:dc88:ee11:f2::1 remote-as 4242423088
 neighbor fdc8:dc88:ee11:f2::1 peer-group DN42fullpeer
 neighbor fdc8:dc88:ee11:f2::1 description sunnut-dn42-us-buff1
 neighbor fdc8:dc88:ee11:f2::1 update-source wg3088
 neighbor fe80::1:1050 remote-as 4200000010
 neighbor fe80::1:1050 peer-group DN42selfpeer
 neighbor fe80::1:1050 description ffm1-de-napshome
 neighbor fe80::1:1050 interface wg1050-1
 neighbor fe80::1:1050 update-source wg1050-1
 neighbor fe80::2:1050 remote-as 4200000020
 neighbor fe80::2:1050 peer-group DN42selfpeer
 neighbor fe80::2:1050 description mcr1-uk-napshome
 neighbor fe80::2:1050 interface wg1050-2
 neighbor fe80::2:1050 update-source wg1050-2
 !
 address-family ipv4 unicast
  network 172.22.169.128/28
  network 172.22.169.128/32 route-map rm-no-export
  network 172.22.169.130/32 route-map rm-no-export
  network 172.22.169.142/32 route-map rm-no-export
  redistribute connected route-map rm-no-export
  neighbor Burble-GRC prefix-list Burble-GRC-in in
  neighbor Burble-GRC prefix-list DN42-out out
  neighbor DN42fullpeer route-map rm-fullpeer-in in
  neighbor DN42fullpeer route-map rm-fullpeer-out out
  neighbor DN42fullpeer attribute-unchanged med
  neighbor DN42selfpeer next-hop-self force
  neighbor DN42selfpeer soft-reconfiguration inbound
  neighbor DN42selfpeer route-map rm-selfpeer-in in
  neighbor DN42selfpeer route-map rm-selfpeer-out out
  neighbor DN42selfpeer attribute-unchanged med
  neighbor fd42:4242:2601:ac12::1 activate
  neighbor fd42:4242:2601:ac12::1 addpath-tx-bestpath-per-AS
  neighbor 172.20.6.249 activate
  neighbor 172.20.16.139 activate
  neighbor 172.20.53.98 activate
  neighbor 172.21.99.133 activate
  neighbor 172.21.99.133 soft-reconfiguration inbound
  neighbor 172.21.100.194 activate
  neighbor 172.22.159.60 activate
  neighbor 172.23.109.169 activate
  neighbor fd42:8ffc:36a:17::1 activate
  neighbor fd86:bad:11b7:9::1 activate
  neighbor fda9:26a9:1c47:d42:42:42:2502:1051 activate
  neighbor fda9:26a9:1c47:d42:42:42:2575:1051 activate
  neighbor fda9:26a9:1c47:d42:42:42:2601:1051 activate
  neighbor fe80::1:1050 activate
  neighbor fe80::2:1050 activate
  import vrf default
 exit-address-family
 !
 address-family ipv6 unicast
  network fda9:26a9:1c47::/48
  network fda9:26a9:1c47:d42::/64
  redistribute connected route-map rm-no-export
  neighbor Burble-GRC prefix-list Burble-GRC-in in
  neighbor Burble-GRC prefix-list DN42-out out
  neighbor DN42fullpeer soft-reconfiguration inbound
  neighbor DN42fullpeer route-map rm-fullpeer-in in
  neighbor DN42fullpeer route-map rm-fullpeer-out out
  neighbor DN42fullpeer attribute-unchanged med
  neighbor DN42selfpeer next-hop-self force
  neighbor DN42selfpeer soft-reconfiguration inbound
  neighbor DN42selfpeer route-map rm-selfpeer-in in
  neighbor DN42selfpeer route-map rm-selfpeer-out out
  neighbor DN42selfpeer attribute-unchanged med
  neighbor fd42:4242:2601:ac12::1 activate
  neighbor fd42:4242:2601:ac12::1 addpath-tx-bestpath-per-AS
  neighbor 172.21.99.133 activate
  neighbor 172.21.99.133 soft-reconfiguration inbound
  neighbor fd42:8ffc:36a:17::1 activate
  neighbor fd86:bad:11b7:9::1 activate
  neighbor fda9:26a9:1c47:d42:42:42:1114:1051 activate
  neighbor fda9:26a9:1c47:d42:42:42:2237:1051 activate
  neighbor fda9:26a9:1c47:d42:42:42:2502:1051 activate
  neighbor fda9:26a9:1c47:d42:42:42:2575:1051 activate
  neighbor fda9:26a9:1c47:d42:42:42:2601:1051 activate
  neighbor fda9:26a9:1c47:d42:42:42:2904:1051 activate
  neighbor fda9:26a9:1c47:d42:42:42:3914:1051 activate
  neighbor fdc8:dc88:ee11:f2::1 activate
  neighbor fe80::1:1050 activate
  neighbor fe80::2:1050 activate
  import vrf default
 exit-address-family
!
ip prefix-list Burble-GRC-in description Burble GRC Inbound Deny all
ip prefix-list Burble-GRC-in seq 1 deny any
ip prefix-list DN42-in description DN42 BGP IPv4 Import filter
ip prefix-list DN42-in seq 1 deny 0.0.0.0/0
ip prefix-list DN42-in seq 11 permit 172.20.0.0/24 ge 28
ip prefix-list DN42-in seq 12 permit 172.21.0.0/24 ge 28
ip prefix-list DN42-in seq 13 permit 172.22.0.0/24 ge 28
ip prefix-list DN42-in seq 14 permit 172.23.0.0/24 ge 28
ip prefix-list DN42-in seq 15 permit 172.20.0.0/14 ge 21 le 29
ip prefix-list DN42-in seq 21 permit 10.100.0.0/14 le 32
ip prefix-list DN42-in seq 22 permit 10.0.0.0/8 ge 15 le 24
ip prefix-list DN42-in seq 4 deny 172.22.169.128/28 le 32
ip prefix-list DN42-in seq 40 permit 172.31.0.0/16 le 32
ip prefix-list DN42-in seq 5 deny 10.0.0.0/20 le 32
ip prefix-list DN42-in seq 9 deny 172.22.166.0/24 le 32
ip prefix-list DN42-out description DN42 BGP IPv4 Export filter
ip prefix-list DN42-out seq 1 permit 172.22.169.128/28
ip prefix-list DN42-out seq 1000 deny any
ip prefix-list DN42-out seq 11 permit 172.20.0.0/24 ge 28
ip prefix-list DN42-out seq 12 permit 172.21.0.0/24 ge 28
ip prefix-list DN42-out seq 13 permit 172.22.0.0/24 ge 28
ip prefix-list DN42-out seq 14 permit 172.23.0.0/24 ge 28
ip prefix-list DN42-out seq 15 permit 172.20.0.0/14 ge 21 le 29
ip prefix-list DN42-out seq 2 deny 0.0.0.0/0
ip prefix-list DN42-out seq 21 permit 10.100.0.0/14 le 32
ip prefix-list DN42-out seq 22 permit 10.0.0.0/8 ge 15 le 24
ip prefix-list DN42-out seq 40 permit 172.31.0.0/16 le 32
ip prefix-list DN42-out seq 5 deny 10.0.0.0/20 le 32
ip prefix-list DN42-out seq 9 deny 172.22.166.0/24 le 32
ip prefix-list Own-Add-Space description My Own Address Space
ip prefix-list Own-Add-Space seq 1 permit 10.0.0.0/20 le 32
ip prefix-list Own-Add-Space seq 2 permit 172.22.169.128/28 le 32
ip prefix-list Self-DN42-in description DN42 BGP IPv4 Import filter
ip prefix-list Self-DN42-in seq 1 deny 0.0.0.0/0
ip prefix-list Self-DN42-in seq 11 permit 172.20.0.0/24 ge 28
ip prefix-list Self-DN42-in seq 12 permit 172.21.0.0/24 ge 28
ip prefix-list Self-DN42-in seq 13 permit 172.22.0.0/24 ge 28
ip prefix-list Self-DN42-in seq 14 permit 172.23.0.0/24 ge 28
ip prefix-list Self-DN42-in seq 15 permit 172.20.0.0/14 ge 21
ip prefix-list Self-DN42-in seq 21 permit 10.100.0.0/14 le 32
ip prefix-list Self-DN42-in seq 22 permit 10.0.0.0/8 ge 15 le 24
ip prefix-list Self-DN42-in seq 4 permit 172.22.169.128/28 le 32
ip prefix-list Self-DN42-in seq 40 permit 172.31.0.0/16 le 32
ip prefix-list Self-DN42-in seq 5 deny 10.0.0.0/20 le 32
ip prefix-list Self-DN42-in seq 9 deny 172.22.166.0/24 le 32
ip prefix-list Self-DN42-out description DN42 BGP IPv4 Export filter
ip prefix-list Self-DN42-out seq 1 permit 172.22.169.128/28 le 32
ip prefix-list Self-DN42-out seq 1000 deny any
ip prefix-list Self-DN42-out seq 11 permit 172.20.0.0/24 ge 28
ip prefix-list Self-DN42-out seq 12 permit 172.21.0.0/24 ge 28
ip prefix-list Self-DN42-out seq 13 permit 172.22.0.0/24 ge 28
ip prefix-list Self-DN42-out seq 14 permit 172.23.0.0/24 ge 28
ip prefix-list Self-DN42-out seq 15 permit 172.20.0.0/14 ge 21
ip prefix-list Self-DN42-out seq 2 deny 0.0.0.0/0
ip prefix-list Self-DN42-out seq 21 permit 10.100.0.0/14 le 32
ip prefix-list Self-DN42-out seq 22 permit 10.0.0.0/8 ge 15 le 24
ip prefix-list Self-DN42-out seq 40 permit 172.31.0.0/16 le 32
ip prefix-list Self-DN42-out seq 5 deny 10.0.0.0/20 le 32
ip prefix-list Self-DN42-out seq 9 deny 172.22.166.0/24 le 32
ip prefix-list localpeer seq 1 deny 0.0.0.0/0
ip prefix-list localpeer seq 10 permit 10.0.0.0/20 le 32
!
ipv6 prefix-list Burble-GRC-in description Burble GRC Inbound Deny all
ipv6 prefix-list Burble-GRC-in seq 1 deny any
ipv6 prefix-list DN42-in description DN42 BGP IPv6 Import filter
ipv6 prefix-list DN42-in seq 1 deny ::/0
ipv6 prefix-list DN42-in seq 10 permit fd00::/8 ge 9 le 64
ipv6 prefix-list DN42-in seq 5 deny fda9:26a9:1c47::/48 le 128
ipv6 prefix-list DN42-out description DN42 BGP IPv6 Export filter
ipv6 prefix-list DN42-out seq 1 permit fda9:26a9:1c47::/48
ipv6 prefix-list DN42-out seq 10 permit fd00::/8 ge 9 le 64
ipv6 prefix-list DN42-out seq 1000 deny any
ipv6 prefix-list DN42-out seq 2 deny ::/0
ipv6 prefix-list Own-Add-Space description My Own Address Space
ipv6 prefix-list Own-Add-Space seq 1 permit fda9:26a9:1c47::/48 le 128
ipv6 prefix-list Own-Add-Space seq 2 permit 2001:470:****::/48 le 128
ipv6 prefix-list Own-Add-Space seq 3 permit 2001:470:****:****::/64 le 128
ipv6 prefix-list Self-DN42-in description DN42 Self Peer Import Filter
ipv6 prefix-list Self-DN42-in seq 1 deny ::/0
ipv6 prefix-list Self-DN42-in seq 10 permit fd00::/8 ge 9
ipv6 prefix-list Self-DN42-in seq 5 permit fda9:26a9:1c47::/48 le 128
ipv6 prefix-list Self-DN42-out description DN42 Self Peer Export Filter
ipv6 prefix-list Self-DN42-out seq 1 permit fda9:26a9:1c47::/48 le 128
ipv6 prefix-list Self-DN42-out seq 10 permit fd00::/8 ge 9
ipv6 prefix-list Self-DN42-out seq 1000 deny any
ipv6 prefix-list Self-DN42-out seq 2 deny ::/0
ipv6 prefix-list localpeer seq 10 permit fda9:26a9:1c47::/48 le 128
ipv6 prefix-list localpeer seq 2 deny ::/3
ipv6 prefix-list localpeer seq 20 permit 2001:470:****::/48 le 128
ipv6 prefix-list localpeer seq 30 permit 2001:470:****:****::/64 le 128
!
bgp as-path access-list atl1-us permit ^4200000001_
bgp as-path access-list burble permit ^4242422601_
bgp as-path access-list confed permit ^(\([0-9 ]+\))$
bgp as-path access-list ffm1-de permit ^4200000010_
bgp as-path access-list mcr1-uk permit ^4200000020_
bgp as-path access-list nia permit ^4242421331_
bgp as-path access-list nia permit _4242421331_
bgp as-path access-list santoastneo permit _4201270009_
bgp as-path access-list selfAS permit ^()$
bgp as-path access-list zhaofeng permit ^4242421926_
!
route-map rm-fullpeer-in deny 20
 match as-path nia
!
route-map rm-fullpeer-in deny 21
 match as-path santoastneo
!
route-map rm-fullpeer-in permit 10
 description check for blackholing,
 match community blackhole
 on-match next
 set community no-export additive
 set local-preference 200
!
route-map rm-fullpeer-in permit 40
 call rpki
 on-match next
!
route-map rm-fullpeer-in permit 90
!
route-map rm-fullpeer-out permit 10
!
route-map rm-fullpeer-out permit 90
!
route-map rm-no-advertise permit 10
 set community no-advertise additive
!
route-map rm-no-advertise permit 20
!
route-map rm-no-export permit 10
 set community no-export additive
!
route-map rm-no-export permit 20
!
route-map rm-selfpeer-in permit 40
 match peer fe80::1:1050
 on-match goto 90
 set metric +1080
!
route-map rm-selfpeer-in permit 50
 match peer fe80::2:1050
 on-match goto 90
 set metric +950
!
route-map rm-selfpeer-in permit 90
!
route-map rm-selfpeer-out permit 10
!
route-map rm-selfpeer-out permit 90
!
route-map rpki permit 10
 match rpki valid
 on-match goto 90
 set local-preference 100
!
route-map rpki permit 20
 match rpki invalid
 on-match goto 90
 set community no-export additive
 set local-preference 10
!
route-map rpki permit 30
 match rpki notfound
 on-match goto 90
 set community no-export additive
 set local-preference 80
!
route-map rpki permit 4
 match ipv6 address prefix-list Own-Add-Space
 on-match goto 90
!
route-map rpki permit 5
 match ip address prefix-list Own-Add-Space
 on-match goto 90
!
route-map rpki permit 90
!
line vty
!
ton31337 commented 3 years ago

Are you able to gather coredump?

Napsterbater commented 3 years ago

If you can get me info on how I would be glad to.

Napsterbater commented 3 years ago

Is this what you need?

From /var/tmp/frr https://drive.google.com/file/d/1J2qDwBmb6KUyyUztTRv17IuMTyU09OOn/view?usp=sharing

Napsterbater commented 3 years ago

NVM, this is what you wanted, i think.

I have 2 in a 7z file. .crash1 is the first time this happened and it generated the dump that I didn't know about. .crash2 is one I just made. Same peer, same commands, practically identical test, only on a different day.

https://drive.google.com/file/d/1Frafmbw_iyWk_30fGhge-MlgWMVcsx5I/view?usp=sharing

ton31337 commented 3 years ago

Could you tell me how did you grab those core-dumps? Seems not as I was expecting.

Napsterbater commented 3 years ago

Got the from /var/crash

How should a generate a proper core dump?

ton31337 commented 3 years ago

is not a core dump: File format not recognized.

Use these sysctl settings:

sysctl -w kernel.core_pattern=/var/crash/core.%u.%e.%p
sysctl -w kernel.suid_dumpable=2
Napsterbater commented 3 years ago

So apparently those were Apport crash files. And encoded in them was the dump file.

I have unpacked them (apport-unpack _usr_lib_frr_bgpd.0.crash1 crash1/), and packed them into the .7z, there is 3 folders, in those 3 folders is the core dumps from 3 different crashes related to this, plus other misc info about the process.

All three used the same peer/commands, only difference is the day they were run/dumped.

https://drive.google.com/file/d/16AcgZWgZQxSaSROOwU7FMChmysakZ1hg/view?usp=sharing

ton31337 commented 3 years ago

Can you install gdb in your machine and run gdb -ex 'bt' --batch /usr/lib/frr/bgpd crash1/CoreDump? The output from there will be helpful unless there is something like I get:

% gdb -ex 'bt' --batch /usr/lib/frr/bgpd CoreDump
[New LWP 1672]
[New LWP 1673]
[New LWP 1675]
[New LWP 1674]
[New LWP 1705]
Core was generated by `/usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M rpki'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f931a55718b in ?? ()
[Current thread is 1 (LWP 1672)]
#0  0x00007f931a55718b in ?? ()
#1  0xfffffffe7ffbdedf in ?? ()
#2  0x00007f931a6fe1a0 in ?? ()
#3  0x0000000000000000 in ?? ()
Napsterbater commented 3 years ago

Crash 1

napsterbater@ATL1-US:/var/crash$ gdb -ex 'bt' --batch /usr/lib/frr/bgpd /var/crash/crash1/CoreDump
[New LWP 1672]
[New LWP 1673]
[New LWP 1675]
[New LWP 1674]
[New LWP 1705]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M rpki'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f931a10c7c0 (LWP 1672))]
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f931a536859 in __GI_abort () at abort.c:79
#2  0x00007f931a9162f4 in ?? () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#3  <signal handler called>
#4  0x000055c0e0f83802 in ?? ()
#5  0x000055c0e0f85b6a in bgp_zebra_announce ()
#6  0x000055c0e0f395a2 in ?? ()
#7  0x000055c0e0f396ee in ?? ()
#8  0x00007f931a92fd08 in work_queue_run () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#9  0x00007f931a925c1a in thread_call () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#10 0x00007f931a8ed298 in frr_run () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#11 0x000055c0e0ee8d89 in main ()

Crash 2

napsterbater@ATL1-US:/var/crash$ gdb -ex 'bt' --batch /usr/lib/frr/bgpd /var/crash/crash2/CoreDump
[New LWP 54497]
[New LWP 54498]
[New LWP 54500]
[New LWP 54499]
[New LWP 54505]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M rpki'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f8c56e3e7c0 (LWP 54497))]
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f8c57268859 in __GI_abort () at abort.c:79
#2  0x00007f8c576482f4 in ?? () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#3  <signal handler called>
#4  0x000055c2b195d802 in ?? ()
#5  0x000055c2b195fb6a in bgp_zebra_announce ()
#6  0x000055c2b19135a2 in ?? ()
#7  0x000055c2b19136ee in ?? ()
#8  0x00007f8c57661d08 in work_queue_run () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#9  0x00007f8c57657c1a in thread_call () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#10 0x00007f8c5761f298 in frr_run () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#11 0x000055c2b18c2d89 in main ()

Crash 3

napsterbater@ATL1-US:/var/crash$ gdb -ex 'bt' --batch /usr/lib/frr/bgpd /var/crash/crash3/CoreDump
[New LWP 54822]
[New LWP 54825]
[New LWP 54830]
[New LWP 54824]
[New LWP 54823]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M rpki'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f8c781717c0 (LWP 54822))]
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f8c7859b859 in __GI_abort () at abort.c:79
#2  0x00007f8c7897b2f4 in ?? () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#3  <signal handler called>
#4  0x000055c232759802 in ?? ()
#5  0x000055c23275bb6a in bgp_zebra_announce ()
#6  0x000055c23270f5a2 in ?? ()
#7  0x000055c23270f6ee in ?? ()
#8  0x00007f8c78994d08 in work_queue_run () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#9  0x00007f8c7898ac1a in thread_call () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#10 0x00007f8c78952298 in frr_run () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#11 0x000055c2326bed89 in main ()
ton31337 commented 3 years ago

Please install frr-dbgsym and frr-rpki-rtrlib-dbgsym packages and try again with core dump.

Napsterbater commented 3 years ago

Please install frr-dbgsym and frr-rpki-rtrlib-dbgsym packages and try again with core dump.

Seems those are not available/built(?) for ubuntu?

napsterbater@ATL1-US:~$ sudo apt search frr
Sorting... Done
Full Text Search... Done
frr/unknown,now 7.5-0~ubuntu20.04 amd64 [installed]
  FRRouting suite of internet protocols (BGP, OSPF, IS-IS, ...)

frr-doc/unknown,now 7.5-0~ubuntu20.04 all [installed]
  FRRouting suite - user manual

frr-pythontools/unknown,now 7.5-0~ubuntu20.04 all [installed]
  FRRouting suite - Python tools

frr-rpki-rtrlib/unknown,now 7.5-0~ubuntu20.04 amd64 [installed]
  FRRouting suite - BGP RPKI support (rtrlib)

frr-snmp/unknown,now 7.5-0~ubuntu20.04 amd64 [installed]
  FRRouting suite - SNMP support

napsterbater@ATL1-US:~$ sudo apt search frr-dbgsym
Sorting... Done
Full Text Search... Done

napsterbater@ATL1-US:~$ sudo apt install frr-rpki-rtrlib-dbgsym frr-dbgsym
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package frr-rpki-rtrlib-dbgsym
E: Unable to locate package frr-dbgsym

Looking here, I only see them for Deb 9/10 https://deb.frrouting.org/frr/pool/frr-stable/f/frr/

ton31337 commented 3 years ago

@Napsterbater you are now able to install those packages, already added into the repo.

Napsterbater commented 3 years ago

sudo apt install frr-rpki-rtrlib-dbgsym frr-dbgsym Was successful.

Crash1

napsterbater@ATL1-US:/var/crash$ gdb -ex 'bt' --batch /usr/lib/frr/bgpd /var/crash/crash1/CoreDump
[New LWP 1672]
[New LWP 1673]
[New LWP 1675]
[New LWP 1674]
[New LWP 1705]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M rpki'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f931a10c7c0 (LWP 1672))]
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f931a536859 in __GI_abort () at abort.c:79
#2  0x00007f931a9162f4 in core_handler (signo=11, siginfo=0x7ffc19b986f0, context=<optimized out>) at lib/sigevent.c:228
#3  <signal handler called>
#4  0x000055c0e0f83802 in bgp_path_info_to_ipv6_nexthop (ifindex=ifindex@entry=0x7ffc19b98bdc, path=<optimized out>, path=<optimized out>) at bgpd/bgp_zebra.c:912
#5  0x000055c0e0f85b6a in bgp_zebra_announce (dest=dest@entry=0x55c0e25b27e0, p=p@entry=0x55c0e25b27e0, info=info@entry=0x55c0e32334f0, bgp=bgp@entry=0x55c0e1f7d620, afi=afi@entry=AFI_IP6, safi=safi@entry=SAFI_UNICAST) at bgpd/bgp_zebra.c:1387
#6  0x000055c0e0f395a2 in bgp_process_main_one (safi=SAFI_UNICAST, afi=AFI_IP6, dest=0x55c0e25b27e0, bgp=0x55c0e1f7d620) at bgpd/bgp_route.c:2820
#7  bgp_process_main_one (bgp=0x55c0e1f7d620, dest=0x55c0e25b27e0, afi=AFI_IP6, safi=SAFI_UNICAST) at bgpd/bgp_route.c:2591
#8  0x000055c0e0f396ee in bgp_process_wq (wq=<optimized out>, data=0x55c0e29aa410) at bgpd/bgp_route.c:2926
#9  0x00007f931a92fd08 in work_queue_run (thread=0x7ffc19ba9210) at lib/workqueue.c:291
#10 0x00007f931a925c1a in thread_call (thread=thread@entry=0x7ffc19ba9210) at lib/thread.c:1581
#11 0x00007f931a8ed298 in frr_run (master=0x55c0e19ff7a0) at lib/libfrr.c:1099
#12 0x000055c0e0ee8d89 in main (argc=8, argv=0x7ffc19ba9598) at bgpd/bgp_main.c:513

Crash 2

napsterbater@ATL1-US:/var/crash$ gdb -ex 'bt' --batch /usr/lib/frr/bgpd /var/crash/crash2/CoreDump
[New LWP 54497]
[New LWP 54498]
[New LWP 54500]
[New LWP 54499]
[New LWP 54505]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M rpki'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f8c56e3e7c0 (LWP 54497))]
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f8c57268859 in __GI_abort () at abort.c:79
#2  0x00007f8c576482f4 in core_handler (signo=11, siginfo=0x7ffcf9f7e1f0, context=<optimized out>) at lib/sigevent.c:228
#3  <signal handler called>
#4  0x000055c2b195d802 in bgp_path_info_to_ipv6_nexthop (ifindex=ifindex@entry=0x7ffcf9f7e6ac, path=<optimized out>, path=<optimized out>) at bgpd/bgp_zebra.c:912
#5  0x000055c2b195fb6a in bgp_zebra_announce (dest=dest@entry=0x55c2b4bbef20, p=p@entry=0x55c2b4bbef20, info=info@entry=0x55c2b5ba5bb0, bgp=bgp@entry=0x55c2b3e5c090, afi=afi@entry=AFI_IP6, safi=safi@entry=SAFI_UNICAST) at bgpd/bgp_zebra.c:1387
#6  0x000055c2b19135a2 in bgp_process_main_one (safi=SAFI_UNICAST, afi=AFI_IP6, dest=0x55c2b4bbef20, bgp=0x55c2b3e5c090) at bgpd/bgp_route.c:2820
#7  bgp_process_main_one (bgp=0x55c2b3e5c090, dest=0x55c2b4bbef20, afi=AFI_IP6, safi=SAFI_UNICAST) at bgpd/bgp_route.c:2591
#8  0x000055c2b19136ee in bgp_process_wq (wq=<optimized out>, data=0x55c2b5ba5a10) at bgpd/bgp_route.c:2926
#9  0x00007f8c57661d08 in work_queue_run (thread=0x7ffcf9f8ece0) at lib/workqueue.c:291
#10 0x00007f8c57657c1a in thread_call (thread=thread@entry=0x7ffcf9f8ece0) at lib/thread.c:1581
#11 0x00007f8c5761f298 in frr_run (master=0x55c2b38db7a0) at lib/libfrr.c:1099
#12 0x000055c2b18c2d89 in main (argc=8, argv=0x7ffcf9f8f068) at bgpd/bgp_main.c:513

Crash 3

napsterbater@ATL1-US:/var/crash$ gdb -ex 'bt' --batch /usr/lib/frr/bgpd /var/crash/crash3/CoreDump
[New LWP 54822]
[New LWP 54825]
[New LWP 54830]
[New LWP 54824]
[New LWP 54823]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M rpki'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f8c781717c0 (LWP 54822))]
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f8c7859b859 in __GI_abort () at abort.c:79
#2  0x00007f8c7897b2f4 in core_handler (signo=11, siginfo=0x7ffdb18ff9b0, context=<optimized out>) at lib/sigevent.c:228
#3  <signal handler called>
#4  0x000055c232759802 in bgp_path_info_to_ipv6_nexthop (ifindex=ifindex@entry=0x7ffdb18ffe7c, path=<optimized out>, path=<optimized out>) at bgpd/bgp_zebra.c:912
#5  0x000055c23275bb6a in bgp_zebra_announce (dest=dest@entry=0x55c233c12700, p=p@entry=0x55c233c12700, info=info@entry=0x55c233ac9d50, bgp=bgp@entry=0x55c23361e090, afi=afi@entry=AFI_IP6, safi=safi@entry=SAFI_UNICAST) at bgpd/bgp_zebra.c:1387
#6  0x000055c23270f5a2 in bgp_process_main_one (safi=SAFI_UNICAST, afi=AFI_IP6, dest=0x55c233c12700, bgp=0x55c23361e090) at bgpd/bgp_route.c:2820
#7  bgp_process_main_one (bgp=0x55c23361e090, dest=0x55c233c12700, afi=AFI_IP6, safi=SAFI_UNICAST) at bgpd/bgp_route.c:2591
#8  0x000055c23270f6ee in bgp_process_wq (wq=<optimized out>, data=0x55c234af8240) at bgpd/bgp_route.c:2926
#9  0x00007f8c78994d08 in work_queue_run (thread=0x7ffdb19104b0) at lib/workqueue.c:291
#10 0x00007f8c7898ac1a in thread_call (thread=thread@entry=0x7ffdb19104b0) at lib/thread.c:1581
#11 0x00007f8c78952298 in frr_run (master=0x55c23309d7a0) at lib/libfrr.c:1099
#12 0x000055c2326bed89 in main (argc=8, argv=0x7ffdb1910838) at bgpd/bgp_main.c:513
ton31337 commented 3 years ago

@Napsterbater that sounds much better. One last thing to ask you :) gdb -ex 'bt full' --batch /usr/lib/frr/bgpd /var/crash/crash3/CoreDump.

Napsterbater commented 3 years ago

Here is the output for all three.

crash1.txt crash2.txt crash3.txt

ton31337 commented 3 years ago

OK, this sounds like a prefix with the wrong attributes is handled. I need more context here. It looks like it's originated locally and then crashes when handling next-hop. That could be due to import vrf maybe. I pinged you in Slack, please provide me the details I asked you.

ton31337 commented 3 years ago

I managed to replicate this. This is as my first thought was (due to import vrf).

Napsterbater commented 3 years ago

Just copying some info from the slack convo.

Ok, so disabling all "import vrf" and bringing the affected peer up does not crash FRR. Reenabling the import vrf statements with that one peer up kills FRR. So I agree definitely related to VRFs/importing VRFs

FRR still crashes with 172.20.16.139 shutdown

debug.log show bgp vrf dn42 summary.txt show bgp ipv6 unicast.txt

ton31337 commented 3 years ago

Fixed in master, backported to 7.5, but not yet released as .deb. I believe soon you will have a new release.

ton31337 commented 3 years ago

@polychaeta autoclose in 2 days.

Napsterbater commented 3 years ago

Just wanted to confirm, this is indeed fixed for me on 7.5.1. Thanks!