search

FRRouting / frr

The FRRouting Protocol Suite
https://frrouting.org/
Other
3.34k stars 1.25k forks source link

Zebra crashes with segv working with netns-based VRFs #15164

Open odivlad opened 9 months ago

odivlad commented 9 months ago

I run FRR with next /etc/frr/daemons config:

bgpd=yes
vtysh_enable=yes
zebra_options="  -A 127.0.0.1 -s 90000000 --vrfwnetns"
bgpd_options="   -A 127.0.0.1"

Sometimes when adding or deleting netns-based VRF zebra daemon crashes with SIGSEGV.

When create with these commands:

ip netns add dxpeer-A0869961
ip link add dxpeer-A0869961 type veth peer name eth0 netns dxpeer-A0869961
ip netns exec dxpeer-A0869961 ip link set eth0 up
ip link set dxpeer-A0869961 up

sometimes daemon crashes with next trace.

zebra[30200]: [GDAET-J7Q7G] NS notify : created VRF dxpeer-A0869961 NS /run/netns/dxpeer-A0869961
ZEBRA[30200]: Received signal 11 at 1705439706 (si_addr 0x14fcaab94, PC 0x475388); aborting...
ZEBRA[30200]: /usr/lib64/libfrr.so.0(zlog_backtrace_sigsafe+0x74) [0x7fc7f09a6a44]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(zlog_signal+0xec) [0x7fc7f09a6c5c]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(+0xc2662) [0x7fc7f09cf662]
ZEBRA[30200]: /usr/lib64/libpthread.so.0(+0xf630) [0x7fc7ef64e630]
ZEBRA[30200]: /usr/lib/frr/zebra() [0x475388]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(work_queue_run+0xd0) [0x7fc7f09e9630]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(thread_call+0x7b) [0x7fc7f09df53b]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(frr_run+0xe0) [0x7fc7f099f8f0]
ZEBRA[30200]: /usr/lib/frr/zebra(main+0x378) [0x41e868]
ZEBRA[30200]: /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fc7ef293555]
ZEBRA[30200]: /usr/lib/frr/zebra() [0x41f4e8]
ZEBRA[30200]: in thread work_queue_run scheduled from lib/workqueue.c:140 work_queue_schedule()

When delete with these commands:

ip netns exec dxvif-13C916E0 ip link set eth0 down
ip netns delete dxvif-13C916E0
vtysh -c 'conf t
no vrf dxvif-13C916E0
end
write memory'

sometimes I get this trace:

ZEBRA[10400]: Received signal 11 at 1705438022 (si_addr 0x100000041, PC 0x475388); aborting...
ZEBRA[10400]: /usr/lib64/libfrr.so.0(zlog_backtrace_sigsafe+0x74) [0x7fae5d743a44]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(zlog_signal+0xec) [0x7fae5d743c5c]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(+0xc2662) [0x7fae5d76c662]
ZEBRA[10400]: /usr/lib64/libpthread.so.0(+0xf630) [0x7fae5c3eb630]
ZEBRA[10400]: /usr/lib/frr/zebra() [0x475388]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(work_queue_run+0xd0) [0x7fae5d786630]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(thread_call+0x7b) [0x7fae5d77c53b]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(frr_run+0xe0) [0x7fae5d73c8f0]
ZEBRA[10400]: /usr/lib/frr/zebra(main+0x378) [0x41e868]
ZEBRA[10400]: /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fae5c030555]
ZEBRA[10400]: /usr/lib/frr/zebra() [0x41f4e8]
watchfrr[17333]: [HD38Q-0HBRT][EC 268435457] zebra state -> down : read returned EOF

Strace output right before and during crash:

[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000001}, msg_namelen=12, msg_iov=[{iov_base={{len=1284, type=RTM_NEWLINK, flags=0, seq=0, pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_ETHER, ifi_index=if_nametoindex("dxpeer-BFDCD441"), ifi_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST|IFF_LOWER_UP, ifi_change=0}, [{{nla_len=20, nla_type=IFLA_IFNAME}, "dxpeer-BFDCD441"}, {{nla_len=8, nla_type=IFLA_TXQLEN}, 1000}, {{nla_len=5, nla_type=IFLA_OPERSTATE}, 6}, {{nla_len=5, nla_type=IFLA_LINKMODE}, 0}, {{nla_len=8, nla_type=IFLA_MTU}, 1500}, {{nla_len=8, nla_type=IFLA_GROUP}, 0}, {{nla_len=8, nla_type=IFLA_PROMISCUITY}, 0}, {{nla_len=8, nla_type=IFLA_NUM_TX_QUEUES}, 1}, {{nla_len=8, nla_type=IFLA_GSO_MAX_SEGS}, 65535}, {{nla_len=8, nla_type=IFLA_GSO_MAX_SIZE}, 65536}, {{nla_len=8, nla_type=IFLA_NUM_RX_QUEUES}, 1}, {{nla_len=5, nla_type=IFLA_CARRIER}, 1}, {{nla_len=12, nla_type=IFLA_QDISC}, "noqueue"}, {{nla_len=8, nla_type=IFLA_CARRIER_CHANGES}, 2}, {{nla_len=5, nla_type=IFLA_PROTO_DOWN}, 0}, {{nla_len=36, nla_type=IFLA_MAP}, {mem_start=0, mem_end=0, base_addr=0, irq=0, dma=0, port=0}}, {{nla_len=10, nla_type=IFLA_ADDRESS}, "\xd6\x79\xe5\x6c\xda\x71"}, {{nla_len=10, nla_type=IFLA_BROADCAST}, "\xff\xff\xff\xff\xff\xff"}, {{nla_len=196, nla_type=IFLA_STATS64}, {rx_packets=0, tx_packets=0, rx_bytes=0, tx_bytes=0, rx_errors=0, tx_errors=0, rx_dropped=0, tx_dropped=0, multicast=0, collisions=0, rx_length_errors=0, rx_over_errors=0, rx_crc_errors=0, rx_frame_errors=0, rx_fifo_errors=0, rx_missed_errors=0, tx_aborted_errors=0, tx_carrier_errors=0, tx_fifo_errors=0, tx_heartbeat_errors=0, tx_window_errors=0, rx_compressed=0, tx_compressed=0, rx_nohandler=0}}, {{nla_len=100, nla_type=IFLA_STATS}, {rx_packets=0, tx_packets=0, rx_bytes=0, tx_bytes=0, rx_errors=0, tx_errors=0, rx_dropped=0, tx_dropped=0, multicast=0, collisions=0, rx_length_errors=0, rx_over_errors=0, rx_crc_errors=0, rx_frame_errors=0, rx_fifo_errors=0, rx_missed_errors=0, tx_aborted_errors=0, tx_carrier_errors=0, tx_fifo_errors=0, tx_heartbeat_errors=0, tx_window_errors=0, rx_compressed=0, tx_compressed=0, rx_nohandler=0}}, {{nla_len=16, nla_type=IFLA_LINKINFO}, {{nla_len=9, nla_type=IFLA_INFO_KIND}, "veth"}}, {{nla_len=8, nla_type=IFLA_LINK_NETNSID}, 4}, {{nla_len=8, nla_type=IFLA_LINK}, 8}, {{nla_len=728, nla_type=IFLA_AF_SPEC}, [{{nla_len=120, nla_type=AF_INET}, {{nla_len=116, nla_type=IFLA_INET_CONF}, [[IPV4_DEVCONF_FORWARDING-1] = 0, [IPV4_DEVCONF_MC_FORWARDING-1] = 0, [IPV4_DEVCONF_PROXY_ARP-1] = 0, [IPV4_DEVCONF_ACCEPT_REDIRECTS-1] = 0, [IPV4_DEVCONF_SECURE_REDIRECTS-1] = 1, [IPV4_DEVCONF_SEND_REDIRECTS-1] = 0, [IPV4_DEVCONF_SHARED_MEDIA-1] = 1, [IPV4_DEVCONF_RP_FILTER-1] = 1, [IPV4_DEVCONF_ACCEPT_SOURCE_ROUTE-1] = 0, [IPV4_DEVCONF_BOOTP_RELAY-1] = 0, [IPV4_DEVCONF_LOG_MARTIANS-1] = 0, [IPV4_DEVCONF_TAG-1] = 0, [IPV4_DEVCONF_ARPFILTER-1] = 0, [IPV4_DEVCONF_MEDIUM_ID-1] = 0, [IPV4_DEVCONF_NOXFRM-1] = 0, [IPV4_DEVCONF_NOPOLICY-1] = 0, [IPV4_DEVCONF_FORCE_IGMP_VERSION-1] = 0, [IPV4_DEVCONF_ARP_ANNOUNCE-1] = 0, [IPV4_DEVCONF_ARP_IGNORE-1] = 0, [IPV4_DEVCONF_PROMOTE_SECONDARIES-1] = 1, [IPV4_DEVCONF_ARP_ACCEPT-1] = 0, [IPV4_DEVCONF_ARP_NOTIFY-1] = 0, [IPV4_DEVCONF_ACCEPT_LOCAL-1] = 0, [IPV4_DEVCONF_SRC_VMARK-1] = 0, [IPV4_DEVCONF_PROXY_ARP_PVLAN-1] = 0, [IPV4_DEVCONF_ROUTE_LOCALNET-1] = 0, [IPV4_DEVCONF_IGMPV2_UNSOLICITED_REPORT_INTERVAL-1] = 10000, [IPV4_DEVCONF_IGMPV3_UNSOLICITED_REPORT_INTERVAL-1] = 1000]}}, {{nla_len=604, nla_type=AF_INET6}, [{{nla_len=8, nla_type=IFLA_INET6_FLAGS}, IF_READY}, {{nla_len=20, nla_type=IFLA_INET6_CACHEINFO}, {max_reasm_len=65535, tstamp=1342870334, reachable_time=22610, retrans_time=1000}}, {{nla_len=192, nla_type=IFLA_INET6_CONF}, [[DEVCONF_FORWARDING] = 0, [DEVCONF_HOPLIMIT] = 64, [DEVCONF_MTU6] = 1500, [DEVCONF_ACCEPT_RA] = 1, [DEVCONF_ACCEPT_REDIRECTS] = 0, [DEVCONF_AUTOCONF] = 1, [DEVCONF_DAD_TRANSMITS] = 1, [DEVCONF_RTR_SOLICITS] = 3, [DEVCONF_RTR_SOLICIT_INTERVAL] = 4000, [DEVCONF_RTR_SOLICIT_DELAY] = 1000, [DEVCONF_USE_TEMPADDR] = 0, [DEVCONF_TEMP_VALID_LFT] = 604800, [DEVCONF_TEMP_PREFERED_LFT] = 86400, [DEVCONF_REGEN_MAX_RETRY] = 3, [DEVCONF_MAX_DESYNC_FACTOR] = 600, [DEVCONF_MAX_ADDRESSES] = 16, [DEVCONF_FORCE_MLD_VERSION] = 0, [DEVCONF_ACCEPT_RA_DEFRTR] = 1, [DEVCONF_ACCEPT_RA_PINFO] = 1, [DEVCONF_ACCEPT_RA_RTR_PREF] = 1, [DEVCONF_RTR_PROBE_INTERVAL] = 60000, [DEVCONF_ACCEPT_RA_RT_INFO_MAX_PLEN] = 0, [DEVCONF_PROXY_NDP] = 0, [DEVCONF_OPTIMISTIC_DAD] = 0, [DEVCONF_ACCEPT_SOURCE_ROUTE] = 0, [DEVCONF_MC_FORWARDING] = 0, [DEVCONF_DISABLE_IPV6] = 1, [DEVCONF_ACCEPT_DAD] = 1, [DEVCONF_FORCE_TLLAO] = 0, [DEVCONF_NDISC_NOTIFY] = 0, [DEVCONF_MLDV1_UNSOLICITED_REPORT_INTERVAL] = 10000, [DEVCONF_MLDV2_UNSOLICITED_REPORT_INTERVAL] = 1000, [DEVCONF_SUPPRESS_FRAG_NDISC] = 0, [DEVCONF_ACCEPT_RA_FROM_LOCAL] = 0, [DEVCONF_USE_OPTIMISTIC] = 0, [DEVCONF_ACCEPT_RA_MTU] = 0, [DEVCONF_STABLE_SECRET] = 0, [DEVCONF_USE_OIF_ADDRS_ONLY] = 0, [DEVCONF_ACCEPT_RA_MIN_HOP_LIMIT] = 0, [DEVCONF_IGNORE_ROUTES_WITH_LINKDOWN] = 0, [DEVCONF_DROP_UNICAST_IN_L2_MULTICAST] = 0, [DEVCONF_DROP_UNSOLICITED_NA] = 0, [DEVCONF_KEEP_ADDR_ON_DOWN] = 0, [DEVCONF_RTR_SOLICIT_MAX_INTERVAL] = 0, [DEVCONF_SEG6_ENABLED] = 0, [DEVCONF_SEG6_REQUIRE_HMAC] = 0, [DEVCONF_ENHANCED_DAD] = 1]}, {{nla_len=300, nla_type=IFLA_INET6_STATS}, [[IPSTATS_MIB_NUM] = 37, [IPSTATS_MIB_INPKTS] = 0, [IPSTATS_MIB_INOCTETS] = 0, [IPSTATS_MIB_INDELIVERS] = 0, [IPSTATS_MIB_OUTFORWDATAGRAMS] = 0, [IPSTATS_MIB_OUTPKTS] = 0, [IPSTATS_MIB_OUTOCTETS] = 0, [IPSTATS_MIB_INHDRERRORS] = 0, [IPSTATS_MIB_INTOOBIGERRORS] = 0, [IPSTATS_MIB_INNOROUTES] = 0, [IPSTATS_MIB_INADDRERRORS] = 0, [IPSTATS_MIB_INUNKNOWNPROTOS] = 0, [IPSTATS_MIB_INTRUNCATEDPKTS] = 0, [IPSTATS_MIB_INDISCARDS] = 0, [IPSTATS_MIB_OUTDISCARDS] = 0, [IPSTATS_MIB_OUTNOROUTES] = 0, [IPSTATS_MIB_REASMTIMEOUT] = 0, [IPSTATS_MIB_REASMREQDS] = 0, [IPSTATS_MIB_REASMOKS] = 0, [IPSTATS_MIB_REASMFAILS] = 0, [IPSTATS_MIB_FRAGOKS] = 0, [IPSTATS_MIB_FRAGFAILS] = 0, [IPSTATS_MIB_FRAGCREATES] = 0, [IPSTATS_MIB_INMCASTPKTS] = 0, [IPSTATS_MIB_OUTMCASTPKTS] = 0, [IPSTATS_MIB_INBCASTPKTS] = 0, [IPSTATS_MIB_OUTBCASTPKTS] = 0, [IPSTATS_MIB_INMCASTOCTETS] = 0, [IPSTATS_MIB_OUTMCASTOCTETS] = 0, [IPSTATS_MIB_INBCASTOCTETS] = 0, [IPSTATS_MIB_OUTBCASTOCTETS] = 0, [IPSTATS_MIB_CSUMERRORS] = 0, [IPSTATS_MIB_NOECTPKTS] = 0, [IPSTATS_MIB_ECT1PKTS] = 0, [IPSTATS_MIB_ECT0PKTS] = 0, [IPSTATS_MIB_CEPKTS] = 0, [36 /* IPSTATS_MIB_??? */] = 0]}, {{nla_len=52, nla_type=IFLA_INET6_ICMP6STATS}, [[ICMP6_MIB_NUM] = 6, [ICMP6_MIB_INMSGS] = 0, [ICMP6_MIB_INERRORS] = 0, [ICMP6_MIB_OUTMSGS] = 0, [ICMP6_MIB_OUTERRORS] = 0, [ICMP6_MIB_CSUMERRORS] = 0]}, {{nla_len=20, nla_type=IFLA_INET6_TOKEN}, ::}, {{nla_len=5, nla_type=IFLA_INET6_ADDR_GEN_MODE}, IN6_ADDR_GEN_MODE_EUI64}]}]}]}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1284
[pid  4888] read(54,  <unfinished ...>
[pid  4885] read(49,  <unfinished ...>
[pid  4876] recvmsg(12,  <unfinished ...>
[pid  4888] <... read resumed>0x7f3f721f0c30, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4885] <... read resumed>0x7f3f729f1c30, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4888] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000004}, msg_namelen=12, msg_iov=[{iov_base={{len=76, type=RTM_NEWNEIGH, flags=0, seq=0, pid=0}, {ndm_family=AF_INET, ndm_ifindex=if_nametoindex("cluster"), ndm_state=NUD_REACHABLE, ndm_flags=0, ndm_type=RTN_UNICAST}, [{{nla_len=8, nla_type=NDA_DST}, 10.1.1.21}, {{nla_len=10, nla_type=NDA_LLADDR}, 82.84.0.73}, {{nla_len=8, nla_type=NDA_PROBES}, 1}, {{nla_len=20, nla_type=NDA_CACHEINFO}, {ndm_confirmed=451, ndm_used=451, ndm_updated=0, ndm_refcnt=2}}]}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 76
[pid  4888] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4885] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4888] ppoll([{fd=53, events=POLLIN|POLLOUT}, {fd=56, events=POLLIN}, {fd=54, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] recvmsg(12,  <unfinished ...>
[pid  4888] <... ppoll resumed>)        = 1 ([{fd=53, revents=POLLOUT}])
[pid  4885] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4876] <... recvmsg resumed>{msg_namelen=12}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4888] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4885] ppoll([{fd=37, events=POLLIN|POLLOUT}, {fd=51, events=POLLIN}, {fd=49, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] write(8, "\1", 1 <unfinished ...>
[pid  4888] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4885] <... ppoll resumed>)        = 1 ([{fd=37, revents=POLLOUT}])
[pid  4876] <... write resumed>)        = 1
[pid  4888] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4885] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4876] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4888] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=4447}, ru_stime={tv_sec=0, tv_usec=28615}, ...}) = 0
[pid  4885] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4876] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=295500}, ...}) = 0
[pid  4888] writev(53, [{iov_base="\0T\376\6\0\0\0\0\0\5dxpeer-BFDCD441\0\0\0\0\0\0\0\t)\5\0\0\0\0\0\1\20\3\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\326y\345l\332q\0\0T\376\6\0\0\0\0\0\4dxpeer-BFDCD441\0\0\0\0\0\0\0\t)\5\0\0\0\0\0\1\20C\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\326y\345l\332q\0", iov_len=168}], 1 <unfinished ...>
[pid  4885] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4876] recvmsg(33,  <unfinished ...>
[pid  4888] <... writev resumed>)       = 168
[pid  4885] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=9704}, ru_stime={tv_sec=0, tv_usec=33073}, ...}) = 0
[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000400}, msg_namelen=12, msg_iov=[{iov_base={{len=116, type=0x18 /* NLMSG_??? */, flags=0, seq=0, pid=0}, "\x0a\x08\x00\x00\xff\x03\x00\x01\x00\x00\x00\x00\x08\x00\x0f\x00\xff\x00\x00\x00\x14\x00\x01\x00\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x04\x00\x08\x00\x00\x00\x08\x00\x06\x00\x00\x01\x00\x00\x24\x00\x0c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x14\x00\x00\x00\x00\x00"}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 116
[pid  4888] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4885] writev(37, [{iov_base="\0T\376\6\0\0\0\0\0\5dxpeer-BFDCD441\0\0\0\0\0\0\0\t)\5\0\0\0\0\0\1\20\3\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\326y\345l\332q\0\0T\376\6\0\0\0\0\0\4dxpeer-BFDCD441\0\0\0\0\0\0\0\t)\5\0\0\0\0\0\1\20C\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\326y\345l\332q\0", iov_len=168}], 1 <unfinished ...>
[pid  4876] write(8, "\1", 1 <unfinished ...>
[pid  4888] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=4447}, ru_stime={tv_sec=0, tv_usec=28680}, ...}) = 0
[pid  4885] <... writev resumed>)       = 168
[pid  4876] <... write resumed>)        = 1
[pid  4888] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] recvmsg(33,  <unfinished ...>
[pid  4888] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4885] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000400}, msg_namelen=12, msg_iov=[{iov_base={{len=116, type=0x18 /* NLMSG_??? */, flags=0, seq=0, pid=0}, "\x0a\x40\x00\x00\xfe\x02\x00\x01\x00\x00\x00\x00\x08\x00\x0f\x00\xfe\x00\x00\x00\x14\x00\x01\x00\xfe\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x04\x00\x08\x00\x00\x00\x08\x00\x06\x00\x00\x01\x00\x00\x24\x00\x0c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x14\x00\x00\x00\x00\x00"}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 116
[pid  4888] ppoll([{fd=53, events=POLLIN}, {fd=56, events=POLLIN}, {fd=54, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4885] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=9704}, ru_stime={tv_sec=0, tv_usec=33132}, ...}) = 0
[pid  4876] recvmsg(33,  <unfinished ...>
[pid  4885] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000001}, msg_namelen=12, msg_iov=[{iov_base={{len=1276, type=0x10 /* NLMSG_??? */, flags=0, seq=0, pid=0}, "\x00\x00\x01\x00\x08\x00\x00\x00\x43\x10\x01\x00\x00\x00\x00\x00\x09\x00\x03\x00\x65\x74\x68\x30\x00\x00\x00\x00\x08\x00\x0d\x00\xe8\x03\x00\x00\x05\x00\x10\x00\x06\x00\x00\x00\x05\x00\x11\x00\x00\x00\x00\x00\x08\x00\x04\x00\xdc\x05\x00\x00\x08\x00\x1b\x00\x00\x00\x00\x00\x08\x00\x1e\x00\x00\x00\x00\x00\x08\x00\x1f\x00\x01\x00\x00\x00\x08\x00\x28\x00\xff\xff\x00\x00\x08\x00\x29\x00\x00\x00\x01\x00\x08\x00\x20\x00\x01\x00\x00\x00\x05\x00\x21\x00\x01\x00\x00\x00\x0c\x00\x06\x00\x6e\x6f\x71\x75\x65\x75\x65\x00\x08\x00\x23\x00\x02\x00\x00\x00\x05\x00\x27\x00\x00\x00\x00\x00\x24\x00\x0e\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0a\x00\x01\x00\xfe\xcf\x39\x0e\xcb\x94\x00\x00\x0a\x00\x02\x00\xff\xff\xff\xff\xff\xff\x00\x00\xc4\x00\x17\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x64\x00\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x12\x00\x09\x00\x01\x00\x76\x65\x74\x68\x00\x00\x00\x00\x08\x00\x25\x00\x00\x00\x00\x00\x08\x00\x05\x00\x29\x09\x00\x00\xd8\x02\x1a\x00\x78\x00\x02\x00\x74\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x27\x00\x00\xe8\x03\x00\x00\x5c\x02\x0a\x00\x08\x00\x01\x00\x00\x00\x00\x80\x14\x00\x05\x00\xff\xff\x00\x00\x3e\x93\x0a\x50\x0d\x75\x00\x00\xe8\x03\x00\x00\xc0\x00\x02\x00\x00\x00\x00\x00\x40\x00\x00\x00\xdc\x05\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x03\x00\x00\x00\xa0\x0f\x00\x00\xe8\x03\x00\x00\x00\x00\x00\x00\x80\x3a\x09\x00\x80\x51\x01\x00\x03\x00\x00\x00\x58\x02\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x60\xea\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x27\x00\x00\xe8\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x2c\x01\x03\x00\x25\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"...}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1276
[pid  4885] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4876] write(50, "\1", 1 <unfinished ...>
[pid  4885] ppoll([{fd=37, events=POLLIN}, {fd=51, events=POLLIN}, {fd=49, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] <... write resumed>)        = 1
[pid  4885] <... ppoll resumed>)        = 1 ([{fd=49, revents=POLLIN}])
[pid  4876] write(55, "\1", 1 <unfinished ...>
[pid  4885] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4876] <... write resumed>)        = 1
[pid  4888] <... ppoll resumed>)        = 1 ([{fd=54, revents=POLLIN}])
[pid  4885] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4876] recvmsg(33,  <unfinished ...>
[pid  4888] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4885] read(49,  <unfinished ...>
[pid  4876] <... recvmsg resumed>{msg_namelen=12}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4888] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4885] <... read resumed>"\1", 64) = 1
[pid  4876] write(8, "\1", 1 <unfinished ...>
[pid  4888] read(54,  <unfinished ...>
[pid  4885] read(49,  <unfinished ...>
[pid  4876] <... write resumed>)        = 1
[pid  4888] <... read resumed>"\1", 64) = 1
[pid  4885] <... read resumed>0x7f3f729f1c30, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4876] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4888] read(54,  <unfinished ...>
[pid  4885] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=295744}, ...}) = 0
[pid  4888] <... read resumed>0x7f3f721f0c30, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4885] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4888] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4885] ppoll([{fd=37, events=POLLIN|POLLOUT}, {fd=51, events=POLLIN}, {fd=49, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4888] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4876] rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 TERM],  <unfinished ...>
[pid  4888] ppoll([{fd=53, events=POLLIN|POLLOUT}, {fd=56, events=POLLIN}, {fd=54, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4885] <... ppoll resumed>)        = 1 ([{fd=37, revents=POLLOUT}])
[pid  4888] <... ppoll resumed>)        = 1 ([{fd=53, revents=POLLOUT}])
[pid  4885] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4876] <... rt_sigprocmask resumed>[], 8) = 0
[pid  4888] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4885] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4888] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4876] ppoll([{fd=12, events=POLLIN}, {fd=15, events=POLLIN}, {fd=43, events=POLLIN}, {fd=47, events=POLLIN}, {fd=48, events=POLLIN}, {fd=58, events=POLLIN}, {fd=18, events=POLLIN}, {fd=23, events=POLLIN}, {fd=28, events=POLLIN}, {fd=33, events=POLLIN}, {fd=7, events=POLLIN}], 11, {tv_sec=0, tv_nsec=0}, [], 8 <unfinished ...>
[pid  4888] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4885] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4888] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=4447}, ru_stime={tv_sec=0, tv_usec=28807}, ...}) = 0
[pid  4885] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=9704}, ru_stime={tv_sec=0, tv_usec=33261}, ...}) = 0
[pid  4888] writev(53, [{iov_base="\0T\376\6\0\0\0\2\0\4eth0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\5\0\0\0\0\0\1\20C\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\376\3179\16\313\224\0", iov_len=84}], 1 <unfinished ...>
[pid  4876] <... ppoll resumed>)        = 1 ([{fd=7, revents=POLLIN}], left {tv_sec=0, tv_nsec=0})
[pid  4888] <... writev resumed>)       = 84
[pid  4885] writev(37, [{iov_base="\0T\376\6\0\0\0\2\0\4eth0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\5\0\0\0\0\0\1\20C\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\376\3179\16\313\224\0", iov_len=84}], 1 <unfinished ...>
[pid  4876] rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid  4885] <... writev resumed>)       = 84
[pid  4888] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4876] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4888] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=4447}, ru_stime={tv_sec=0, tv_usec=28836}, ...}) = 0
[pid  4876] read(7,  <unfinished ...>
[pid  4888] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] <... read resumed>"\1\1\1\1", 64) = 4
[pid  4888] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4885] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4888] ppoll([{fd=53, events=POLLIN}, {fd=56, events=POLLIN}, {fd=54, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] read(7,  <unfinished ...>
[pid  4885] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=9704}, ru_stime={tv_sec=0, tv_usec=33293}, ...}) = 0
[pid  4876] <... read resumed>0x7ffe5328a300, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4885] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4885] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4876] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=295828}, ...}) = 0
[pid  4885] ppoll([{fd=37, events=POLLIN}, {fd=51, events=POLLIN}, {fd=49, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] getrusage(RUSAGE_THREAD, {ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=295873}, ...}) = 0
[pid  4876] rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 TERM], [], 8) = 0
[pid  4876] ppoll([{fd=12, events=POLLIN}, {fd=15, events=POLLIN}, {fd=43, events=POLLIN}, {fd=47, events=POLLIN}, {fd=48, events=POLLIN}, {fd=58, events=POLLIN}, {fd=18, events=POLLIN}, {fd=23, events=POLLIN}, {fd=28, events=POLLIN}, {fd=33, events=POLLIN}, {fd=7, events=POLLIN}], 11, {tv_sec=0, tv_nsec=5000000}, [], 8) = 0 (Timeout)
[pid  4876] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  4876] rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 TERM], [], 8) = 0
[pid  4876] ppoll([{fd=12, events=POLLIN}, {fd=15, events=POLLIN}, {fd=43, events=POLLIN}, {fd=47, events=POLLIN}, {fd=48, events=POLLIN}, {fd=58, events=POLLIN}, {fd=18, events=POLLIN}, {fd=23, events=POLLIN}, {fd=28, events=POLLIN}, {fd=33, events=POLLIN}, {fd=7, events=POLLIN}], 11, {tv_sec=0, tv_nsec=0}, [], 8) = 0 (Timeout)
[pid  4876] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  4876] getrusage(RUSAGE_THREAD, {ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=296070}, ...}) = 0
[pid  4876] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x41} ---
[pid  4876] rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f3f76a29630}, NULL, 8) = 0
[pid  4876] rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f3f76a29630}, NULL, 8) = 0
[pid  4876] rt_sigprocmask(SIG_UNBLOCK, [ALRM], NULL, 8) = 0
[pid  4876] alarm(1)                    = 0
[pid  4876] openat(3, "crashlog", O_WRONLY|O_CREAT|O_APPEND, 0600) = 59
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="Received signal 11 at 1705442216 (si_addr 0x41, PC 0x475388); aborting...", iov_len=73}, {iov_base="\n", iov_len=1}], 4) = 81
[pid  4876] socket(AF_UNIX, SOCK_DGRAM, 0) = 60
[pid  4876] connect(60, {sa_family=AF_UNIX, sun_path="/dev/log"}, 110) = 0
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="Received signal 11 at 1705442216 (si_addr 0x41, PC 0x475388); aborting...", iov_len=73}], 2) = 89
[pid  4876] futex(0x7f3f76a17cc0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  4876] futex(0x7f3f76e4b190, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(zlog_backtrace_sigsafe+0x74) [0x7f3f77d81a44]", iov_len=68}, {iov_base="\n", iov_len=1}], 4) = 76
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(zlog_backtrace_sigsafe+0x74) [0x7f3f77d81a44]", iov_len=68}], 2) = 84
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(zlog_signal+0xec) [0x7f3f77d81c5c]", iov_len=57}, {iov_base="\n", iov_len=1}], 4) = 65
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(zlog_signal+0xec) [0x7f3f77d81c5c]", iov_len=57}], 2) = 73
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(+0xc2662) [0x7f3f77daa662]", iov_len=49}, {iov_base="\n", iov_len=1}], 4) = 57
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(+0xc2662) [0x7f3f77daa662]", iov_len=49}], 2) = 65
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libpthread.so.0(+0xf630) [0x7f3f76a29630]", iov_len=52}, {iov_base="\n", iov_len=1}], 4) = 60
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libpthread.so.0(+0xf630) [0x7f3f76a29630]", iov_len=52}], 2) = 68
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib/frr/zebra() [0x475388]", iov_len=31}, {iov_base="\n", iov_len=1}], 4) = 39
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib/frr/zebra() [0x475388]", iov_len=31}], 2) = 47
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(work_queue_run+0xd0) [0x7f3f77dc4630]", iov_len=60}, {iov_base="\n", iov_len=1}], 4) = 68
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(work_queue_run+0xd0) [0x7f3f77dc4630]", iov_len=60}], 2) = 76
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(thread_call+0x7b) [0x7f3f77dba53b]", iov_len=57}, {iov_base="\n", iov_len=1}], 4) = 65
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(thread_call+0x7b) [0x7f3f77dba53b]", iov_len=57}], 2) = 73
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(frr_run+0xe0) [0x7f3f77d7a8f0]", iov_len=53}, {iov_base="\n", iov_len=1}], 4) = 61
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(frr_run+0xe0) [0x7f3f77d7a8f0]", iov_len=53}], 2) = 69
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib/frr/zebra(main+0x378) [0x41e868]", iov_len=41}, {iov_base="\n", iov_len=1}], 4) = 49
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib/frr/zebra(main+0x378) [0x41e868]", iov_len=41}], 2) = 57
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f3f7666e555]", iov_len=61}, {iov_base="\n", iov_len=1}], 4) = 69
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f3f7666e555]", iov_len=61}], 2) = 77
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib/frr/zebra() [0x41f4e8]", iov_len=31}, {iov_base="\n", iov_len=1}], 4) = 39
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib/frr/zebra() [0x41f4e8]", iov_len=31}], 2) = 47
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="in thread work_queue_run scheduled from lib/workqueue.c:140 work_queue_schedule()", iov_len=81}, {iov_base="\n", iov_len=1}], 4) = 89
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="in thread work_queue_run scheduled from lib/workqueue.c:140 work_queue_schedule()", iov_len=81}], 2) = 97
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="in thread work_queue_run scheduled from lib/workqueue.c:140 work_queue_schedule()", iov_len=81}], 2) = 97
[pid  4876] write(2, "core_handler: showing active allocations in memory group libfrr\n", 64) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Buffer                        :      4 *         24\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Host config                   :      3 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Command Tokens                :   4144 *         72\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Command Token Text            :   2967 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Command Token Help            :   2967 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Command Argument Name         :   1043 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  RCU thread                    :      4 *        128\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  FRR POSIX Thread              :      8 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  POSIX sync primitives         :      8 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Graph                         :     25 *          8\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Graph Node                    :   4848 *         32\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Hash                          :    548 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Hash Bucket                   :    947 *         32\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Hash Index                    :    274 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Hook entry                    :      8 *         48\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Interface                     :     55 *        272\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Connected                     :     11 *         48\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Link List                     :    313 *         40\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Link Node                     :    350 *         24\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Temporary memory              :      4 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Nexthop                       :     21 *        136\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  NetNS Context                 :     11 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  NetNS Name                    :      5 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Northbound Node               :    655 *       1192\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Northbound Configuration      :      2 *         16\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Northbound Configuration Entry:     11 *       1032\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Prefix                        :     11 *         48\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Privilege information         :      4 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route map                     :      4 *        120\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route map name                :      4 *         23\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route map index               :      2 *        152\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Stream                        :      4 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Stream FIFO                   :      5 *         64\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route table                   :    119 *         56\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route node                    :    198 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Thread                        :     41 *        152\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Thread master                 :     20 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Thread Poll Info              :     10 *       8192\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Thread stats                  :     26 *         88\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Typed-hash bucket             :     43 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Typed-heap array              :      1 *        576\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Vector                        :   9752 *         16\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Vector index                  :   9752 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  VRF                           :      5 *        216\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  VRF bit-map                   :     11 *          8\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  VTY                           :      2 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Work queue                    :      2 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Work queue item               :      1 *         24\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Work queue name string        :      1 *         22\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  YANG module                   :      7 *         48\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  log thread-local buffer       :      5 *      10272\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: showing active allocations in memory group logging subsystem\n", 75) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  syslog target                 :      1 *         56\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: showing active allocations in memory group Label Manager\n", 71) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: showing active allocations in memory group Table Manager\n", 71) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: showing active allocations in memory group zebra\n", 63) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra Interface Information   :     55 *        472\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra Netlink buffers         :      1 *     131072\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Opaque Data                   :      3 *          8\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra DPlane Provider         :      1 *        232\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Nexthop Group Entry           :     20 *        112\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Nexthop Group Connected       :     19 *         40\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra Name Space              :      5 *        320\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  PTM BFD process registration table.:      1 *         32\n", 82) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route Entry                   :     43 *         96\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  RIB destination               :     61 *         88\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  RIB table info                :     25 *         24\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra VRF table               :     25 *         56\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Nexthop tracking object       :      2 *        240\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  ZEBRA VRF                     :      5 *       5016\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Other Table                   :      4 *         56\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  MH global info                :      1 *        128\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] munmap(0x7f3f783d6000, 8192) = 0
[pid  4876] gettid()                    = 4876
[pid  4876] unlinkat(3, "logbuf.4876", 0) = 0
[pid  4876] rt_sigprocmask(SIG_UNBLOCK, [SEGV ALRM], NULL, 8) = 0
[pid  4876] tgkill(4876, 4876, SIGSEGV) = 0
[pid  4876] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_TKILL, si_pid=4876, si_uid=92} ---
[pid  4888] <... ppoll resumed> <unfinished ...>) = ?
[pid  4885] <... ppoll resumed> <unfinished ...>) = ?
[pid  4879] <... ppoll resumed> <unfinished ...>) = ?
[pid  4878] <... ppoll resumed> <unfinished ...>) = ?
[pid  4877] <... futex resumed>)        = ?
[pid  4888] +++ killed by SIGSEGV +++
[pid  4885] +++ killed by SIGSEGV +++
[pid  4879] +++ killed by SIGSEGV +++
[pid  4878] +++ killed by SIGSEGV +++
[pid  4877] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++

The issue was first observed with FRR 8.0 and confirmed with 9.1. OS: CentOS 7.8 Kernel: 3.10.0-1127.19.1.el7.x86_64

I'm ready to provide additional info if needed for debug.

pguibert6WIND commented 9 months ago

Hi, not reproduced using the inputs of this ticket on upstream. is it possible to get the back trace from GDB ?

odivlad commented 9 months ago

Hi, thanks for the response!

(gdb) bt
#0  zvrf_id (zvrf=0x1bdf8a0) at zebra/zebra_vrf.h:207
#1  rib_process (rn=0x1bdd040) at zebra/zebra_rib.c:1127
#2  process_subq_route (qindex=2 '\002', lnode=0x1bdd160) at zebra/zebra_rib.c:2386
#3  process_subq (qindex=<optimized out>, subq=0x1a00760) at zebra/zebra_rib.c:2428
#4  meta_queue_process (dummy=<optimized out>, data=0x1a00e20) at zebra/zebra_rib.c:2461
#5  0x00007f9a6cad2630 in work_queue_run (thread=0x7ffcb4558aa0) at lib/workqueue.c:291
#6  0x00007f9a6cac853b in thread_call (thread=thread@entry=0x7ffcb4558aa0) at lib/thread.c:1825
#7  0x00007f9a6ca888f0 in frr_run (master=0x184d9e0) at lib/libfrr.c:1155
#8  0x000000000041e868 in main (argc=9, argv=0x7ffcb4558e88) at zebra/main.c:470

Please, note, this backtrace is collected within frr frr-8.0.1-02.el7.x86_64.rpm, 'cause my environment uses it.

odivlad commented 9 months ago

@pguibert6WIND, hi! Do you need any additional information?

github-actions[bot] commented 3 months ago

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

frrbot[bot] commented 3 months ago

This issue will be automatically closed in the specified period unless there is further activity.

odivlad commented 3 months ago

The problem persists.

frrbot[bot] commented 3 months ago

This issue will no longer be automatically closed.