Open ToshikiRen opened 1 week ago
I have been using the following script to reproduce the issue. I have modified the original service file to permit faster restarts so we could see the issue faster, the issue appears with the default configuration as well, if anyone wants to reproduce it using the default configuration you can change the sleep to match the service configuration.
#!/bin/bash
while true; do
zebra_count=$(ps -L -p $(pgrep zebra) | grep zebra_apic | wc -l)
if [ "$zebra_count" -lt 1 ]; then
echo "Number of zebra apic thread is less than 1. Exiting..."
date
pid=$(ps aux | grep "[j]ournalctl -fu frr" | awk '{print $2}')
kill -9 $pid
break
else
ps -L -p $(pgrep zebra)
echo "Restarting FRR..."
date
pid=$(ps aux | grep "[j]ournalctl -fu frr" | awk '{print $2}')
kill -9 $pid
journalctl -fu frr > frr_log.issue &
sudo systemctl restart frr
sleep 11
fi
done
As I understand we should see failed connecting synchronous zclient
in frr_log.issue
, right? But I can't see it running your script.
If the threads are missing and the owner of the socket is root, then the issue occured. The missing failed connecting synchronous zclient
might be related to the debug settings from frr that you have setup. In case that one is missing you can also look for the:
bgpd[9945]: [TBNSW-XXXBM] sendmsg_zebra_rnh: We have not connected yet, cannot send nexthops
bgpd[9945]: [HXW3G-K1M2A] sendmsg_zebra_rnh: sending cmd ZEBRA_NEXTHOP_REGISTER for 20.20.0.1/32 (vrf VRF default)
bgpd[9945]: [YTHK0-FSPPJ][EC 33554500] sendmsg_nexthop: zclient_send_message() failed
Could you test this PR https://github.com/FRRouting/frr/pull/16749?
I still see the issue where the zserv.api
is owned by root instead of the frr
user on PR #16749
I looked with a friend at the code and we think that the issue might be in zserv_start
. If something goes wrong here the socket could be bound to user root
, but it should not be the case since the socket is an unix socket, right?
~# netstat -an | grep zserv
unix 2 [ ACC ] STREAM LISTENING 230024 /var/run/frr/zserv.api
The above netstat -an
output is from a device for which the issue occured. When it is working the output is the following:
~# netstat -an | grep zserv
unix 2 [ ACC ] STREAM LISTENING 337256 /var/run/frr/zserv.api
unix 3 [ ] STREAM CONNECTED 337320 /var/run/frr/zserv.api
Could you show the whole /var/run/frr
directory?
This is the entire content of the /run/frr
from when the issue occurs:
total 16K
drwxr-xr-x 2 frr frr 280 Sep 5 14:06 .
drwxr-xr-x 24 root root 920 Sep 5 08:38 ..
-rw-r--r-- 1 frr frr 7 Sep 5 14:06 mgmtd.pid
srwxrwx--- 1 frr frrvty 0 Sep 5 14:06 mgmtd.vty
srwx------ 1 frr frr 0 Sep 5 14:06 mgmtd_be.sock
srwx------ 1 frr frr 0 Sep 5 14:06 mgmtd_fe.sock
-rw-r--r-- 1 frr frr 7 Sep 5 14:06 staticd.pid
srwxrwx--- 1 frr frrvty 0 Sep 5 14:06 staticd.vty
-rw-r--r-- 1 root root 7 Sep 5 14:06 watchfrr.pid
-rw-r----- 1 root root 0 Sep 5 14:06 watchfrr.started
srwxrwx--- 1 root frrvty 0 Sep 5 14:06 watchfrr.vty
-rw-r--r-- 1 frr frr 7 Sep 5 14:06 zebra.pid
srwxrwx--- 1 frr frrvty 0 Sep 5 14:06 zebra.vty
srwx------ 1 root frr 0 Sep 5 14:06 zserv.api
Can you reopen it since the issue persists? P.S: Please let me know if there is anything else that I may provide in order to help
Discussed in https://github.com/FRRouting/frr/discussions/16638