search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
3.94k stars 3.45k forks source link

19.07: net-snmp: snmpd not listening after reboot #13728

Open karlp opened 3 years ago

karlp commented 3 years ago

Maintainer: @stintel Environment: ath79, openwrt-19.07 branch (works fine in master!)

Description: snmpd responds perfectly well on first boot, but after reboot, it is non-responsive until it is restarted. snmp requests time out, no logging. Process is running, conf file is generated in /var/run/snmpd.conf as normal

Only hint is that the agentXSocket file is NOT created! (This is how I first came to this, my subagent wasn't working)

root@OpenWrt:/# pgrep -lfa snmpd
1248 /usr/sbin/snmpd -Lf /dev/null -f
root@OpenWrt:/# cat /var/run/snmpd.conf 
agentaddress UDP:161,UDP6:161
master agentx
agentXSocket /var/run/agentx.sock
sysLocation office
sysContact bofh@example.com
sysName HeartOfGold
com2sec ro default public
com2sec rw localhost private
com2sec6 ro default public
com2sec6 rw localhost private
group public v1 ro
group public v2c ro
group public usm ro
group private v1 rw
group private v2c rw
group private usm rw
view all included .1 
access public "" any noauth exact all none none
access private "" any noauth exact all all all
exec  filedescriptors /bin/cat /proc/sys/fs/file-nr
engineIDType 3
engineIDNic eth0
root@OpenWrt:/# ls -l /var/run/ag*
ls: /var/run/agent*: No such file or directory

After restart, works just fine, and agentx file is created and snmp requests are answered

root@OpenWrt:/# /etc/init.d/snmpd restart
root@OpenWrt:/# ls -l /var/run/ag*
srwxr-xr-x    1 root     root             0 Oct 20 10:27 /var/run/agentx.sock
root@OpenWrt:/#

Openwrt build info. (Tried various points to very early on 19.07 branch, but that was still with my own extra apps, this is a perfectly clean one)

root@OpenWrt:/# cat /etc/openwrt_release 
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07-SNAPSHOT'
DISTRIB_REVISION='r11221-7dd822983b'
DISTRIB_TARGET='ath79/generic'
DISTRIB_ARCH='mips_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07-SNAPSHOT r11221-7dd822983b'
DISTRIB_TAINTS='no-all'
root@OpenWrt:/#

Diffconfig on clean 19.07 branch

$ ./scripts/diffconfig.sh 
CONFIG_TARGET_ath79=y
CONFIG_TARGET_ath79_generic=y
CONFIG_TARGET_MULTI_PROFILE=y
CONFIG_TARGET_DEVICE_ath79_generic_DEVICE_etactica_eg200=y
CONFIG_DEVEL=y
CONFIG_DOWNLOAD_FOLDER="/home/karlp/src/dl"
CONFIG_PACKAGE_libnetsnmp=y
CONFIG_PACKAGE_libpci=y
CONFIG_PACKAGE_snmpd=y
karlp commented 3 years ago

"Someone" didn't entirely trust my builds, so I retested with https://downloads.openwrt.org/releases/19.07.4/targets/ath79/generic/openwrt-19.07.4-ath79-generic-etactica_eg200-squashfs-sysupgrade.bin which matches my hardware, and then

opkg update
opkg install snmpd

and it still fails to respond after reboot, with the same symptoms. process is running, config file references an agentx socket that hasn't been created.

karlp commented 3 years ago

strace output of the when stuck. (lengthy!) it is happily "running" and reading system metrics, and it does get a line when I make a snmp request, but it doesn't answer. Annotates prefixed with >>>>

_newselect(8, [4 6 7], [], [], {tv_sec=1, tv_usec=971968}

>>>>>periodic sleep here,

) = 0 (Timeout)
_newselect(8, [4 6 7], [], [], {tv_sec=0, tv_usec=19057}) = 0 (Timeout)
open("/proc/stat", O_RDONLY|O_LARGEFILE) = 8
read(8, "cpu  4545 0 3713 76699 0 0 32 0 "..., 4095) = 317
close(8)                                = 0
open("/proc/vmstat", O_RDONLY|O_LARGEFILE) = 8
read(8, "nr_free_pages 8051\nnr_zone_inact"..., 4095) = 842
close(8)                                = 0
_newselect(8, [4 6 7], [], [], {tv_sec=0, tv_usec=110056}) = 0 (Timeout)
open("/proc/net/dev", O_RDONLY|O_LARGEFILE) = 8
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 9
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(10, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(10, SIOCGIFINDEX, {ifr_name="lo", }) = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x02\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=76, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3509681667}, "\x02\x08\x80\xfe\x00\x00\x00\x01\x00\x08\x00\x01\x7f\x00\x00\x01\x00\x08\x00\x02\x7f\x00\x00\x01\x00\x07\x00\x03\x6c\x6f\x00\x00"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3509681667}, "\x02\x18\x80\x00\x00\x00\x00\x04\x00\x08\x00\x01\xc0\xa8\x58\xc0\x00\x08\x00\x02\xc0\xa8\x58\xc0\x00\x08\x00\x04\xc0\xa8\x58\xff"...}], 16384, 0, NULL, NULL) = 164
close(11)                               = 0
ioctl(10, SIOCGIFNETMASK, {ifr_name="lo", ifr_netmask={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("255.0.0.0")}}) = 0
ioctl(10, SIOCGIFFLAGS, {ifr_name="lo", ifr_flags=IFF_UP|IFF_LOOPBACK|IFF_RUNNING}) = 0
ioctl(10, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x02\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=76, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3345413570}, "\x02\x08\x80\xfe\x00\x00\x00\x01\x00\x08\x00\x01\x7f\x00\x00\x01\x00\x08\x00\x02\x7f\x00\x00\x01\x00\x07\x00\x03\x6c\x6f\x00\x00"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3345413570}, "\x02\x18\x80\x00\x00\x00\x00\x04\x00\x08\x00\x01\xc0\xa8\x58\xc0\x00\x08\x00\x02\xc0\xa8\x58\xc0\x00\x08\x00\x04\xc0\xa8\x58\xff"...}], 16384, 0, NULL, NULL) = 164
close(11)                               = 0
ioctl(10, SIOCGIFNETMASK, {ifr_name="br-lan", ifr_netmask={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("255.255.255.0")}}) = 0
ioctl(10, SIOCGIFFLAGS, {ifr_name="br-lan", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
close(10)                               = 0
open("/proc/net/if_inet6", O_RDONLY|O_LARGEFILE) = 10
read(10, "fe80000000000000c69300fffe0366bd"..., 1024) = 108
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 11
ioctl(11, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2685241717}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2685241717}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3368625778}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3368625778}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 11
ioctl(11, SIOCGIFINDEX, {ifr_name="lo", }) = 0
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=4201870302}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=4201870302}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, 
{{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2620068482}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2620068482}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
read(10, "", 1024)                      = 0
close(10)                               = 0
read(8, "Inter-|   Receive               "..., 1024) = 692
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="wlan0", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="wlan0", }) = 0
close(10)                               = 0
open("/sys/class/net/wlan0/device/vendor", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(9, SIOCGIFHWADDR, {ifr_name="wlan0", ifr_hwaddr={sa_family=ARPHRD_ETHER, sa_data=c4:93:00:03:66:bb}}) = 0
ioctl(9, SIOCETHTOOL, 0x7fc8487c)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGMIIPHY, 0x7fc847f4)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGIFFLAGS, {ifr_name="wlan0", ifr_flags=IFF_BROADCAST|IFF_MULTICAST}) = 0
ioctl(9, SIOCGIFMTU, {ifr_name="wlan0", ifr_mtu=1500}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="eth0", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="eth0", }) = 0
close(10)                               = 0
open("/sys/class/net/eth0/device/vendor", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(9, SIOCGIFHWADDR, {ifr_name="eth0", ifr_hwaddr={sa_family=ARPHRD_ETHER, sa_data=c4:93:00:03:66:bd}}) = 0
ioctl(9, SIOCETHTOOL, 0x7fc8487c)       = 0
ioctl(9, SIOCGIFFLAGS, {ifr_name="eth0", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
ioctl(9, SIOCGIFMTU, {ifr_name="eth0", ifr_mtu=1500}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
close(10)                               = 0
open("/sys/class/net/br-lan/device/ve
ndor", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(9, SIOCGIFHWADDR, {ifr_name="br-lan", ifr_hwaddr={sa_family=ARPHRD_ETHER, sa_data=c4:93:00:03:66:bd}}) = 0
ioctl(9, SIOCETHTOOL, 0x7fc8487c)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGMIIPHY, 0x7fc847f4)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGIFFLAGS, {ifr_name="br-lan", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
ioctl(9, SIOCGIFMTU, {ifr_name="br-lan", ifr_mtu=1500}) = 0
open("/proc/sys/net/ipv4/neigh/br-lan/retrans_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "1000\n", 1024)                = 5
close(10)                               = 0
open("/proc/sys/net/ipv6/neigh/br-lan/retrans_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "1000\n", 1024)                = 5
close(10)                               = 0
open("/proc/sys/net/ipv6/conf/br-lan/forwarding", O_RDONLY|O_LARGEFILE) = 10
read(10, "1\n", 1024)                   = 2
close(10)                               = 0
open("/proc/sys/net/ipv6/neigh/br-lan/base_reachable_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "30000\n", 1024)               = 6
close(10)                               = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="lo", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="lo", }) = 0
close(10)                               = 0
open("/sys/class/net/lo/device/vendor", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(9, SIOCGIFHWADDR, {ifr_name="lo", ifr_hwaddr={sa_family=ARPHRD_LOOPBACK, sa_data=00:00:00:00:00:00}}) = 0
ioctl(9, SIOCGIFFLAGS, {ifr_name="lo", ifr_flags=IFF_UP|IFF_LOOPBACK|IFF_RUNNING}) = 0
ioctl(9, SIOCGIFMTU, {ifr_name="lo", ifr_mtu=65536}) = 0
open("/proc/sys/net/ipv4/neigh/lo/retrans_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "1000\n", 1024)                = 5
close(10)                               = 0
open("/proc/sys/net/ipv6/neigh/lo/retrans_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "1000\n", 1024)                = 5
close(10)                               = 0
open("/proc/sys/net/ipv6/conf/lo/forwarding", O_RDONLY|O_LARGEFILE) = 10
read(10, "1\n", 1024)                   = 2
close(10)                               = 0
open("/proc/sys/net/ipv6/neigh/lo/base_reachable_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "30000\n", 1024)               = 6
close(10)                               = 0
read(8, "", 1024)                       = 0
close(8)                                = 0
close(9)                                = 0
_newselect(8, [4 6 7], [], [], {tv_sec=2, tv_usec=105456}

>>>> next periodic sleep, I fire off an snmpwalk from my pc.... 

) = 1 (in [7], left {tv_sec=0, tv_usec=574124})

>>>> and we see it receive from my PC (My PC isw .88.210, snmpd is on 88.192)  (it still doesn't respond)

recvmsg(7, {msg_name={sa_family=AF_INET, sin_port=htons(44347), sin_addr=inet_addr("192.168.88.210")}, msg_namelen=28->16, msg_iov=[{iov_base="0&\2\1\1\4\6public\241\31\2\4:\364\324@\2\1\0\2\1\0000\v0\t\6"..., iov_len=65536}], msg_iovlen=1, msg_control=[{cmsg_len=24, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("br-lan"), ipi_spec_dst=inet_addr("192.168.88.192"), ipi_addr=inet_addr("192.168.88.192")}}], msg_controllen=24, msg_flags=0}, MSG_DONTWAIT) = 40
getsockname(7, {sa_family=AF_INET, sin_port=htons(161), sin_addr=inet_addr("0.0.0.0")}, [28->16]) = 0
_newselect(8, [4 6 7], [], [], {tv_sec=0, tv_usec=570610}) = 0 (Timeout)
open("/proc/net/dev", O_RDONLY|O_LARGEFILE) = 8
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 9
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(10, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(10, SIOCGIFINDEX, {ifr_name="lo", }) = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x02\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=76, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2533117968}, "\x02\x08\x80\xfe\x00\x00\x00\x01\x00\x08\x00\x01\x7f\x00\x00\x01\x00\x08\x00\x02\x7f\x00\x00\x01\x00\x07\x00\x03\x6c\x6f\x00\x00"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2533117968}, "\x02\x18\x80\x00\x00\x00\x00\x04\x00\x08\x00\x01\xc0\xa8\x58\xc0\x00\x08\x00\x02\xc0\xa8\x58\xc0\x00\x08\x00\x04\xc0\xa8\x58\xff"...}], 16384, 0, NULL, NULL) = 164
close(11)                               = 0
ioctl(10, SIOCGIFNETMASK, {ifr_name="lo", ifr_netmask={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("255.0.0.0")}}) = 0
ioctl(10, SIOCGIFFLAGS, {ifr_name="lo", ifr_flags=IFF_UP|IFF_LOOPBACK|IFF_RUNNING}) = 0
ioctl(10, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x02\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=76, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=4119379592}, "\x02\x08\x80\xfe\x00\x00\x00\x01\x00\x08\x00\x01\x7f\x00\x00\x01\x00\x08\x00\x02\x7f\x00\x00\x01\x00\x07\x00\x03\x6c\x6f\x00\x00"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=4119379592}, "\x02\x18\x80\x00\x00\x00\x00\x04\x00\x08\x00\x01\xc0\xa8\x58\xc0\x00\x08\x00\x02\xc0\xa8\x58\xc0\x00\x08\x00\x04\xc0\xa8\x58\xff"...}], 16384, 0, NULL, NULL) = 164
close(11)                               = 0
ioctl(10, SIOCGIFNETMASK, {ifr_name="br-lan", ifr_netmask={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("255.255.255.0")}}) = 0
ioctl(10, SIOCGIFFLAGS, {ifr_name="br-lan", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
close(10)                               = 0
open("/proc/net/if_inet6", O_RDONLY|O_LARGEFILE) = 10
read(10, "fe80000000000000c69300fffe0366bd"..., 1024) = 108
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 11
ioctl(11, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3296207586}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3296207586}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2593428483}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2593428483}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 11
ioctl(11, SIOCGIFINDEX, {ifr_name="lo", }) = 0
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2279467639}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2279467639}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2790995786}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2790995786}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
read(10, "", 1024)                      = 0
close(10)                               = 0
read(8, "Inter-|   Receive               "..., 1024) = 692
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="wlan0", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="wlan0", }) = 0
close(10)                               = 0
open("/sys/class/net/wlan0/device/vendor", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(9, SIOCGIFHWADDR, {ifr_name="wlan0", ifr_hwaddr={sa_family=ARPHRD_ETHER, sa_data=c4:93:00:03:66:bb}}) = 0
ioctl(9, SIOCETHTOOL, 0x7fc8487c)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGMIIPHY, 0x7fc847f4)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGIFFLAGS, {ifr_name="wlan0", ifr_flags=IFF_BROADCAST|IFF_MULTICAST}) = 0
ioctl(9, SIOCGIFMTU, {ifr_name="wlan0", ifr_mtu=1500}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="eth0", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="eth0", }) = 0
close(10)                               = 0
open("/sys/class/net/eth0/device/vendor", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(9, SIOCGIFHWADDR, {ifr_name="eth0", ifr_hwaddr={sa_family=ARPHRD_ETHER, sa_data=c4:93:00:03:66:bd}}) = 0
ioctl(9, SIOCETHTOOL, 0x7fc8487c)       = 0
ioctl(9, SIOCGIFFLAGS, {ifr_name="eth0", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
ioctl(9, SIOCGIFMTU, {ifr_name="eth0", ifr_mtu=1500}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
close(10)                               = 0
open("/sys/class/net/br-lan/device/vendor", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(9, SIOCGIFHWADDR, {ifr_name="br-lan", ifr_hwaddr={sa_family=ARPHRD_ETHER, sa_data=c4:93:00:03:66:bd}}) = 0
ioctl(9, SIOCETHTOOL, 0x7fc8487c)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGMIIPHY, 0x7fc847f4)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGIFFLAGS, {ifr_name="br-lan", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
ioctl(9, SIOCGIFMTU, {ifr_name="br-lan", ifr_mtu=1500}) = 0
open("/proc/sys/net/ipv4/neigh/br-lan/retrans_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "1000\n", 1024)                = 5
close(10)                               = 0
open("/proc/sys/net/ipv6/neigh/br-lan/retrans_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "1000\n", 1024)                = 5
close(10)                               = 0
open("/proc/sys/net/ipv6/conf/br-lan/forwarding", O_RDONLY|O_LARGEFILE) = 10
read(10, "1\n", 1024)                   = 2
close(10)                               = 0
open("/proc/sys/net/ipv6/neigh/br-lan/base_reachable_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "30000\n", 1024)               = 6
close(10)                               = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="lo", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="lo", }) = 0
close(10)                               = 0
open("/sys/class/net/lo/device/vendor", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(9, SIOCGIFHWADDR, {ifr_name="lo", ifr_hwaddr={sa_family=ARPHRD_LOOPBACK, sa_data=00:00:00:00:00:00}}) = 0
ioctl(9, SIOCGIFFLAGS, {ifr_name="lo", ifr_flags=IFF_UP|IFF_LOOPBACK|IFF_RUNNING}) = 0
ioctl(9, SIOCGIFMTU, {ifr_name="lo", ifr_mtu=65536}) = 0
open("/proc/sys/net/ipv4/neigh/lo/retrans_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "1000\n", 1024)                = 5
close(10)                               = 0
open("/proc/sys/net/ipv6/neigh/lo/retrans_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "1000\n", 1024)                = 5
close(10)                               = 0
open("/proc/sys/net/ipv6/conf/lo/forwarding", O_RDONLY|O_LARGEFILE) = 10
read(10, "1\n", 1024)                   = 2
close(10)                               = 0
open("/proc/sys/net/ipv6/neigh/lo/base_reachable_time_ms", O_RDONLY|O_LARGEFILE) = 10
read(10, "30000\n", 1024)               = 6
close(10)                               = 0
read(8, "", 1024)                       = 0
close(8)                                = 0
close(9)                                = 0
_newselect(8, [4 6 7], [], [], {tv_sec=0, tv_usec=990864}) = 1 (in [7], left {tv_sec=0, tv_usec=990825})
recvmsg(7, {msg_name={sa_family=AF_INET, sin_port=htons(44347), sin_addr=inet_addr("192.168.88.210")}, msg_namelen=28->16, msg_iov=[{iov_base="0&\2\1\1\4\6public\241\31\2\4:\364\324@\2\1\0\2\1\0000\v0\t\6"..., iov_len=65536}], msg_iovlen=1, msg_control=[{cmsg_len=24, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("br-lan"), ipi_spec_dst=inet_addr("192.168.88.192"), ipi_addr=inet_addr("192.168.88.192")}}], msg_controllen=24, msg_flags=0}, MSG_DONTWAIT) = 40
getsockname(7, {sa_family=AF_INET, sin_port=htons(161), sin_addr=inet_addr("0.0.0.0")}, [28->16]) = 0
_newselect(8, [4 6 7], [], [], {tv_sec=0, tv_usec=987548}) = 1 (in [7], left {tv_sec=0, tv_usec=459397})
recvmsg(7, {msg_name={sa_family=AF_INET, sin_port=htons(44347), sin_addr=inet_addr("192.168.88.210")}, msg_namelen=28->16, msg_iov=[{iov_base="0&\2\1\1\4\6public\241\31\2\4:\364\324@\2\1\0\2\1\0000\v0\t\6"..., iov_len=65536}], msg_iovlen=1, msg_control=[{cmsg_len=24, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("br-lan"), ipi_spec_dst=inet_addr("192.168.88.192"), ipi_addr=inet_addr("192.168.88.192")}}], msg_controllen=24, msg_flags=0}, MSG_DONTWAIT) = 40
getsockname(7, {sa_family=AF_INET, sin_port=htons(161), sin_addr=inet_addr("0.0.0.0")}, [28->16]) = 0
_newselect(8, [4 6 7], [], [], {tv_sec=0, tv_usec=456097}) = 0 (Timeout)
open("/proc/stat", O_RDONLY|O_LARGEFILE) = 8
read(8, "cpu  4575 0 3795 77086 0 0 33 0 "..., 4095) = 317
close(8)                                = 0
open("/proc/vmstat", O_RDONLY|O_LARGEFILE) = 8
read(8, "nr_free_pages 8059\nnr_zone_inact"..., 4095) = 842
close(8)                                = 0
_newselect(8, [4 6 7], [], [], {tv_sec=1, tv_usec=111027}) = 1 (in [7], left {tv_sec=0, tv_usec=573284})
recvmsg(7, {msg_name={sa_family=AF_INET, sin_port=htons(44347), sin_addr=inet_addr("192.168.88.210")}, msg_namelen=28->16, msg_iov=[{iov_base="0&\2\1\1\4\6public\241\31\2\4:\364\324@\2\1\0\2\1\0000\v0\t\6"..., iov_len=65536}], msg_iovlen=1, msg_control=[{cmsg_len=24, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("br-lan"), ipi_spec_dst=inet_addr("192.168.88.192"), ipi_addr=inet_addr("192.168.88.192")}}], msg_controllen=24, msg_flags=0}, MSG_DONTWAIT) = 40
getsockname(7, {sa_family=AF_INET, sin_port=htons(161), sin_addr=inet_addr("0.0.0.0")}, [28->16]) = 0
_newselect(8, [4 6 7], [], [], {tv_sec=0, tv_usec=569979}) = 0 (Timeout)
open("/proc/net/dev", O_RDONLY|O_LARGEFILE) = 8
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 9
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(10, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(10, SIOCGIFINDEX, {ifr_name="lo", }) = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x02\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=76, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=4121819864}, "\x02\x08\x80\xfe\x00\x00\x00\x01\x00\x08\x00\x01\x7f\x00\x00\x01\x00\x08\x00\x02\x7f\x00\x00\x01\x00\x07\x00\x03\x6c\x6f\x00\x00"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=4121819864}, "\x02\x18\x80\x00\x00\x00\x00\x04\x00\x08\x00\x01\xc0\xa8\x58\xc0\x00\x08\x00\x02\xc0\xa8\x58\xc0\x00\x08\x00\x04\xc0\xa8\x58\xff"...}], 16384, 0, NULL, NULL) = 164
close(11)                               = 0
ioctl(10, SIOCGIFNETMASK, {ifr_name="lo", ifr_netmask={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("255.0.0.0")}}) = 0
ioctl(10, SIOCGIFFLAGS, {ifr_name="lo", ifr_flags=IFF_UP|IFF_LOOPBACK|IFF_RUNNING}) = 0
ioctl(10, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x02\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=76, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3813894834}, "\x02\x08\x80\xfe\x00\x00\x00\x01\x00\x08\x00\x01\x7f\x00\x00\x01\x00\x08\x00\x02\x7f\x00\x00\x01\x00\x07\x00\x03\x6c\x6f\x00\x00"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3813894834}, "\x02\x18\x80\x00\x00\x00\x00\x04\x00\x08\x00\x01\xc0\xa8\x58\xc0\x00\x08\x00\x02\xc0\xa8\x58\xc0\x00\x08\x00\x04\xc0\xa8\x58\xff"...}], 16384, 0, NULL, NULL) = 164
close(11)                               = 0
ioctl(10, SIOCGIFNETMASK, {ifr_name="br-lan", ifr_netmask={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("255.255.255.0")}}) = 0
ioctl(10, SIOCGIFFLAGS, {ifr_name="br-lan", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
close(10)                               = 0
open("/proc/net/if_inet6", O_RDONLY|O_LARGEFILE) = 10
read(10, "fe80000000000000c69300fffe0366bd"..., 1024) = 108
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 11
ioctl(11, SIOCGIFINDEX, {ifr_name="br-lan", }) = 0
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3471933647}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3471933647}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2222043607}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=2222043607}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 11
ioctl(11, SIOCGIFINDEX, {ifr_name="lo", }) = 0
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3584839336}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3584839336}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 11
sendto(11, {{len=24, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x100, seq=0, pid=0}, "\x0a\x00\x00\x00\x00\x00\x00\x00"}, 24, 0, NULL, 0) = 24
recvfrom(11, [{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3648794628}, "\x0a\x80\x80\xfe\x00\x00\x00\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x14\x00\x06"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=0, pid=3648794628}, "\x0a\x40\x80\xfd\x00\x00\x00\x04\x00\x14\x00\x01\xfe\x80\x00\x00\x00\x00\x00\x00\xc6\x93\x00\xff\xfe\x03\x66\xbd\x00\x14\x00\x06"...}], 16384, 0, NULL, NULL) = 144
close(11)                               = 0
read(10, "", 1024)                      = 0
close(10)                               = 0
read(8, "Inter-|   Receive               "..., 1024) = 692
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="wlan0", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="wlan0", }) = 0
close(10)                               = 0
open("/sys/class/net/wlan0/device/vendor", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(9, SIOCGIFHWADDR, {ifr_name="wlan0", ifr_hwaddr={sa_family=ARPHRD_ETHER, sa_data=c4:93:00:03:66:bb}}) = 0
ioctl(9, SIOCETHTOOL, 0x7fc8487c)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGMIIPHY, 0x7fc847f4)       = -1 EOPNOTSUPP (Not supported)
ioctl(9, SIOCGIFFLAGS, {ifr_name="wlan0", ifr_flags=IFF_BROADCAST|IFF_MULTICAST}) = 0
ioctl(9, SIOCGIFMTU, {ifr_name="wlan0", ifr_mtu=1500}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="eth0", }) = 0
close(10)                               = 0
ioctl(9, SIOCGIFCONF, {ifc_len=8 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
ioctl(9, SIOCGIFCONF, {ifc_len=16 * sizeof(struct ifreq) => 2 * sizeof(struct ifreq), ifc_buf=[{ifr_name="lo", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}}, {ifr_name="br-lan", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.88.192")}}]}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 10
ioctl(10, SIOCGIFINDEX, {ifr_name="eth0", }) = 0
close(10)                               = 0
open("/sys/class/net/eth0/devic^C|O_LARGEFILEstrace: Process 1427 detached
 <detached ...>
karlp commented 3 years ago

And tcpdump requested, matches expectations, udp packets arrive, udp packets never leave. this is when it's non-responsive

root@OpenWrt:/# tcpdump port 161 and udp
[ 1347.387495] device br-lan entered promiscuous mode
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
10:50:07.667406 IP 192.168.88.210.57295 > 192.168.88.192.161:  GetNextRequest(25)  
10:50:08.668428 IP 192.168.88.210.57295 > 192.168.88.192.161:  GetNextRequest(25)  
10:50:09.669425 IP 192.168.88.210.57295 > 192.168.88.192.161:  GetNextRequest(25)  
10:50:10.670415 IP 192.168.88.210.57295 > 192.168.88.192.161:  GetNextRequest(25)  
10:50:11.671391 IP 192.168.88.210.57295 > 192.168.88.192.161:  GetNextRequest(25)  
10:50:12.672417 IP 192.168.88.210.57295 > 192.168.88.192.161:  GetNextRequest(25)
arrmo commented 3 years ago

FYI, having the same issue - fix is pretty easy, move snmpd down in the boot order. And in reality, it's not critical, agreed?

Similar thread here, https://forum.openwrt.org/t/snmpd-not-starting-on-boot/87882/23

Submit a PR, to change from S50 to S90? This seems to work for me, avoid the race condition (as it seems to be).

Thanks!

stintel commented 3 years ago

Looks like I might be hitting this on master now, on realtek (Netgear GS108Tv3). Can't reliably reproduce it though.

arrmo commented 3 years ago

Yes, it seems to happen to me ... most of the time. Timing is on the edge, so sometimes OK, sometimes not. At S90 - very reliably starts up.

Thanks!

stintel commented 3 years ago

So after writing that I was unable to reproduce it again. I think the previous two times were after sysupgrade. I vaguely recall running into it on some of my APs a while ago. This leads me to believe it is easier to trigger on slow, single-core devices.

Ideally we report this upstream, but since it's difficult to reproduce ... I'll do some more testing locally, if I don't find anything in reasonable time will go with START=90 - thanks for the suggestion!

stintel commented 3 years ago

Would you mind trying this:

diff --git a/net/net-snmp/files/snmpd.init b/net/net-snmp/files/snmpd.init
index fbae9c286..4d6c5533e 100644
--- a/net/net-snmp/files/snmpd.init
+++ b/net/net-snmp/files/snmpd.init
@@ -347,7 +347,7 @@ service_triggers(){
        local name=$(basename ${script:-$initscript})

        procd_open_trigger
-       procd_add_raw_trigger "interface.*" 2000 /etc/init.d/$name reload
+       procd_add_raw_trigger "interface.*" 5000 /etc/init.d/$name reload
        procd_close_trigger

        procd_add_reload_trigger 'snmpd'
arrmo commented 3 years ago

Sure, NP! Actually, will just edit /etc/init.d/snmpd on the router, reboot a few times. Will let you know.

arrmo commented 3 years ago

Seems to work! Not sure I quite understand it ... LOL. At least for me, 5 reboots, no issues.

Thanks!

arrmo commented 3 years ago

If this is working (for other as well), get it in the OpenWrt code? Thanks!

luizluca commented 3 years ago

I also have this issue. I have dozens of equal conf but different hw routers (79) with this issue that happens randomly both on boot and sometimes when wan is restarted. My routers use VPN that might slow down the boot process. I just tested the patch:

-       procd_add_raw_trigger "interface.*" 2000 /etc/init.d/$name reload
+       procd_add_raw_trigger "interface.*" 5000 /etc/init.d/$name reload

It didn't work. After a reboot, 31% (25/79) didn't had snmpd working. I also tested with 9000 instead of 5000. “If brute force fails, you're not using enough.” None failed this time.

-       procd_add_raw_trigger "interface.*" 2000 /etc/init.d/$name reload
+       procd_add_raw_trigger "interface.*" 9000 /etc/init.d/$name reload

@stintel , just increasing the timeout might not solve the issue but just make it less likely. Anyway, 5000 is too small, at least for me. Do you have any idea of what is happening? I noticed that this started to happen with me after I added a second VPN connection (now I have three). Would it be related to the issue this patch is trying to fix ?

As a workaround, I've been running this oneliner script every hour for months:

pgrep snmpd >/dev/null && ! [ -e /var/run/agentx.sock ] && /etc/init.d/snmpd restart && echo snmp restarted || :

It is dirty but do the job

stintel commented 2 years ago

I am hitting this again, again on GS108Tv3. I just spotted something that might lead us closer to finding the root cause.

root@sw3:~# netstat -anput | grep snmpd
udp        0      0 0.0.0.0:161             0.0.0.0:*                           2745/snmpd

vs

root@sw3:~# netstat -anput | grep snmpd
udp        0      0 0.0.0.0:161             0.0.0.0:*                           3560/snmpd
udp        0      0 :::161                  :::*                                3560/snmpd

When snmpd is not listening on IPv6, it does not respond. When it does, it does.

I cannot reliably reproduce it, but so far, when I'm able to reproduce it, snmpd is not listening on IPv6.

hnyman commented 1 year ago

@stintel Is this related to just the EoL 19.07? or also newer versions?

stintel commented 1 year ago

This happened on master before 22.03 also, iirc.