Open deajan opened 4 months ago
I realized that my above diagnostic is a bit botched.
So I added a rule forcing ICMP to use the offline marked gateway for one of my machines behind OPNSense.
From that machine, pinging 1.1.1.1
with that rule still works, so I can confirm that something isn't right with the way dpinger thinks my gateway is offline, when monitor IP is 1.1.1.1
.
I've also tried the following from the OPNsense console
Ping works
ping -S 172.25.XX.XX 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 172.252.236.42: 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=53 time=21.643 ms
Dpinger fails from my second WAN which is marked offline
dpinger -f -B 172.25.X.X -s 1s -l 4s -t 60s -d 1 1.1.1.1
send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 1000ms data_len 0 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 1.1.1.1 bind_addr 172.25.XX.XX identifier ""
0 0 0
0 0 100
But dpinger works from my primary WAN
dpinger -f -B 194.87.XX.XX -s 1s -l 4s -t 60s -d 1.1.1.1
send_interval 1000ms loss_interval 4000ms time_period 60000ms report_interval 1000ms data_len 1 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 194.87.XX.XX identifier ""
26746 0 0
26672 179 0
26722 92 0
Investigating further, it seems that when dpinger's payload is set to 1, it fails. If I set the payload to 4 or higher, it works. I could confirm this by using ping:
The following works:
ping -S 172.25.XX.XX -s 4 1.1.1.1
The following doesn't;
ping -S 172.25.XX.XX -s 1 1.1.1.1
Playing with the packet size, I realized that every packet size between 4 and 172 bytes works, and others don't. This only happens on the secondary link, the primary link accepts usual packet sizes.
So end of diagnostic: It seems that there's something fishy with the link itself, and not OPNSense. Sorry for the noise.
I have yet to find what the problem is at my datacenter. If anyone has a clue, I will gladly take it ;)
Please let me know if you found anything - I'm having the exact same problem but only since 24.7 came along.
@dwkirw I've pushed the diagnostics. Whenever I use values for ping -s n
where 4 <= n <= 172, I get to ping.
My datacenter guys told me that they have no filter whatsoever, but on the second link they provide I don't have that problem, so to be honest I doubt that this is an OPNSense problem.
Btw, I'm running OPNsense 24.1.10_3-amd64 on this unit, so I had this problem prior to 24.7.
What exactly is your problem ? Did you try adding a payload to the gateway monitor ? Does it work ?
Thank you - I've had a better read of what you wrote and tried some of the stuff. Also, disregard the only on 24.7 thing - I've only had a backup WAN about 6 weeks and likely havent noticed this
I have not played around with dpinger before.
From the primary wan (which is currently marked as down) i get stuff such as this.
dpinger -f -B 139.5.x.x -s 1s -l 4s -t 60s -d 1 1.1.1.1
7650 0 0
7288 371 0
7370 316 0
7289 324 0
From backup (currently showing Online) i get this dpinger -f -B 192.168.1.2 -s 1s -l 4s -t 60s -d 1 1.1.1.1 36686 0 0 33661 3030 0 35521 3611 0 34479 3608 0
My primary is PPPoE, its IP changes fairly often. Heres its settings. The IP has been left blank for this reason. . While the primary wan is appearing to be down but still working if i go to System>Configuration>Gateways and hit the Apply button it goes to Online again. While its marked Offline my port forwards are flakey though sometimes working/not...
If you can see any dumb things I've missed or should try I'd appreciate knowing about them, ta!
It looks like the patch from issue #7027 is not included yet in 24.7.1? @fichtner ? I'm having a very similar issue, both with 24.1, as well as 24.7. Can reproduce like clockwork if needed.
It's happening with 1.1.1.1 as monitor host as well as the PPPOE connection's gateway. The IP given to dpinger -S
is the correct one.
It's always triggered by a PPPOE reconnect, which sends dpinger into this state:
2024-09-26T06:06:17 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
...
2024-09-26T04:15:18 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:17 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:16 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:16 Notice dpinger ALERT: WAN_PPPOE (Addr: 1.1.1.1 Alarm: none -> down RTT: 0.0 ms RTTd: 0.0 ms Loss: 100.0 %)
2024-09-26T04:15:15 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:14 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:13 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:12 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:11 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:11 Warning dpinger send_interval 1000ms loss_interval 4000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 1.1.1.1 bind_addr 95.117.229.255 identifier "WAN_PPPOE "
2024-09-26T04:15:05 Notice dpinger Reloaded gateway watcher configuration on SIGHUP
2024-09-26T04:15:05 Warning dpinger exiting on signal 15
2024-09-26T04:15:05 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:04 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:03 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
2024-09-26T04:15:02 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
It's immediately fixed by reloading the monitor plugin config.
Here's some debug output:
root@OPNsense:~ # pluginctl -r return_gateways_status
{
"dpinger": {
"WAN_DHCP6": {
"status": "none",
"monitor": "~",
"name": "WAN_DHCP6",
"stddev": "~",
"delay": "~",
"loss": "~"
},
"WAN_PPPOE": {
"status": "down",
"monitor": "1.1.1.1",
"name": "WAN_PPPOE",
"stddev": "0.0 ms",
"delay": "0.0 ms",
"loss": "100.0 %"
},
"WAN_LTE_DHCP": {
"status": "none",
"monitor": "192.168.22.1",
"name": "WAN_LTE_DHCP",
"stddev": "5.8 ms",
"delay": "1.4 ms",
"loss": "0.0 %"
}
}
}
root@OPNsense:~ # ps auxf | grep dping
root 84723 0.0 0.1 17572 2780 - Is 04:15 0:04.21 /usr/local/bin/dpinger -f -S -r 0 -i WAN_PPPOE -B 95.117.229.XXX -p /var/run/dpinger_WAN_PPPOE.pid -u /var/run/dpinger_WAN_PPPOE.sock -s 1s -l 4s -t 60s -d 1 1.1.1.1
root 90835 0.0 0.1 17572 2792 - Is 19:57 0:14.78 /usr/local/bin/dpinger -f -S -r 0 -i WAN_LTE_DHCP -B 192.168.22.101 -p /var/run/dpinger_WAN_LTE_DHCP.pid -u /var/run/dpinger_WAN_LTE_DHCP.sock -s 1s -l 4s -t 60s -d 1 192
root@OPNsense:~ # pluginctl -c monitor
Setting up gateway monitors...done.
root@OPNsense:~ # ps auxf | grep dping
root 67987 10.6 0.1 13344 2716 - Ss 06:59 0:00.01 /usr/local/bin/dpinger -f -S -r 0 -i WAN_LTE_DHCP -B 192.168.22.101 -p /var/run/dpinger_WAN_LTE_DHCP.pid -u /var/run/dpinger_WAN_LTE_DHCP.sock -s 1s -l 4s -t 60s -d 1 192
root 67240 10.4 0.1 13344 2716 - Ss 06:59 0:00.01 /usr/local/bin/dpinger -f -S -r 0 -i WAN_PPPOE -B 95.117.229.XXX -p /var/run/dpinger_WAN_PPPOE.pid -u /var/run/dpinger_WAN_PPPOE.sock -s 1s -l 4s -t 60s -d 1 1.1.1.1
root@OPNsense:~ # pluginctl -r return_gateways_status
{
"dpinger": {
"WAN_DHCP6": {
"status": "none",
"monitor": "~",
"name": "WAN_DHCP6",
"stddev": "~",
"delay": "~",
"loss": "~"
},
"WAN_PPPOE": {
"status": "none",
"monitor": "1.1.1.1",
"name": "WAN_PPPOE",
"stddev": "119.4 ms",
"delay": "49.2 ms",
"loss": "0.0 %"
},
"WAN_LTE_DHCP": {
"status": "none",
"monitor": "192.168.22.1",
"name": "WAN_LTE_DHCP",
"stddev": "125.7 ms",
"delay": "42.6 ms",
"loss": "0.0 %"
}
}
}
And the general log:
2024-09-26T04:19:32 Error dhclient send_packet: Host is down
2024-09-26T04:18:23 Error dhclient send_packet: Host is down
2024-09-26T04:17:31 Error dhclient send_packet: Host is down
2024-09-26T04:16:09 Error opnsense /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 10158'
2024-09-26T04:16:09 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,opt2))
2024-09-26T04:16:09 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,opt2))
2024-09-26T04:16:09 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())
2024-09-26T04:16:09 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,opt2))
2024-09-26T04:16:09 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,opt2))
2024-09-26T04:16:09 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())
2024-09-26T04:16:09 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())
2024-09-26T04:16:09 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,opt2))
2024-09-26T04:16:08 Warning igmpproxy select() failure; Errno(4): Interrupted system call
2024-09-26T04:16:08 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())
2024-09-26T04:16:08 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())
2024-09-26T04:16:08 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,opt2)
2024-09-26T04:16:08 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,opt2))
2024-09-26T04:16:08 Notice opnsense /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface LAN_WLAN.
2024-09-26T04:16:08 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,opt2))
2024-09-26T04:16:08 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,opt2))
2024-09-26T04:16:08 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,opt2)
2024-09-26T04:16:08 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,lan))
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,lan))
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,lan))
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,lan))
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,lan))
2024-09-26T04:16:06 Warning igmpproxy select() failure; Errno(4): Interrupted system call
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())
2024-09-26T04:16:06 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,lan)
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,lan))
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface LAN_SERVER.
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,lan))
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,lan))
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,lan)
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,wan))
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,wan))
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,wan))
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,wan))
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())
2024-09-26T04:16:05 Error opnsense /usr/local/etc/rc.newwanipv6: Primary WAN pppoe0 change detected, killing states of failover WAN igb2_vlan22 / 192.168.22.101
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,wan))
2024-09-26T04:16:05 Warning igmpproxy select() failure; Errno(4): Interrupted system call
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())
2024-09-26T04:16:05 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,wan)
2024-09-26T04:16:04 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,wan))
2024-09-26T04:15:59 Warning opnsense /usr/local/etc/rc.newwanip: Interface '' (ovpns1) is disabled or empty, nothing to do.
2024-09-26T04:15:58 Notice opnsense /usr/local/etc/rc.newwanipv6: OpenVPN client 2 instance started on PID 37591.
2024-09-26T04:15:58 Notice kernel <6>ovpns1: link state changed to UP
2024-09-26T04:15:58 Notice opnsense /usr/local/etc/rc.newwanipv6: OpenVPN server 1 instance started on PID 33027.
2024-09-26T04:15:58 Notice kernel <6>ovpns1: link state changed to DOWN
2024-09-26T04:15:58 Notice opnsense /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.
2024-09-26T04:15:58 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,wan))
2024-09-26T04:15:58 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure route_reload (execute task : core_routing_batch(,[]))
2024-09-26T04:15:58 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure route_reload (,[])
2024-09-26T04:15:55 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,wan))
2024-09-26T04:15:55 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,wan)
2024-09-26T04:15:51 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0
2024-09-26T04:15:51 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
2024-09-26T04:15:51 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4
2024-09-26T04:15:51 Error dhclient send_packet: Host is down
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'opt2'
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'lan'
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
2024-09-26T04:15:50 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4
2024-09-26T04:15:49 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'
2024-09-26T04:15:49 Warning radvd exiting, 1 sigterm(s) received
2024-09-26T04:15:48 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
2024-09-26T04:15:48 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
2024-09-26T04:15:48 Notice opnsense /usr/local/etc/rc.newwanipv6: IP renewal starting (address: fe80::20d:b9ff:fe48:9070%pppoe0, interface: wan, device: pppoe0)
2024-09-26T04:15:45 Error opnsense /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 32728'
2024-09-26T04:15:44 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,opt2))
2024-09-26T04:15:44 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,opt2))
2024-09-26T04:15:44 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())
2024-09-26T04:15:44 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,opt2))
2024-09-26T04:15:44 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,opt2))
2024-09-26T04:15:44 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())
2024-09-26T04:15:44 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())
2024-09-26T04:15:44 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,opt2))
2024-09-26T04:15:43 Warning igmpproxy select() failure; Errno(4): Interrupted system call
2024-09-26T04:15:43 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())
2024-09-26T04:15:43 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())
2024-09-26T04:15:43 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,opt2)
2024-09-26T04:15:42 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,opt2))
2024-09-26T04:15:42 Notice opnsense /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface LAN_WLAN.
2024-09-26T04:15:42 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,opt2))
2024-09-26T04:15:42 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,opt2))
2024-09-26T04:15:42 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,opt2)
2024-09-26T04:15:42 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,lan))
2024-09-26T04:15:41 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,lan))
2024-09-26T04:15:41 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())
2024-09-26T04:15:41 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,lan))
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,lan))
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,lan))
2024-09-26T04:15:40 Warning igmpproxy select() failure; Errno(4): Interrupted system call
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,lan)
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,lan))
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface LAN_SERVER.
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,lan))
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,lan))
2024-09-26T04:15:40 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,lan)
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,wan))
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,wan))
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4
2024-09-26T04:15:39 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: not a valid opt4 interface gateway address: 'missing'
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: treating '62.52.192.83' as far gateway for '95.117.229.XXX/32'
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,wan))
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,wan))
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())
2024-09-26T04:15:39 Error opnsense /usr/local/etc/rc.newwanipv6: Primary WAN pppoe0 change detected, killing states of failover WAN igb2_vlan22 / 192.168.22.101
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,wan))
2024-09-26T04:15:39 Warning igmpproxy select() failure; Errno(4): Interrupted system call
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,wan)
2024-09-26T04:15:39 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
2024-09-26T04:15:38 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,wan))
2024-09-26T04:15:34 Error opnsense /usr/local/etc/rc.newwanip: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 57282'
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : wireguard_sync(,wan))
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,wan))
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,wan))
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,wan))
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do())
2024-09-26T04:15:34 Error opnsense /usr/local/etc/rc.newwanip: Primary WAN pppoe0 change detected, killing states of failover WAN igb2_vlan22 / 192.168.22.101
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : reset_failover_states(,wan))
2024-09-26T04:15:34 Warning igmpproxy select() failure; Errno(4): Interrupted system call
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : igmpproxy_configure_do())
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())
2024-09-26T04:15:34 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure newwanip (,wan)
2024-09-26T04:15:33 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : wireguard_configure_do(,wan))
2024-09-26T04:15:29 Warning opnsense /usr/local/etc/rc.newwanip: Interface '' (ovpns1) is disabled or empty, nothing to do.
2024-09-26T04:15:29 Notice opnsense /usr/local/etc/rc.newwanipv6: OpenVPN client 2 instance started on PID 40848.
2024-09-26T04:15:28 Notice kernel <6>ovpns1: link state changed to UP
2024-09-26T04:15:27 Notice opnsense /usr/local/etc/rc.newwanipv6: OpenVPN server 1 instance started on PID 28432.
2024-09-26T04:15:27 Notice kernel <6>ovpns1: link state changed to DOWN
2024-09-26T04:15:27 Notice opnsense /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.
2024-09-26T04:15:27 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,wan))
2024-09-26T04:15:27 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure route_reload (execute task : core_routing_batch(,[]))
2024-09-26T04:15:27 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure route_reload (,[])
2024-09-26T04:15:24 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,wan))
2024-09-26T04:15:24 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,wan)
2024-09-26T04:15:22 Warning opnsense /usr/local/etc/rc.newwanip: Interface '' (ovpns1) is disabled or empty, nothing to do.
2024-09-26T04:15:22 Notice opnsense /usr/local/etc/rc.newwanip: OpenVPN client 2 instance started on PID 94442.
2024-09-26T04:15:21 Notice kernel <6>ovpnc2: link state changed to DOWN
2024-09-26T04:15:21 Notice kernel <6>ovpns1: link state changed to UP
2024-09-26T04:15:21 Notice opnsense /usr/local/etc/rc.newwanip: OpenVPN server 1 instance started on PID 84548.
2024-09-26T04:15:21 Notice kernel <7>cannot forward src fe80:2::81bc:8d60:3c1b:248b, dst 2a02:3100:a106:ea23:20d:b9ff:YYYY:XXXX, nxt 58, rcvif igb1, outif pppoe0
2024-09-26T04:15:21 Notice kernel <6>ovpns1: link state changed to DOWN
2024-09-26T04:15:20 Notice opnsense /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
2024-09-26T04:15:20 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,wan))
2024-09-26T04:15:20 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure route_reload (execute task : core_routing_batch(,[]))
2024-09-26T04:15:20 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure route_reload (,[])
2024-09-26T04:15:20 Notice kernel <7>cannot forward src fe80:2::81bc:8d60:3c1b:248b, dst 2a02:3100:a106:ea23:20d:b9ff:YYYY:XXXX, nxt 58, rcvif igb1, outif pppoe0
2024-09-26T04:15:19 Notice kernel <7>cannot forward src fe80:2::81bc:8d60:3c1b:248b, dst 2a02:3100:a106:ea23:20d:b9ff:YYYY:XXXX, nxt 58, rcvif igb1, outif pppoe0
2024-09-26T04:15:17 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0
2024-09-26T04:15:17 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan
2024-09-26T04:15:17 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4
2024-09-26T04:15:17 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: not a valid opt4 interface gateway address: 'missing'
2024-09-26T04:15:17 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: treating '62.52.192.83' as far gateway for '95.117.229.XXX/32'
2024-09-26T04:15:17 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,wan))
2024-09-26T04:15:17 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure vpn (,wan)
2024-09-26T04:15:17 Notice opnsense /usr/local/etc/rc.newwanip: IP address change detected, killing states of old ip 77.7.62.207
2024-09-26T04:15:16 Notice opnsense /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
2024-09-26T04:15:16 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0
2024-09-26T04:15:16 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
2024-09-26T04:15:16 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP
2024-09-26T04:15:15 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'opt2'
2024-09-26T04:15:15 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0
2024-09-26T04:15:15 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
2024-09-26T04:15:15 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP
2024-09-26T04:15:14 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'lan'
2024-09-26T04:15:14 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))
2024-09-26T04:15:14 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)
2024-09-26T04:15:14 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0
2024-09-26T04:15:14 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
2024-09-26T04:15:14 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP
2024-09-26T04:15:14 Notice opnsense /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'
2024-09-26T04:15:14 Warning radvd exiting, 1 sigterm(s) received
2024-09-26T04:15:13 Warning opnsense /usr/local/etc/rc.newwanipv6: dhcpd_radvd_configure(manual) found no suitable IPv6 address on opt2(igb2_vlan59)
2024-09-26T04:15:13 Notice dhcp6c dhcp6c_script: REQUEST on pppoe0 renewal
2024-09-26T04:15:13 Warning opnsense /usr/local/etc/rc.newwanipv6: dhcpd_radvd_configure(manual) found no suitable IPv6 address on lan(igb1)
2024-09-26T04:15:13 Notice dhcp6c dhcp6c_script: REQUEST on pppoe0 executing
2024-09-26T04:15:13 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
2024-09-26T04:15:13 Notice opnsense /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
2024-09-26T04:15:13 Notice opnsense /usr/local/etc/rc.newwanipv6: IP renewal starting (address: fe80::20d:b9ff:fe48:9070%pppoe0, interface: wan, device: pppoe0)
2024-09-26T04:15:11 Notice dhcp6c RTSOLD script - Sending SIGHUP to dhcp6c
2024-09-26T04:15:11 Notice opnsense /usr/local/etc/rc.newwanip: Chose to bind WAN_PPPOE on 95.117.229.XXX since we could not find a proper match.
2024-09-26T04:15:11 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_PPPOE))
2024-09-26T04:15:11 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_PPPOE)
2024-09-26T04:15:11 Notice opnsense /usr/local/etc/rc.newwanip: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP
2024-09-26T04:15:11 Notice opnsense /usr/local/etc/rc.newwanip: ROUTING: treating '62.52.192.83' as far gateway for '95.117.229.XXX/32'
2024-09-26T04:15:10 Notice opnsense /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
2024-09-26T04:15:10 Notice dhcp6c RTSOLD script - Sending SIGHUP to dhcp6c
2024-09-26T04:15:10 Notice opnsense /usr/local/etc/rc.newwanip: IP renewal starting (new: 95.117.229.XXX, old: 77.7.62.X, interface: wan, device: pppoe0, force: yes)
2024-09-26T04:15:09 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure dns (execute task : unbound_configure_do(1))
2024-09-26T04:15:09 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure dns (execute task : dnsmasq_configure_do(1))
2024-09-26T04:15:09 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure dns (1)
2024-09-26T04:15:08 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))
2024-09-26T04:15:08 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure dhcp (1)
2024-09-26T04:15:08 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure route_reload (execute task : core_routing_batch(1,[]))
2024-09-26T04:15:08 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure route_reload (1,[])
2024-09-26T04:15:05 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure ipsec (execute task : ipsec_configure_do(1,wan))
2024-09-26T04:15:05 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure ipsec (1,wan)
2024-09-26T04:15:05 Warning opnsense /usr/local/etc/rc.configure_interface: The required WAN_PPPOE IPv4 interface address could not be found, skipping.
2024-09-26T04:15:05 Warning opnsense /usr/local/etc/rc.configure_interface: Skipping gateway WAN_PPPOE due to empty 'gateway' property.
2024-09-26T04:15:05 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WAN_PPPOE))
2024-09-26T04:15:05 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WAN_PPPOE)
2024-09-26T04:15:05 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WAN_DHCP6))
2024-09-26T04:15:05 Notice opnsense /usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WAN_DHCP6)
2024-09-26T04:15:05 Notice opnsense /usr/local/etc/rc.configure_interface: ROUTING: ignoring down gateways: WAN_LTE_DHCP
2024-09-26T04:15:05 Warning opnsense /usr/local/etc/rc.configure_interface: ROUTING: refusing to set interface route on addressless wan(pppoe0)
2024-09-26T04:15:05 Notice opnsense /usr/local/etc/rc.configure_interface: ROUTING: entering configure using 'wan'
2024-09-26T04:15:04 Notice kernel <6>ng0: changing name to 'pppoe0'
2024-09-26T04:15:04 Warning rtsold <rtsock_input_ifannounce> interface ng0 removed
2024-09-26T04:15:04 Warning rtsold <rtsock_input_ifannounce> interface pppoe0 removed
2024-09-26T04:15:03 Notice opnsense /usr/local/etc/rc.newwanip: Failed to detect IP for interface wan
Looking at https://github.com/dennypage/dpinger/blob/master/dpinger.c#L366 I wonder if this shouldn't be fixed directly in dpinger? As long as dpinger is running as root, it should be possible to either re-bind()
or re-create the socket.
@KiNgMaR the patch is included and the cron job can fix the issue as it happens. However, we've been unable to find a good fix in the core for dpinger being stuck. A SIGHUP to rebind would be nice there, but the trigger for it is elusive. We're suspecting pf states may play a role here, because normally dpinger is able to recover (except maybe in the pppoe case but then it should also restart by default which it does). It's all entirely unclear in the grand scheme of things.
@fichtner I can reproduce with nearly 100% "success" rate, let me know if you want to debug together.
@KiNgMaR Good, can you look up the states for the stuck gateway monitor and try to clear them to see if that helps it to recover on its own?
@fichtner not sure if this response is better in here or in #7027 but you are mentioning states here :) For context I've just set up a Mikrotik LTE router in passthrough mode for Multi WAN failover and see this issue every time the 4g Connection drops and reconnects, at which time the interface in OPNSense gets a new IP. Restarting the gateway monitoring fixes it. I looked at the state table and this is what I found. Just after reconnection while the monitor is showing 100% packet loss :
root@opnsense:~ # pfctl -s states -vv | grep 8.8.4.4 -A3
No ALTQ support in kernel
ALTQ related functions disabled
all icmp [10.167.116.75:12671](http://10.167.116.75:12671/) -> [8.8.4.4:12671](http://8.8.4.4:12671/) 0:0
age 00:01:44, expires in 00:00:09, 104:0 pkts, 3016:0 bytes, rule 104, allow-opts
id: f1bd926800000000 creatorid: 829fe43d route-to: 10.193.49.46@vlan04
origif: vlan04
if I then remove this state the monitor starts working without needing restarted and the entry in the state table gets recreated as :
root@opnsense:~ # pfctl -s states -vv | grep 8.8.4.4 -A3
No ALTQ support in kernel
ALTQ related functions disabled
all icmp [10.167.116.75:12671](http://10.167.116.75:12671/) -> [8.8.4.4:12671](http://8.8.4.4:12671/) 0:0
age 00:00:06, expires in 00:00:09, 6:6 pkts, 174:174 bytes, rule 104, allow-opts
id: e6c5926800000000 creatorid: 829fe43d route-to: 10.167.116.76@vlan04
origif: vlan04
note the route-to has changed ... in the first one it is the old gateway ip and in the 2nd one it is the new gateway ip.
Hope that gives you something to work with.
@LucidityCrash Looks related to my other issue in https://github.com/opnsense/core/issues/7637
@LucidityCrash best fit seems #7027 indeed. Looks like route-to sends to the old gateway which makes this fail permanently until the state is flushed. Thanks a lot for this piece of the puzzle. Let me see if that state can be targeted.
@LucidityCrash is rule 104 one of your rules? Or at least a route where you added the gateway (route-to) yourself? Does it point to a gateway group or a single gateway?
@LucidityCrash can you try this patch? 73e837d7d3
# opnsense-patch 73e837d7d3
@LucidityCrash is rule 104 one of your rules? Or at least a route where you added the gateway (route-to) yourself? Does it point to a gateway group or a single gateway?
Not an expert but :
the rule is :
@104 pass out route-to (vlan04 10.167.116.76) inet from (vlan04:1) to ! (vlan04:network:1) flags S/SA keep state allow-opts label "dca43ea815d5974296ad34b7b77d1f97"
[ Evaluations: 84200 Packets: 4906 Bytes: 142274 States: 1 ]
[ Inserted: uid 0 pid 72809 State Creations: 1 ]
[ Last Active Time: Thu Nov 7 12:28:12 2024 ]
If I click on the link to the rule in the UI it takes me to the Firewall -> Settings -> Advanced -> Disable Forced Gateway option (which is unticked) so I believe is it an auto generated rule.
@LucidityCrash can you try this patch? 73e837d
# opnsense-patch 73e837d7d3
Not done that before, I'm currently running 24.7.5 and not on site, so a) can I apply this to a "not the latest" release, and b) will it require a reboot ?
I feel I should have updated to the latest before I added my comments but I found this bug while researching why the monitor didn't recover after a link reconnect and collected info and added it here without thinking straight :)
I should be able to update and test the patch properly tonight or tomorrow it that's OK.
"Firewall -> Settings -> Advanced -> Disable Forced Gateway" makes sense as the origin. So disabling this (all strings attached) would be a workaround.
Take your time. I believe this will require FreeBSD 14 / OPNsense 24.7 anyway. The change to pfctl is not that old.
"Firewall -> Settings -> Advanced -> Disable Forced Gateway" makes sense as the origin. So disabling this (all strings attached) would be a workaround.
Take your time. I believe this will require FreeBSD 14 / OPNsense 24.7 anyway. The change to pfctl is not that old.
Sorry for the possibly stupid question but can I double check my understanding of that option ? Ticking this will result in it not adding the 3 rules (one for each of my WAN interfaces - ipv4 and ipv6 on the main one an just ipv4 on the backup) that say "if the source is the WAN interface and the destination is not the WAN network then send the traffic to the WAN Gateway". This means that it will just use the system routing table only, other than not forcing all packets from the Backup_WAN interface to go via the Backup_WAN gateway are there any other downsides ?
In multi-WAN you want this behaviour in general so it's harder to switch the rules off although it only pertains to outgoing connections (and maybe outbound NAT). It really depends on your setup/requirements. For gateway monitoring you have a route ensuring that the right interface will be used on top of that either way.
What version are you on?
Cheers, Franco
24.7.5
Hmm, I think 24.7.5 should patch just fine :)
Applied the patch ... as you say a 1 line difference ... didn't appear to help in my case
During the problematic WAN switch? The modem should reconnect and cause rc.newwanip logs to appear in the system log.
Log entries for a reconnect situation
<13>1 2024-11-07T15:04:25+00:00 opnsense.telinformix.com dhclient 95917 - [meta sequenceId="4267"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4268"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4269"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4270"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to XXX.XXX.XXX.XXX
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4271"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4272"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to fe80::XXXXXXXXX%igc0
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4273"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1,[])
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4274"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1,[]))
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4275"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4276"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: 4G
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4277"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4278"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to XXX.XXX.XXX.XXX
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4279"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4280"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to fe80::XXXXXXXXXXX%igc0
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4281"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1,[4G])
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4282"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1,[4G]))
<13>1 2024-11-07T15:05:26+00:00 opnsense.telinformix.com dhclient 5613 - [meta sequenceId="4283"] dhclient-script: Reason EXPIRE on vlan04 executing
<13>1 2024-11-07T15:05:26+00:00 opnsense.telinformix.com dhclient 10195 - [meta sequenceId="4284"] dhclient-script: Reason PREINIT on vlan04 executing
<13>1 2024-11-07T15:05:27+00:00 opnsense.telinformix.com kernel - - [meta sequenceId="4285"] <7>arpresolve: can't allocate llinfo for 10.165.119.142 on vlan04
<13>1 2024-11-07T15:05:27+00:00 opnsense.telinformix.com dhclient 12474 - [meta sequenceId="4286"] dhclient-script: Reason ARPSEND on vlan04 executing
<13>1 2024-11-07T15:05:27+00:00 opnsense.telinformix.com kernel - - [meta sequenceId="4287"] <7>arpresolve: can't allocate llinfo for 10.165.119.142 on vlan04
<13>1 2024-11-07T15:05:28+00:00 opnsense.telinformix.com kernel - - [meta sequenceId="4288"] <7>arpresolve: can't allocate llinfo for 10.165.119.142 on vlan04
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 14022 - [meta sequenceId="4289"] dhclient-script: Reason ARPCHECK on vlan04 executing
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 15391 - [meta sequenceId="4290"] dhclient-script: Reason BOUND on vlan04 executing
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 19312 - [meta sequenceId="4291"] dhclient-script: New IP Address (vlan04): 10.192.250.79
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 20864 - [meta sequenceId="4292"] dhclient-script: New Subnet Mask (vlan04): 255.255.255.224
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 22572 - [meta sequenceId="4293"] dhclient-script: New Broadcast Address (vlan04): 10.192.250.95
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 24329 - [meta sequenceId="4294"] dhclient-script: New Routers (vlan04): 10.192.250.80
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 27032 - [meta sequenceId="4295"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4296"] /usr/local/etc/rc.newwanip: IP renewal starting (new: 10.192.250.79, old: 10.165.119.141, interface: opt3, device:
vlan04, force: yes)
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4297"] /usr/local/etc/rc.newwanip: ROUTING: entering configure using opt3
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4298"] /usr/local/etc/rc.newwanip: ROUTING: ignoring down gateways: 4G
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4299"] /usr/local/etc/rc.newwanip: ROUTING: configuring inet default gateway on wan
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4300"] /usr/local/etc/rc.newwanip: ROUTING: keeping inet default route to XXX.XXX.XXX.XXX
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4301"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,[4G])
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4302"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,[4G]))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4303"] /usr/local/etc/rc.newwanip: IP address change detected, killing states of old ip 10.165.119.141
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4304"] /usr/local/etc/rc.newwanip: plugins_configure vpn_map (,opt3,inet)
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4305"] /usr/local/etc/rc.newwanip: plugins_configure vpn_map (execute task : ipsec_configure_do(,opt3))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4306"] /usr/local/etc/rc.newwanip: plugins_configure vpn_map (execute task : openvpn_configure_do(,opt3))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4307"] /usr/local/etc/rc.newwanip: plugins_configure vpn_map (execute task : wireguard_configure_do())
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4308"] /usr/local/etc/rc.newwanip: plugins_configure vpn (,opt3)
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4309"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (,opt3)
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4310"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,opt3))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4311"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (,opt3,inet)
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4312"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : dhcrelay_configure_map(,opt3,inet))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4313"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : dnsmasq_configure_do())
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4314"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : miniupnpd_configure_do())
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4315"] /usr/local/etc/rc.newwanip: miniupnpd: Starting service on interface: lan
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4316"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : ntpd_configure_do())
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4317"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : opendns_configure_do())
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4318"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : openssh_configure_do(,opt3))
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4319"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : vxlan_configure_do())
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4320"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : webgui_configure_do(,opt3))
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4321"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : wireguard_sync())
<13>1 2024-11-07T15:05:59+00:00 opnsense.telinformix.com dhclient 81000 - [meta sequenceId="4322"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:05:59+00:00 opnsense.telinformix.com dhclient 81884 - [meta sequenceId="4323"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:06:29+00:00 opnsense.telinformix.com dhclient 65497 - [meta sequenceId="4324"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:06:29+00:00 opnsense.telinformix.com dhclient 67181 - [meta sequenceId="4325"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:06:59+00:00 opnsense.telinformix.com dhclient 69806 - [meta sequenceId="4326"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:06:59+00:00 opnsense.telinformix.com dhclient 71790 - [meta sequenceId="4327"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:07:29+00:00 opnsense.telinformix.com dhclient 57041 - [meta sequenceId="4328"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:07:29+00:00 opnsense.telinformix.com dhclient 58240 - [meta sequenceId="4329"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:07:59+00:00 opnsense.telinformix.com dhclient 69402 - [meta sequenceId="4330"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:07:59+00:00 opnsense.telinformix.com dhclient 69994 - [meta sequenceId="4331"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:08:29+00:00 opnsense.telinformix.com dhclient 50529 - [meta sequenceId="4332"] dhclient-script: Reason RENEW on vlan04 executing
Well according to here it ran into the execution:
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4303"] /usr/local/etc/rc.newwanip: IP address change detected, killing states of old ip 10.165.119.141 Check the stuck state first like before. Does running it manually help then? # pfctl -k gateway -k 10.165.119.141 Check states again. Cheers, FrancoHehe ... I literally just did that ... and yes it fixed the problem ... old state entry now removed and new one with the right route-to address is added and the monitor started working
Ok, good. Make sure the change is still applied to the file and see what happens on the next reconnect. It looks like this will start working eventually :)
File is still the same (ie has the extra line in) ... forced a reconnect and the bad state entry is still there ... this kinda looks like a race condition. It is almost as if the interface gets a new IP, the newwanip script runs removes any existing states then a new state entry is added (I'm assuming because dpinger is restarted and starts trying to ping the monitor address) with the new wan ip as the source but the old gateway ip as the route-to.
routes, monitors and filters are reloaded before the cache IP purge and this looks like the correct order. Can you add a sleep before the 3 pfctl invokes? just a simple sleep(2);
fraid not ... I bumped that to 10 Seconds and still no luck though it did allow me to capture this from the states table :
root@opnsense:/var/log/system # pfctl -s states -vv | grep 8\.8\.4\.4 -A3
No ALTQ support in kernel
ALTQ related functions disabled
all icmp 10.50.82.1:27599 -> 8.8.4.4:27599 0:0
age 00:00:27, expires in 00:00:07, 25:0 pkts, 725:0 bytes, rule 104, allow-opts
id: f337966800000000 creatorid: 829fe43d route-to: 10.50.82.2@vlan04
origif: vlan04
--
all icmp 10.192.250.79:46510 -> 8.8.4.4:46510 0:0
age 00:00:03, expires in 00:00:10, 3:0 pkts, 87:0 bytes, rule 104, allow-opts
id: af38966800000000 creatorid: 829fe43d route-to: 10.50.82.2@vlan04
origif: vlan04
Just wanna add, sorry for the back and forth and thank you for your amazing support in trying to resolve this problem
Ah sorry we need to kill the old gateway not the old local IP (:
Sorry ... What does this mean with regards to how it is fixed ?
I’ll propose a patch in a bit. Needs more shuffling as the gateway is not cached yet.
Cool ... was just worried you expected me to understand what needed to be done ... I'm not that smart 😀
Neither am I, but anyway: 270aac573d9 applies on top of the other one
# opnsense-patch 270aac573d9
Cheers, Franco
I updated to 24.7.8 this morining and applied those 2 patches. The first time I forced a reconnection it didn't seem to work, and I removed the state entry manually. I have since forced a reconnect multiple times and it has worked each time.
Thank you so much.
The first time not working is normal because it didn't cache the gateway IP yet :) This sounds great, thanks a lot for your debugging help!
I'll keep an eye out for it in a release ... looking at some of the threads in the forums there may be some happy users ... seemed to have hit a number of Starlink users in particular.
Starlink users always have issues from past experiences. ;)
Apparently #7027 isn't the same, but the overall patch looks ok to be introduced in 24.7.9 anyway. @deajan can you confirm as well?
Just to confirm I had an un forced reconnection last night and the new code handled it perfectly. (you never know there may have been a slight difference in the failure state compared to the manually forced reconnect)
Applied those two patches eight days ago. And while I didn't have a maintenance window to properly do some exploration, the nightly re-connect of one of the WANs has been handled perfectly since - which didn't work before, dpinger would always get stuck. Thank you @fichtner! 💪
@KiNgMaR happy to hear, the change will hit 24.7.9 early this week :)
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
The following bug has been noticed and experienced from OPNSense 24.1.6 to 24.1.10_3 (current).
I have a multi-WAN setup with two links. As of today, I tend to use
1.1.1.1
,1.1.1.3
,4.4.4.4
,8.8.8.8
and9.9.9.9
as monitor IP for gateway monitoring. So far so good, my seconary WAN link seems down according to gateway monitoring:Still, I can use the link, and can also ping via OPNSense interface/diagnostic page:
I have checked that the monitor IP
1.1.1.1
is not bound to any interface in DNS page: Nor bound to in interface in the routing table:Updating OPNSense and rebooting doesn't help. Gateway monitor settings are pretty basic:
My ISP blocks ICMP pings to the gateway, hence the reason I use
1.1.1.1
as IP for gateway monitoring.Is there any reason that the gateway is marked offline, whereas it can ping
1.1.1.1
? Perhaps dpinger works like traceroute and doesn't like the fact that the first hop isn't pingable ?[EDIT] I can confirm that the gateway that is marked offline works, since a test from VM behind the OPNsense shows that the offline gateway is used (I'm using a rule with that explicit gateway):
To Reproduce
Sorry, there's no reproducer that I can suggest here.
Expected behavior
Since OPNSense can ping
1.1.1.1
through the gateway which uses that IP as monitoring IP, I would expect the gateway to be online intead of offline.Software version used and hardware type if relevant, e.g.:
OPNsense 24.1.10_3 (amd64) on KVM with virtio NICs