opnsense / core

OPNsense GUI, API and systems backend
https://opnsense.org/
BSD 2-Clause "Simplified" License
3.38k stars 759 forks source link

Gateway monitoring says link is down whereas link actually works (probably not OPNsense related) #7635

Open deajan opened 4 months ago

deajan commented 4 months ago

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

The following bug has been noticed and experienced from OPNSense 24.1.6 to 24.1.10_3 (current).

I have a multi-WAN setup with two links. As of today, I tend to use 1.1.1.1, 1.1.1.3, 4.4.4.4, 8.8.8.8 and 9.9.9.9 as monitor IP for gateway monitoring. So far so good, my seconary WAN link seems down according to gateway monitoring:

image

Still, I can use the link, and can also ping via OPNSense interface/diagnostic page: image

I have checked that the monitor IP 1.1.1.1 is not bound to any interface in DNS page: image Nor bound to in interface in the routing table: image

Updating OPNSense and rebooting doesn't help. Gateway monitor settings are pretty basic: image

My ISP blocks ICMP pings to the gateway, hence the reason I use 1.1.1.1 as IP for gateway monitoring.

Is there any reason that the gateway is marked offline, whereas it can ping 1.1.1.1 ? Perhaps dpinger works like traceroute and doesn't like the fact that the first hop isn't pingable ?

[EDIT] I can confirm that the gateway that is marked offline works, since a test from VM behind the OPNsense shows that the offline gateway is used (I'm using a rule with that explicit gateway):

curl 'http://api.ipify.org?format=json'
{"ip":"172.25.XX.XX"}

To Reproduce

Sorry, there's no reproducer that I can suggest here.

Expected behavior

Since OPNSense can ping 1.1.1.1 through the gateway which uses that IP as monitoring IP, I would expect the gateway to be online intead of offline.

Software version used and hardware type if relevant, e.g.:

OPNsense 24.1.10_3 (amd64) on KVM with virtio NICs

deajan commented 4 months ago

I realized that my above diagnostic is a bit botched.

So I added a rule forcing ICMP to use the offline marked gateway for one of my machines behind OPNSense. From that machine, pinging 1.1.1.1 with that rule still works, so I can confirm that something isn't right with the way dpinger thinks my gateway is offline, when monitor IP is 1.1.1.1.

I've also tried the following from the OPNsense console

Ping works

ping -S 172.25.XX.XX 1.1.1.1

PING 1.1.1.1 (1.1.1.1) from 172.252.236.42: 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=53 time=21.643 ms

Dpinger fails from my second WAN which is marked offline

dpinger -f -B 172.25.X.X -s 1s -l 4s -t 60s -d 1 1.1.1.1

send_interval 500ms  loss_interval 2000ms  time_period 60000ms  report_interval 1000ms  data_len 0  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr 172.25.XX.XX identifier ""
0 0 0
0 0 100

But dpinger works from my primary WAN

dpinger -f -B 194.87.XX.XX  -s 1s -l 4s -t 60s -d 1.1.1.1
send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 1000ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 8.8.8.8  bind_addr 194.87.XX.XX  identifier ""
26746 0 0
26672 179 0
26722 92 0

Investigating further, it seems that when dpinger's payload is set to 1, it fails. If I set the payload to 4 or higher, it works. I could confirm this by using ping:

The following works:

ping -S 172.25.XX.XX -s 4 1.1.1.1

The following doesn't;

ping -S 172.25.XX.XX -s 1 1.1.1.1

Playing with the packet size, I realized that every packet size between 4 and 172 bytes works, and others don't. This only happens on the secondary link, the primary link accepts usual packet sizes.

So end of diagnostic: It seems that there's something fishy with the link itself, and not OPNSense. Sorry for the noise.

I have yet to find what the problem is at my datacenter. If anyone has a clue, I will gladly take it ;)

dwkirw commented 3 months ago

Please let me know if you found anything - I'm having the exact same problem but only since 24.7 came along.

deajan commented 3 months ago

@dwkirw I've pushed the diagnostics. Whenever I use values for ping -s n where 4 <= n <= 172, I get to ping. My datacenter guys told me that they have no filter whatsoever, but on the second link they provide I don't have that problem, so to be honest I doubt that this is an OPNSense problem. Btw, I'm running OPNsense 24.1.10_3-amd64 on this unit, so I had this problem prior to 24.7.

What exactly is your problem ? Did you try adding a payload to the gateway monitor ? Does it work ?

dwkirw commented 3 months ago

Thank you - I've had a better read of what you wrote and tried some of the stuff. Also, disregard the only on 24.7 thing - I've only had a backup WAN about 6 weeks and likely havent noticed this I have not played around with dpinger before. From the primary wan (which is currently marked as down) i get stuff such as this.
dpinger -f -B 139.5.x.x -s 1s -l 4s -t 60s -d 1 1.1.1.1 7650 0 0 7288 371 0 7370 316 0 7289 324 0

From backup (currently showing Online) i get this dpinger -f -B 192.168.1.2 -s 1s -l 4s -t 60s -d 1 1.1.1.1 36686 0 0 33661 3030 0 35521 3611 0 34479 3608 0

My primary is PPPoE, its IP changes fairly often. Heres its settings. The IP has been left blank for this reason. image. While the primary wan is appearing to be down but still working if i go to System>Configuration>Gateways and hit the Apply button it goes to Online again. While its marked Offline my port forwards are flakey though sometimes working/not...

If you can see any dumb things I've missed or should try I'd appreciate knowing about them, ta!

KiNgMaR commented 2 months ago

It looks like the patch from issue #7027 is not included yet in 24.7.1? @fichtner ? I'm having a very similar issue, both with 24.1, as well as 24.7. Can reproduce like clockwork if needed.

It's happening with 1.1.1.1 as monitor host as well as the PPPOE connection's gateway. The IP given to dpinger -S is the correct one.

It's always triggered by a PPPOE reconnect, which sends dpinger into this state:

2024-09-26T06:06:17 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65
...
2024-09-26T04:15:18 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:17 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:16 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:16 Notice  dpinger ALERT: WAN_PPPOE (Addr: 1.1.1.1 Alarm: none -> down RTT: 0.0 ms RTTd: 0.0 ms Loss: 100.0 %) 
2024-09-26T04:15:15 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:14 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:13 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:12 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:11 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:11 Warning dpinger send_interval 1000ms loss_interval 4000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 1.1.1.1 bind_addr 95.117.229.255 identifier "WAN_PPPOE "    
2024-09-26T04:15:05 Notice  dpinger Reloaded gateway watcher configuration on SIGHUP    
2024-09-26T04:15:05 Warning dpinger exiting on signal 15    
2024-09-26T04:15:05 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:04 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:03 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65 
2024-09-26T04:15:02 Warning dpinger WAN_PPPOE 1.1.1.1: sendto error: 65

It's immediately fixed by reloading the monitor plugin config.

Here's some debug output:

root@OPNsense:~ # pluginctl -r return_gateways_status
{
    "dpinger": {
        "WAN_DHCP6": {
            "status": "none",
            "monitor": "~",
            "name": "WAN_DHCP6",
            "stddev": "~",
            "delay": "~",
            "loss": "~"
        },
        "WAN_PPPOE": {
            "status": "down",
            "monitor": "1.1.1.1",
            "name": "WAN_PPPOE",
            "stddev": "0.0 ms",
            "delay": "0.0 ms",
            "loss": "100.0 %"
        },
        "WAN_LTE_DHCP": {
            "status": "none",
            "monitor": "192.168.22.1",
            "name": "WAN_LTE_DHCP",
            "stddev": "5.8 ms",
            "delay": "1.4 ms",
            "loss": "0.0 %"
        }
    }
}

root@OPNsense:~ # ps auxf | grep dping
root    84723   0.0  0.1   17572  2780  -  Is   04:15      0:04.21 /usr/local/bin/dpinger -f -S -r 0 -i WAN_PPPOE -B 95.117.229.XXX -p /var/run/dpinger_WAN_PPPOE.pid -u /var/run/dpinger_WAN_PPPOE.sock -s 1s -l 4s -t 60s -d 1 1.1.1.1
root    90835   0.0  0.1   17572  2792  -  Is   19:57      0:14.78 /usr/local/bin/dpinger -f -S -r 0 -i WAN_LTE_DHCP -B 192.168.22.101 -p /var/run/dpinger_WAN_LTE_DHCP.pid -u /var/run/dpinger_WAN_LTE_DHCP.sock -s 1s -l 4s -t 60s -d 1 192

root@OPNsense:~ # pluginctl -c monitor
Setting up gateway monitors...done.

root@OPNsense:~ # ps auxf | grep dping
root    67987  10.6  0.1   13344  2716  -  Ss   06:59      0:00.01 /usr/local/bin/dpinger -f -S -r 0 -i WAN_LTE_DHCP -B 192.168.22.101 -p /var/run/dpinger_WAN_LTE_DHCP.pid -u /var/run/dpinger_WAN_LTE_DHCP.sock -s 1s -l 4s -t 60s -d 1 192
root    67240  10.4  0.1   13344  2716  -  Ss   06:59      0:00.01 /usr/local/bin/dpinger -f -S -r 0 -i WAN_PPPOE -B 95.117.229.XXX -p /var/run/dpinger_WAN_PPPOE.pid -u /var/run/dpinger_WAN_PPPOE.sock -s 1s -l 4s -t 60s -d 1 1.1.1.1

root@OPNsense:~ # pluginctl -r return_gateways_status
{
    "dpinger": {
        "WAN_DHCP6": {
            "status": "none",
            "monitor": "~",
            "name": "WAN_DHCP6",
            "stddev": "~",
            "delay": "~",
            "loss": "~"
        },
        "WAN_PPPOE": {
            "status": "none",
            "monitor": "1.1.1.1",
            "name": "WAN_PPPOE",
            "stddev": "119.4 ms",
            "delay": "49.2 ms",
            "loss": "0.0 %"
        },
        "WAN_LTE_DHCP": {
            "status": "none",
            "monitor": "192.168.22.1",
            "name": "WAN_LTE_DHCP",
            "stddev": "125.7 ms",
            "delay": "42.6 ms",
            "loss": "0.0 %"
        }
    }
}

And the general log:


2024-09-26T04:19:32 Error   dhclient    send_packet: Host is down   
2024-09-26T04:18:23 Error   dhclient    send_packet: Host is down   
2024-09-26T04:17:31 Error   dhclient    send_packet: Host is down   
2024-09-26T04:16:09 Error   opnsense    /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 10158' 
2024-09-26T04:16:09 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,opt2)) 
2024-09-26T04:16:09 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,opt2))    
2024-09-26T04:16:09 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())  
2024-09-26T04:16:09 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,opt2))   
2024-09-26T04:16:09 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,opt2))   
2024-09-26T04:16:09 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())    
2024-09-26T04:16:09 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())   
2024-09-26T04:16:09 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,opt2))  
2024-09-26T04:16:08 Warning igmpproxy   select() failure; Errno(4): Interrupted system call 
2024-09-26T04:16:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())  
2024-09-26T04:16:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())    
2024-09-26T04:16:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,opt2)    
2024-09-26T04:16:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,opt2))  
2024-09-26T04:16:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface LAN_WLAN.   
2024-09-26T04:16:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,opt2))    
2024-09-26T04:16:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,opt2))  
2024-09-26T04:16:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,opt2) 
2024-09-26T04:16:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,lan))  
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,lan)) 
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())  
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,lan))    
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,lan))    
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())    
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())   
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,lan))   
2024-09-26T04:16:06 Warning igmpproxy   select() failure; Errno(4): Interrupted system call 
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())  
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())    
2024-09-26T04:16:06 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,lan) 
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,lan))   
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface LAN_SERVER. 
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,lan)) 
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,lan))   
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,lan)  
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,wan))  
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,wan)) 
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())  
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,wan))    
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,wan))    
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())    
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())   
2024-09-26T04:16:05 Error   opnsense    /usr/local/etc/rc.newwanipv6: Primary WAN pppoe0 change detected, killing states of failover WAN igb2_vlan22 / 192.168.22.101   
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,wan))   
2024-09-26T04:16:05 Warning igmpproxy   select() failure; Errno(4): Interrupted system call 
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())  
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())    
2024-09-26T04:16:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,wan) 
2024-09-26T04:16:04 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,wan))   
2024-09-26T04:15:59 Warning opnsense    /usr/local/etc/rc.newwanip: Interface '' (ovpns1) is disabled or empty, nothing to do.  
2024-09-26T04:15:58 Notice  opnsense    /usr/local/etc/rc.newwanipv6: OpenVPN client 2 instance started on PID 37591.   
2024-09-26T04:15:58 Notice  kernel  <6>ovpns1: link state changed to UP 
2024-09-26T04:15:58 Notice  opnsense    /usr/local/etc/rc.newwanipv6: OpenVPN server 1 instance started on PID 33027.   
2024-09-26T04:15:58 Notice  kernel  <6>ovpns1: link state changed to DOWN   
2024-09-26T04:15:58 Notice  opnsense    /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.    
2024-09-26T04:15:58 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,wan)) 
2024-09-26T04:15:58 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure route_reload (execute task : core_routing_batch(,[]))   
2024-09-26T04:15:58 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure route_reload (,[])  
2024-09-26T04:15:55 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,wan))   
2024-09-26T04:15:55 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,wan)  
2024-09-26T04:15:51 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0  
2024-09-26T04:15:51 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan 
2024-09-26T04:15:51 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4    
2024-09-26T04:15:51 Error   dhclient    send_packet: Host is down   
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'opt2'  
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0  
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan 
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4    
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'lan'   
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))   
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)    
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0  
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan 
2024-09-26T04:15:50 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4    
2024-09-26T04:15:49 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'   
2024-09-26T04:15:49 Warning radvd   exiting, 1 sigterm(s) received  
2024-09-26T04:15:48 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))  
2024-09-26T04:15:48 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)   
2024-09-26T04:15:48 Notice  opnsense    /usr/local/etc/rc.newwanipv6: IP renewal starting (address: fe80::20d:b9ff:fe48:9070%pppoe0, interface: wan, device: pppoe0)    
2024-09-26T04:15:45 Error   opnsense    /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 32728' 
2024-09-26T04:15:44 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,opt2)) 
2024-09-26T04:15:44 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,opt2))    
2024-09-26T04:15:44 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())  
2024-09-26T04:15:44 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,opt2))   
2024-09-26T04:15:44 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,opt2))   
2024-09-26T04:15:44 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())    
2024-09-26T04:15:44 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())   
2024-09-26T04:15:44 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,opt2))  
2024-09-26T04:15:43 Warning igmpproxy   select() failure; Errno(4): Interrupted system call 
2024-09-26T04:15:43 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())  
2024-09-26T04:15:43 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())    
2024-09-26T04:15:43 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,opt2)    
2024-09-26T04:15:42 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,opt2))  
2024-09-26T04:15:42 Notice  opnsense    /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface LAN_WLAN.   
2024-09-26T04:15:42 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,opt2))    
2024-09-26T04:15:42 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,opt2))  
2024-09-26T04:15:42 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,opt2) 
2024-09-26T04:15:42 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,lan))  
2024-09-26T04:15:41 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,lan)) 
2024-09-26T04:15:41 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())  
2024-09-26T04:15:41 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,lan))    
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,lan))    
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())    
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())   
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,lan))   
2024-09-26T04:15:40 Warning igmpproxy   select() failure; Errno(4): Interrupted system call 
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())  
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())    
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,lan) 
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,lan))   
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface LAN_SERVER. 
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,lan)) 
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,lan))   
2024-09-26T04:15:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,lan)  
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : wireguard_sync(,wan))  
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,wan)) 
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0   
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan  
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4 
2024-09-26T04:15:39 Error   opnsense    /usr/local/etc/rc.routing_configure: ROUTING: not a valid opt4 interface gateway address: 'missing' 
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())  
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: treating '62.52.192.83' as far gateway for '95.117.229.XXX/32'    
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,wan))    
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,wan))    
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())    
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())   
2024-09-26T04:15:39 Error   opnsense    /usr/local/etc/rc.newwanipv6: Primary WAN pppoe0 change detected, killing states of failover WAN igb2_vlan22 / 192.168.22.101   
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : reset_failover_states(,wan))   
2024-09-26T04:15:39 Warning igmpproxy   select() failure; Errno(4): Interrupted system call 
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : igmpproxy_configure_do())  
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())    
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,wan) 
2024-09-26T04:15:39 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults 
2024-09-26T04:15:38 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : wireguard_configure_do(,wan))   
2024-09-26T04:15:34 Error   opnsense    /usr/local/etc/rc.newwanip: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 57282'   
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : wireguard_sync(,wan))    
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,wan))   
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())    
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,wan))  
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,wan))  
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())  
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do()) 
2024-09-26T04:15:34 Error   opnsense    /usr/local/etc/rc.newwanip: Primary WAN pppoe0 change detected, killing states of failover WAN igb2_vlan22 / 192.168.22.101 
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : reset_failover_states(,wan)) 
2024-09-26T04:15:34 Warning igmpproxy   select() failure; Errno(4): Interrupted system call 
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : igmpproxy_configure_do())    
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())  
2024-09-26T04:15:34 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (,wan)   
2024-09-26T04:15:33 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : wireguard_configure_do(,wan)) 
2024-09-26T04:15:29 Warning opnsense    /usr/local/etc/rc.newwanip: Interface '' (ovpns1) is disabled or empty, nothing to do.  
2024-09-26T04:15:29 Notice  opnsense    /usr/local/etc/rc.newwanipv6: OpenVPN client 2 instance started on PID 40848.   
2024-09-26T04:15:28 Notice  kernel  <6>ovpns1: link state changed to UP 
2024-09-26T04:15:27 Notice  opnsense    /usr/local/etc/rc.newwanipv6: OpenVPN server 1 instance started on PID 28432.   
2024-09-26T04:15:27 Notice  kernel  <6>ovpns1: link state changed to DOWN   
2024-09-26T04:15:27 Notice  opnsense    /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.    
2024-09-26T04:15:27 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,wan)) 
2024-09-26T04:15:27 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure route_reload (execute task : core_routing_batch(,[]))   
2024-09-26T04:15:27 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure route_reload (,[])  
2024-09-26T04:15:24 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,wan))   
2024-09-26T04:15:24 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,wan)  
2024-09-26T04:15:22 Warning opnsense    /usr/local/etc/rc.newwanip: Interface '' (ovpns1) is disabled or empty, nothing to do.  
2024-09-26T04:15:22 Notice  opnsense    /usr/local/etc/rc.newwanip: OpenVPN client 2 instance started on PID 94442. 
2024-09-26T04:15:21 Notice  kernel  <6>ovpnc2: link state changed to DOWN   
2024-09-26T04:15:21 Notice  kernel  <6>ovpns1: link state changed to UP 
2024-09-26T04:15:21 Notice  opnsense    /usr/local/etc/rc.newwanip: OpenVPN server 1 instance started on PID 84548. 
2024-09-26T04:15:21 Notice  kernel  <7>cannot forward src fe80:2::81bc:8d60:3c1b:248b, dst 2a02:3100:a106:ea23:20d:b9ff:YYYY:XXXX, nxt 58, rcvif igb1, outif pppoe0 
2024-09-26T04:15:21 Notice  kernel  <6>ovpns1: link state changed to DOWN   
2024-09-26T04:15:20 Notice  opnsense    /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.  
2024-09-26T04:15:20 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,wan))   
2024-09-26T04:15:20 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure route_reload (execute task : core_routing_batch(,[])) 
2024-09-26T04:15:20 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure route_reload (,[])    
2024-09-26T04:15:20 Notice  kernel  <7>cannot forward src fe80:2::81bc:8d60:3c1b:248b, dst 2a02:3100:a106:ea23:20d:b9ff:YYYY:XXXX, nxt 58, rcvif igb1, outif pppoe0 
2024-09-26T04:15:19 Notice  kernel  <7>cannot forward src fe80:2::81bc:8d60:3c1b:248b, dst 2a02:3100:a106:ea23:20d:b9ff:YYYY:XXXX, nxt 58, rcvif igb1, outif pppoe0 
2024-09-26T04:15:17 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0   
2024-09-26T04:15:17 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan  
2024-09-26T04:15:17 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP, WARP_GWv6, WARP_GWv4 
2024-09-26T04:15:17 Error   opnsense    /usr/local/etc/rc.routing_configure: ROUTING: not a valid opt4 interface gateway address: 'missing' 
2024-09-26T04:15:17 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: treating '62.52.192.83' as far gateway for '95.117.229.XXX/32'    
2024-09-26T04:15:17 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,wan)) 
2024-09-26T04:15:17 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (,wan)    
2024-09-26T04:15:17 Notice  opnsense    /usr/local/etc/rc.newwanip: IP address change detected, killing states of old ip 77.7.62.207    
2024-09-26T04:15:16 Notice  opnsense    /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults 
2024-09-26T04:15:16 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0  
2024-09-26T04:15:16 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan 
2024-09-26T04:15:16 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP  
2024-09-26T04:15:15 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'opt2'  
2024-09-26T04:15:15 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0  
2024-09-26T04:15:15 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan 
2024-09-26T04:15:15 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP  
2024-09-26T04:15:14 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'lan'   
2024-09-26T04:15:14 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))   
2024-09-26T04:15:14 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)    
2024-09-26T04:15:14 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::f6e4:51ff:fed3:bc97%pppoe0  
2024-09-26T04:15:14 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan 
2024-09-26T04:15:14 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP  
2024-09-26T04:15:14 Notice  opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'   
2024-09-26T04:15:14 Warning radvd   exiting, 1 sigterm(s) received  
2024-09-26T04:15:13 Warning opnsense    /usr/local/etc/rc.newwanipv6: dhcpd_radvd_configure(manual) found no suitable IPv6 address on opt2(igb2_vlan59) 
2024-09-26T04:15:13 Notice  dhcp6c  dhcp6c_script: REQUEST on pppoe0 renewal    
2024-09-26T04:15:13 Warning opnsense    /usr/local/etc/rc.newwanipv6: dhcpd_radvd_configure(manual) found no suitable IPv6 address on lan(igb1) 
2024-09-26T04:15:13 Notice  dhcp6c  dhcp6c_script: REQUEST on pppoe0 executing  
2024-09-26T04:15:13 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))  
2024-09-26T04:15:13 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)   
2024-09-26T04:15:13 Notice  opnsense    /usr/local/etc/rc.newwanipv6: IP renewal starting (address: fe80::20d:b9ff:fe48:9070%pppoe0, interface: wan, device: pppoe0)    
2024-09-26T04:15:11 Notice  dhcp6c  RTSOLD script - Sending SIGHUP to dhcp6c    
2024-09-26T04:15:11 Notice  opnsense    /usr/local/etc/rc.newwanip: Chose to bind WAN_PPPOE on 95.117.229.XXX since we could not find a proper match.   
2024-09-26T04:15:11 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_PPPOE)) 
2024-09-26T04:15:11 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_PPPOE)  
2024-09-26T04:15:11 Notice  opnsense    /usr/local/etc/rc.newwanip: ROUTING: ignoring down gateways: WAN_PPPOE, WAN_LTE_DHCP    
2024-09-26T04:15:11 Notice  opnsense    /usr/local/etc/rc.newwanip: ROUTING: treating '62.52.192.83' as far gateway for '95.117.229.XXX/32' 
2024-09-26T04:15:10 Notice  opnsense    /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan' 
2024-09-26T04:15:10 Notice  dhcp6c  RTSOLD script - Sending SIGHUP to dhcp6c    
2024-09-26T04:15:10 Notice  opnsense    /usr/local/etc/rc.newwanip: IP renewal starting (new: 95.117.229.XXX, old: 77.7.62.X, interface: wan, device: pppoe0, force: yes)   
2024-09-26T04:15:09 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure dns (execute task : unbound_configure_do(1))   
2024-09-26T04:15:09 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure dns (execute task : dnsmasq_configure_do(1))   
2024-09-26T04:15:09 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure dns (1)    
2024-09-26T04:15:08 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))  
2024-09-26T04:15:08 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure dhcp (1)   
2024-09-26T04:15:08 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure route_reload (execute task : core_routing_batch(1,[])) 
2024-09-26T04:15:08 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure route_reload (1,[])    
2024-09-26T04:15:05 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure ipsec (execute task : ipsec_configure_do(1,wan))   
2024-09-26T04:15:05 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure ipsec (1,wan)  
2024-09-26T04:15:05 Warning opnsense    /usr/local/etc/rc.configure_interface: The required WAN_PPPOE IPv4 interface address could not be found, skipping.  
2024-09-26T04:15:05 Warning opnsense    /usr/local/etc/rc.configure_interface: Skipping gateway WAN_PPPOE due to empty 'gateway' property.  
2024-09-26T04:15:05 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WAN_PPPOE)) 
2024-09-26T04:15:05 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WAN_PPPOE)  
2024-09-26T04:15:05 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WAN_DHCP6)) 
2024-09-26T04:15:05 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WAN_DHCP6)  
2024-09-26T04:15:05 Notice  opnsense    /usr/local/etc/rc.configure_interface: ROUTING: ignoring down gateways: WAN_LTE_DHCP    
2024-09-26T04:15:05 Warning opnsense    /usr/local/etc/rc.configure_interface: ROUTING: refusing to set interface route on addressless wan(pppoe0)  
2024-09-26T04:15:05 Notice  opnsense    /usr/local/etc/rc.configure_interface: ROUTING: entering configure using 'wan'  
2024-09-26T04:15:04 Notice  kernel  <6>ng0: changing name to 'pppoe0'   
2024-09-26T04:15:04 Warning rtsold  <rtsock_input_ifannounce> interface ng0 removed 
2024-09-26T04:15:04 Warning rtsold  <rtsock_input_ifannounce> interface pppoe0 removed  
2024-09-26T04:15:03 Notice  opnsense    /usr/local/etc/rc.newwanip: Failed to detect IP for interface wan

Looking at https://github.com/dennypage/dpinger/blob/master/dpinger.c#L366 I wonder if this shouldn't be fixed directly in dpinger? As long as dpinger is running as root, it should be possible to either re-bind() or re-create the socket.

fichtner commented 2 months ago

@KiNgMaR the patch is included and the cron job can fix the issue as it happens. However, we've been unable to find a good fix in the core for dpinger being stuck. A SIGHUP to rebind would be nice there, but the trigger for it is elusive. We're suspecting pf states may play a role here, because normally dpinger is able to recover (except maybe in the pppoe case but then it should also restart by default which it does). It's all entirely unclear in the grand scheme of things.

KiNgMaR commented 2 months ago

@fichtner I can reproduce with nearly 100% "success" rate, let me know if you want to debug together.

fichtner commented 2 months ago

@KiNgMaR Good, can you look up the states for the stuck gateway monitor and try to clear them to see if that helps it to recover on its own?

LucidityCrash commented 2 weeks ago

@fichtner not sure if this response is better in here or in #7027 but you are mentioning states here :) For context I've just set up a Mikrotik LTE router in passthrough mode for Multi WAN failover and see this issue every time the 4g Connection drops and reconnects, at which time the interface in OPNSense gets a new IP. Restarting the gateway monitoring fixes it. I looked at the state table and this is what I found. Just after reconnection while the monitor is showing 100% packet loss :

root@opnsense:~ # pfctl -s states -vv | grep 8.8.4.4 -A3
No ALTQ support in kernel
ALTQ related functions disabled
all icmp [10.167.116.75:12671](http://10.167.116.75:12671/) -> [8.8.4.4:12671](http://8.8.4.4:12671/)       0:0
   age 00:01:44, expires in 00:00:09, 104:0 pkts, 3016:0 bytes, rule 104, allow-opts
   id: f1bd926800000000 creatorid: 829fe43d route-to: 10.193.49.46@vlan04
   origif: vlan04

if I then remove this state the monitor starts working without needing restarted and the entry in the state table gets recreated as :

root@opnsense:~ # pfctl -s states -vv | grep 8.8.4.4 -A3
No ALTQ support in kernel
ALTQ related functions disabled
all icmp [10.167.116.75:12671](http://10.167.116.75:12671/) -> [8.8.4.4:12671](http://8.8.4.4:12671/)       0:0
   age 00:00:06, expires in 00:00:09, 6:6 pkts, 174:174 bytes, rule 104, allow-opts
   id: e6c5926800000000 creatorid: 829fe43d route-to: 10.167.116.76@vlan04
   origif: vlan04

note the route-to has changed ... in the first one it is the old gateway ip and in the 2nd one it is the new gateway ip.

Hope that gives you something to work with.

deajan commented 2 weeks ago

@LucidityCrash Looks related to my other issue in https://github.com/opnsense/core/issues/7637

fichtner commented 2 weeks ago

@LucidityCrash best fit seems #7027 indeed. Looks like route-to sends to the old gateway which makes this fail permanently until the state is flushed. Thanks a lot for this piece of the puzzle. Let me see if that state can be targeted.

fichtner commented 2 weeks ago

@LucidityCrash is rule 104 one of your rules? Or at least a route where you added the gateway (route-to) yourself? Does it point to a gateway group or a single gateway?

fichtner commented 2 weeks ago

@LucidityCrash can you try this patch? 73e837d7d3

# opnsense-patch 73e837d7d3
LucidityCrash commented 2 weeks ago

@LucidityCrash is rule 104 one of your rules? Or at least a route where you added the gateway (route-to) yourself? Does it point to a gateway group or a single gateway?

Not an expert but :

the rule is :

@104 pass out route-to (vlan04 10.167.116.76) inet from (vlan04:1) to ! (vlan04:network:1) flags S/SA keep state allow-opts label "dca43ea815d5974296ad34b7b77d1f97"
  [ Evaluations: 84200     Packets: 4906      Bytes: 142274      States: 1     ]
  [ Inserted: uid 0 pid 72809 State Creations: 1     ]
  [ Last Active Time: Thu Nov  7 12:28:12 2024 ]

If I click on the link to the rule in the UI it takes me to the Firewall -> Settings -> Advanced -> Disable Forced Gateway option (which is unticked) so I believe is it an auto generated rule.

LucidityCrash commented 2 weeks ago

@LucidityCrash can you try this patch? 73e837d

# opnsense-patch 73e837d7d3

Not done that before, I'm currently running 24.7.5 and not on site, so a) can I apply this to a "not the latest" release, and b) will it require a reboot ?

I feel I should have updated to the latest before I added my comments but I found this bug while researching why the monitor didn't recover after a link reconnect and collected info and added it here without thinking straight :)

I should be able to update and test the patch properly tonight or tomorrow it that's OK.

fichtner commented 2 weeks ago

"Firewall -> Settings -> Advanced -> Disable Forced Gateway" makes sense as the origin. So disabling this (all strings attached) would be a workaround.

Take your time. I believe this will require FreeBSD 14 / OPNsense 24.7 anyway. The change to pfctl is not that old.

LucidityCrash commented 2 weeks ago

"Firewall -> Settings -> Advanced -> Disable Forced Gateway" makes sense as the origin. So disabling this (all strings attached) would be a workaround.

Take your time. I believe this will require FreeBSD 14 / OPNsense 24.7 anyway. The change to pfctl is not that old.

Sorry for the possibly stupid question but can I double check my understanding of that option ? Ticking this will result in it not adding the 3 rules (one for each of my WAN interfaces - ipv4 and ipv6 on the main one an just ipv4 on the backup) that say "if the source is the WAN interface and the destination is not the WAN network then send the traffic to the WAN Gateway". This means that it will just use the system routing table only, other than not forcing all packets from the Backup_WAN interface to go via the Backup_WAN gateway are there any other downsides ?

fichtner commented 2 weeks ago

In multi-WAN you want this behaviour in general so it's harder to switch the rules off although it only pertains to outgoing connections (and maybe outbound NAT). It really depends on your setup/requirements. For gateway monitoring you have a route ensuring that the right interface will be used on top of that either way.

What version are you on?

Cheers, Franco

LucidityCrash commented 2 weeks ago

24.7.5

fichtner commented 2 weeks ago

Hmm, I think 24.7.5 should patch just fine :)

LucidityCrash commented 2 weeks ago

Applied the patch ... as you say a 1 line difference ... didn't appear to help in my case

fichtner commented 2 weeks ago

During the problematic WAN switch? The modem should reconnect and cause rc.newwanip logs to appear in the system log.

LucidityCrash commented 2 weeks ago

Log entries for a reconnect situation

<13>1 2024-11-07T15:04:25+00:00 opnsense.telinformix.com dhclient 95917 - [meta sequenceId="4267"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4268"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4269"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4270"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to XXX.XXX.XXX.XXX
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4271"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4272"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to fe80::XXXXXXXXX%igc0
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4273"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1,[])
<13>1 2024-11-07T15:04:41+00:00 opnsense.telinformix.com opnsense 99280 - [meta sequenceId="4274"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1,[]))
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4275"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4276"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: 4G
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4277"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4278"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to XXX.XXX.XXX.XXX
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4279"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4280"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to fe80::XXXXXXXXXXX%igc0
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4281"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1,[4G])
<13>1 2024-11-07T15:04:52+00:00 opnsense.telinformix.com opnsense 10667 - [meta sequenceId="4282"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1,[4G]))
<13>1 2024-11-07T15:05:26+00:00 opnsense.telinformix.com dhclient 5613 - [meta sequenceId="4283"] dhclient-script: Reason EXPIRE on vlan04 executing
<13>1 2024-11-07T15:05:26+00:00 opnsense.telinformix.com dhclient 10195 - [meta sequenceId="4284"] dhclient-script: Reason PREINIT on vlan04 executing
<13>1 2024-11-07T15:05:27+00:00 opnsense.telinformix.com kernel - - [meta sequenceId="4285"] <7>arpresolve: can't allocate llinfo for 10.165.119.142 on vlan04
<13>1 2024-11-07T15:05:27+00:00 opnsense.telinformix.com dhclient 12474 - [meta sequenceId="4286"] dhclient-script: Reason ARPSEND on vlan04 executing
<13>1 2024-11-07T15:05:27+00:00 opnsense.telinformix.com kernel - - [meta sequenceId="4287"] <7>arpresolve: can't allocate llinfo for 10.165.119.142 on vlan04
<13>1 2024-11-07T15:05:28+00:00 opnsense.telinformix.com kernel - - [meta sequenceId="4288"] <7>arpresolve: can't allocate llinfo for 10.165.119.142 on vlan04
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 14022 - [meta sequenceId="4289"] dhclient-script: Reason ARPCHECK on vlan04 executing
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 15391 - [meta sequenceId="4290"] dhclient-script: Reason BOUND on vlan04 executing
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 19312 - [meta sequenceId="4291"] dhclient-script: New IP Address (vlan04): 10.192.250.79
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 20864 - [meta sequenceId="4292"] dhclient-script: New Subnet Mask (vlan04): 255.255.255.224
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 22572 - [meta sequenceId="4293"] dhclient-script: New Broadcast Address (vlan04): 10.192.250.95
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 24329 - [meta sequenceId="4294"] dhclient-script: New Routers (vlan04): 10.192.250.80
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com dhclient 27032 - [meta sequenceId="4295"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4296"] /usr/local/etc/rc.newwanip: IP renewal starting (new: 10.192.250.79, old: 10.165.119.141, interface: opt3, device:
 vlan04, force: yes)
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4297"] /usr/local/etc/rc.newwanip: ROUTING: entering configure using opt3
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4298"] /usr/local/etc/rc.newwanip: ROUTING: ignoring down gateways: 4G
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4299"] /usr/local/etc/rc.newwanip: ROUTING: configuring inet default gateway on wan
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4300"] /usr/local/etc/rc.newwanip: ROUTING: keeping inet default route to XXX.XXX.XXX.XXX
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4301"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,[4G])
<13>1 2024-11-07T15:05:29+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4302"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,[4G]))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4303"] /usr/local/etc/rc.newwanip: IP address change detected, killing states of old ip 10.165.119.141
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4304"] /usr/local/etc/rc.newwanip: plugins_configure vpn_map (,opt3,inet)
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4305"] /usr/local/etc/rc.newwanip: plugins_configure vpn_map (execute task : ipsec_configure_do(,opt3))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4306"] /usr/local/etc/rc.newwanip: plugins_configure vpn_map (execute task : openvpn_configure_do(,opt3))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4307"] /usr/local/etc/rc.newwanip: plugins_configure vpn_map (execute task : wireguard_configure_do())
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4308"] /usr/local/etc/rc.newwanip: plugins_configure vpn (,opt3)
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4309"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (,opt3)
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4310"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,opt3))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4311"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (,opt3,inet)
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4312"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : dhcrelay_configure_map(,opt3,inet))
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4313"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : dnsmasq_configure_do())
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4314"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : miniupnpd_configure_do())
<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4315"] /usr/local/etc/rc.newwanip: miniupnpd: Starting service on interface: lan
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4316"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : ntpd_configure_do())
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4317"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : opendns_configure_do())
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4318"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : openssh_configure_do(,opt3))
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4319"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : vxlan_configure_do())
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4320"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : webgui_configure_do(,opt3))
<13>1 2024-11-07T15:05:31+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4321"] /usr/local/etc/rc.newwanip: plugins_configure newwanip_map (execute task : wireguard_sync())
<13>1 2024-11-07T15:05:59+00:00 opnsense.telinformix.com dhclient 81000 - [meta sequenceId="4322"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:05:59+00:00 opnsense.telinformix.com dhclient 81884 - [meta sequenceId="4323"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:06:29+00:00 opnsense.telinformix.com dhclient 65497 - [meta sequenceId="4324"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:06:29+00:00 opnsense.telinformix.com dhclient 67181 - [meta sequenceId="4325"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:06:59+00:00 opnsense.telinformix.com dhclient 69806 - [meta sequenceId="4326"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:06:59+00:00 opnsense.telinformix.com dhclient 71790 - [meta sequenceId="4327"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:07:29+00:00 opnsense.telinformix.com dhclient 57041 - [meta sequenceId="4328"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:07:29+00:00 opnsense.telinformix.com dhclient 58240 - [meta sequenceId="4329"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:07:59+00:00 opnsense.telinformix.com dhclient 69402 - [meta sequenceId="4330"] dhclient-script: Reason RENEW on vlan04 executing
<13>1 2024-11-07T15:07:59+00:00 opnsense.telinformix.com dhclient 69994 - [meta sequenceId="4331"] dhclient-script: Creating resolv.conf
<13>1 2024-11-07T15:08:29+00:00 opnsense.telinformix.com dhclient 50529 - [meta sequenceId="4332"] dhclient-script: Reason RENEW on vlan04 executing
fichtner commented 2 weeks ago

Well according to here it ran into the execution:

<13>1 2024-11-07T15:05:30+00:00 opnsense.telinformix.com opnsense 28656 - [meta sequenceId="4303"] /usr/local/etc/rc.newwanip: IP address change detected, killing states of old ip 10.165.119.141 Check the stuck state first like before. Does running it manually help then? # pfctl -k gateway -k 10.165.119.141 Check states again. Cheers, Franco
LucidityCrash commented 2 weeks ago

Hehe ... I literally just did that ... and yes it fixed the problem ... old state entry now removed and new one with the right route-to address is added and the monitor started working

fichtner commented 2 weeks ago

Ok, good. Make sure the change is still applied to the file and see what happens on the next reconnect. It looks like this will start working eventually :)

LucidityCrash commented 2 weeks ago

File is still the same (ie has the extra line in) ... forced a reconnect and the bad state entry is still there ... this kinda looks like a race condition. It is almost as if the interface gets a new IP, the newwanip script runs removes any existing states then a new state entry is added (I'm assuming because dpinger is restarted and starts trying to ping the monitor address) with the new wan ip as the source but the old gateway ip as the route-to.

fichtner commented 2 weeks ago

routes, monitors and filters are reloaded before the cache IP purge and this looks like the correct order. Can you add a sleep before the 3 pfctl invokes? just a simple sleep(2);

LucidityCrash commented 2 weeks ago

fraid not ... I bumped that to 10 Seconds and still no luck though it did allow me to capture this from the states table :

root@opnsense:/var/log/system # pfctl -s states -vv | grep 8\.8\.4\.4 -A3
No ALTQ support in kernel
ALTQ related functions disabled
all icmp 10.50.82.1:27599 -> 8.8.4.4:27599       0:0
   age 00:00:27, expires in 00:00:07, 25:0 pkts, 725:0 bytes, rule 104, allow-opts
   id: f337966800000000 creatorid: 829fe43d route-to: 10.50.82.2@vlan04
   origif: vlan04
--
all icmp 10.192.250.79:46510 -> 8.8.4.4:46510       0:0
   age 00:00:03, expires in 00:00:10, 3:0 pkts, 87:0 bytes, rule 104, allow-opts
   id: af38966800000000 creatorid: 829fe43d route-to: 10.50.82.2@vlan04
   origif: vlan04
LucidityCrash commented 2 weeks ago

Just wanna add, sorry for the back and forth and thank you for your amazing support in trying to resolve this problem

fichtner commented 2 weeks ago

Ah sorry we need to kill the old gateway not the old local IP (:

LucidityCrash commented 2 weeks ago

Sorry ... What does this mean with regards to how it is fixed ?

fichtner commented 2 weeks ago

I’ll propose a patch in a bit. Needs more shuffling as the gateway is not cached yet.

LucidityCrash commented 2 weeks ago

Cool ... was just worried you expected me to understand what needed to be done ... I'm not that smart 😀

fichtner commented 2 weeks ago

Neither am I, but anyway: 270aac573d9 applies on top of the other one

# opnsense-patch 270aac573d9

Cheers, Franco

LucidityCrash commented 2 weeks ago

I updated to 24.7.8 this morining and applied those 2 patches. The first time I forced a reconnection it didn't seem to work, and I removed the state entry manually. I have since forced a reconnect multiple times and it has worked each time.

Thank you so much.

fichtner commented 2 weeks ago

The first time not working is normal because it didn't cache the gateway IP yet :) This sounds great, thanks a lot for your debugging help!

LucidityCrash commented 2 weeks ago

I'll keep an eye out for it in a release ... looking at some of the threads in the forums there may be some happy users ... seemed to have hit a number of Starlink users in particular.

fichtner commented 2 weeks ago

Starlink users always have issues from past experiences. ;)

fichtner commented 2 weeks ago

Apparently #7027 isn't the same, but the overall patch looks ok to be introduced in 24.7.9 anyway. @deajan can you confirm as well?

LucidityCrash commented 2 weeks ago

Just to confirm I had an un forced reconnection last night and the new code handled it perfectly. (you never know there may have been a slight difference in the failure state compared to the manually forced reconnect)

KiNgMaR commented 1 week ago

Applied those two patches eight days ago. And while I didn't have a maintenance window to properly do some exploration, the nightly re-connect of one of the WANs has been handled perfectly since - which didn't work before, dpinger would always get stuck. Thank you @fichtner! 💪

fichtner commented 1 week ago

@KiNgMaR happy to hear, the change will hit 24.7.9 early this week :)