IPv6 Track Interface sometimes fails to assign GUAs after PPPoE reconnect

TheDom42 commented 1 year ago

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug This bug report is based upon the discussion in https://forum.opnsense.org/index.php?topic=31443.0. By now, I believe this is a bug.

When using DHCPv6 with a /56 prefix request on the WAN together with v4 PPPoE and a local interface in track interface mode, opnsense sometimes looses the prefix after a PPPoE reconnect, making it impossible for the other interfaces to generate GUAs. Currently, all the cases that I could find/observe are based on Deutsche Telekom as ISP with full Dual Stack.

Unfortunately, this issue seems to be hard to reproduce as in many cases, there are no problems. However, if you have a daily cron for a reconnect, you can probably observe the behavior about 2 times a week (I would estimate).

To Reproduce

Set the WAN exactly according to https://docs.opnsense.org/manual/how-tos/ipv6_dsl.html with a /56 prefix.
Set one (or multiple) interfaces to "Track Interface - WAN", correctly generating GUAs for the interfaces as well as the hosts in there (everything fine so far)
Trigger a WAN reconnect (either via Overview or e.g. via a Cron)
Sometimes(!): Loose basically all IPv6 connectivity for the hosts 4.1: See that the live view FW log is filling up with default denys from the LAN (probably the old IPv6 addresses with invalid states on that interface) 4.2 See that the line with "prefix" on the WAN interface in the overview is missing (i.e. probable root cause: no prefix is assigned) 4.3 See that the Interface overview on the Dashboard only shows "track6" instead of GUAs 4.4 See that the DHCPv6 service is down and cannot be restarted (probably due to not being able to generate GUAs)
Trigger another reconnect with the same settings on the WAN or reboot FW and usually regain connectivity

Expected behavior

GUAs (based on the /56 prefix on the WAN) should be assigned to the interfaces in track6 mode.

Describe alternatives you considered

As mentioned, the issue seems to be related to the fact that no prefix on the WAN is obtained after the PPPoE reconnect and therefore basically everything relying on v6 GUAs breaks in the process. Currently, it seems hard to tell if it's an Opnsense issue, but to be honest I have been running a pfsense on the same line up until a few weeks ago with no such behavior. As a workaround: maybe, it would be possible to implement a check in the newwanip script to see if a prefix has been obtained (when DHCPv6 is active on WAN and prefix request is ticked) and if not, trigger another reconnect until a prefix is assigned? I'm not sure if that is a feasible idea (e.g. I don't know if some ISPs enforce a rate limiting etc.).

Screenshots

All that can be seen is already described above.

Relevant log files So far, no really relevant log entries could be found. Here is one example. Another (by another user with the same problem can be found at https://forum.opnsense.org/index.php?topic=21682.msg148182#msg148182:

2022-12-31T03:30:23 Error   opnsense    /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog' returned exit code '255', the output was ''    
2022-12-31T03:30:23 Error   opnsense    /usr/local/etc/rc.newwanipv6: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0    
2022-12-31T03:30:23 Error   opnsense    /usr/local/etc/rc.newwanipv6: Warning! dhcpd_radvd_configure(manual) found no suitable IPv6 address on vlan011  
2022-12-31T03:30:23 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'   
2022-12-31T03:30:23 Error   opnsense    /usr/local/etc/rc.newwanipv6: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0    
2022-12-31T03:30:23 Error   opnsense    /usr/local/etc/rc.newwanipv6: Warning! dhcpd_radvd_configure(manual) found no suitable IPv6 address on vlan011  
2022-12-31T03:30:08 Notice  opnsense    plugins_configure newwanip (execute task : webgui_configure_do(,wan))   
2022-12-31T03:30:08 Notice  opnsense    plugins_configure newwanip (execute task : vxlan_configure_do())    
2022-12-31T03:30:07 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_hosts_generate(1))   
2022-12-31T03:30:07 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_resolvconf_generate(1))  
2022-12-31T03:30:07 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (1) 
2022-12-31T03:30:06 Notice  dhcp6c      dhcp6c RELEASE on pppoe0 - running dns reload   
2022-12-31T03:30:06 Notice  opnsense    plugins_configure newwanip (execute task : unbound_configure_do(,wan))  
2022-12-31T03:30:06 Notice  opnsense    plugins_configure newwanip (execute task : openssh_configure_do(,wan))  
2022-12-31T03:30:06 Notice  opnsense    plugins_configure newwanip (execute task : opendns_configure_do())  
2022-12-31T03:30:06 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do()) 
2022-12-31T03:30:06 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())  
2022-12-31T03:30:06 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (,wan)   
2022-12-31T03:30:06 Error   opnsense    /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.  
2022-12-31T03:30:06 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,wan))   
2022-12-31T03:30:06 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,wan)) 
2022-12-31T03:30:06 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (,wan)    
2022-12-31T03:30:06 Error   opnsense    /usr/local/etc/rc.newwanip: IP address renew, killing all previous states   
2022-12-31T03:30:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))  
2022-12-31T03:30:05 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)   
2022-12-31T03:30:05 Error   opnsense    /usr/local/etc/rc.newwanipv6: On (IP address: XXXX:XX:XXXX:XXXX:XXXX:XXX:XXXX:XXXX) (interface: WAN[wan]) (real interface: pppoe0). 
2022-12-31T03:30:05 Error   opnsense    /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'  
2022-12-31T03:30:05 Notice  dhcp6c      dhcp6c REQUEST on pppoe0 - running newipv6  
2022-12-31T03:30:05 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_run return_gateways_status (execute task : dpinger_status())    
2022-12-31T03:30:05 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_run return_gateways_status ()   
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: Adding static route for monitor 8.8.8.8 via xx.xxx.xxx.xxx  
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: Removing static route for monitor 8.8.8.8 via xx.xxx.xxx.xxx    
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: Chose to bind WAN_PPPOE on xxx.xxx.xxx.xxx since we could not find a proper match.  
2022-12-31T03:30:04 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_PPPOE)) 
2022-12-31T03:30:04 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_PPPOE)  
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: The WAN_DHCP6 monitor address is empty, skipping.   
2022-12-31T03:30:04 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6)) 
2022-12-31T03:30:04 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_DHCP6)  
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'feXX::XXXX:XXXX:XXXX:XXXX%pppoe0' 
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to feXX::XXXX:XXXX:XXXX:XXXX    
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to wan    
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'xx.xxx.xxx.xxx'   
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to xx.xxx.xxx.xxx   
2022-12-31T03:30:04 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan    
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))  
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)   
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan' 
2022-12-31T03:30:03 Notice  dhcp6c      RTSOLD script - Sending SIGHUP to dhcp6c    
2022-12-31T03:30:03 Notice  dhcp6c      RTSOLD script - Sending SIGHUP to dhcp6c    
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))  
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure dhcp (1)   
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure ipsec (execute task : ipsec_configure_do(1,wan))   
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure ipsec (1,wan)  
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.newwanipv6: On (IP address: XXXX:XX:XXXX:XXXX:XXXX:XXX:XXXX:XXXX) (interface: WAN[wan]) (real interface: pppoe0). 
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.configure_interface: Adding static route for monitor 8.8.8.8 via xx.xxx.xxx.xxx   
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.configure_interface: Removing static route for monitor 8.8.8.8 via xx.xxx.xxx.xxx 
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.configure_interface: Chose to bind WAN_PPPOE on xxx.xxx.xxx.xxx since we could not find a proper match.   
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'  
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WAN_PPPOE)) 
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WAN_PPPOE)  
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.configure_interface: The WAN_DHCP6 monitor address is empty, skipping.    
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.newwanip: On (IP address: xxx.xxx.xxx.xxx) (interface: WAN[wan]) (real interface: pppoe0).    
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'pppoe0'    
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WAN_DHCP6)) 
2022-12-31T03:30:03 Notice  opnsense    /usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WAN_DHCP6)  
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.configure_interface: ROUTING: skipping IPv6 default route 
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.configure_interface: ROUTING: IPv6 default gateway set to wan 
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.configure_interface: ROUTING: skipping IPv4 default route 
2022-12-31T03:30:03 Error   opnsense    /usr/local/etc/rc.configure_interface: ROUTING: IPv4 default gateway set to opt19   
2022-12-31T03:30:02 Error   opnsense    /usr/local/etc/rc.configure_interface: ROUTING: entering configure using 'wan'

Additional context

No other context available.

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 22.7.10_2-amd64 (but also observed at least with 22.7.9) FreeBSD 13.1-RELEASE-p5 OpenSSL 1.1.1s 1 Nov 2022

Intel® NUC9i5QNX Network Intel® I210-AT

TheDom42 commented 1 year ago

We have now more cases with logs (e.g. https://forum.opnsense.org/index.php?topic=21682.msg152240#msg152240) and judging by the upvotes for this issue, I believe this is not an isolated single case. Also, in the last few days, it has gotten worse, where basically every nightly Cron PPPoE reset triggers those issues. As a workaround, I have now turned off tracking for the LAN interfaces (yes, I could also just turn off my Cron but the issue would still remain if the WAN link would be reset some other way - e.g. from the ISP, power loss etc., so this would not eleminate the root issue). Nevertheless, missing v6 connectivity for all clients in this subnet with the recommended standard configuration is problematic nowadays. How should we proceed? How could this be investigated?

fichtner commented 1 year ago

Happy new year!

PPPoE has always been a bit challenging. I've added https://github.com/opnsense/core/issues/6204 for 23.1 which by default prevents the ISP to give out additional IPv6 addressing which could throw off the system, but it's pretty specific to how the ISP does things.

I can help out here but for the time being I need to ensure that 23.1 will go out end of the month which binds a lot of resources in January.

Cheers, Franco

TheDom42 commented 1 year ago

Thank you for the update! Just for clarification: does #6204 in your eyes help with the situation described above? In any case, I will monitor the situation (also after the update). Thank you for having a look at it! :) Also happy new year to you!

fichtner commented 1 year ago

It really depends on the ISP. The patch merely prevents two IPv6 modes fighting for the primary address. It‘s a problem with no clear winner and a fixed PPPoE supplied IPv6 might make the system think it doesn’t have to reload anything else and/or set the wrong router address.

if you want I can provide a backport tomorrow. I think there were two commits necessary. It’s a rather harmless change to test.

Cheers, Franco

thumax commented 1 year ago

Maybe this is helpful to some affected by this issue: manually triggering a DHCPv6 reload on the WAN interface (via Interfaces – Overview) appears to kick delegation back into action and is often less trouble than a PPPoE reconnect.

fichtner commented 1 year ago

@thumax truth be told it will reload IPv4 as well in that case, see https://github.com/opnsense/core/commit/b33ed9e20702

thumax commented 1 year ago

Looks like I found myself a homeopathic workaround then. 😁 Thanks for the hint.

EuleMitKeule commented 1 year ago

I believe I have the same problem. My ISP also is Deutsche Telekom.

My internet connection sometimes drops multiple times a day, but most often between 3-4 AM. The only thing that works to get the internet connection back up is to restart the VDSL modem (Speedport Smart 3).

While in the faulty state the only relevant log I get is this one again and again: Error dhcp6c transmit failed: Can't assign requested address.

When I restart the firewall while it is in the faulty state, the DHCP6 service is not able to start, but is not logging anything. After restart the modem, I get a a bunch of errors, but after a few minutes everything works again.

lhooge commented 1 year ago

I'm seeing the same issue, after IPv6s renewal my IPv6 connection is mostly gone.

My ISP is a local one but they cooperate with Deutsche Telekom. I'm sitting behind DS Lite.

My workaround is currently to reboot Opnsense at night by cron. Unfortunately the workaround to just reload DHCPv6 on WAN did not help.

Current version FreeBSD 13.1-RELEASE-p5 stable/22.7-n250270-9d1c26e8548 SMP amd64 OPNsense 22.7.10_2 bdce9463f Plugins os-theme-cicada-1.31 os-theme-rebellion-1.8.8 os-theme-vicuna-1.43 os-wireguard-1.13_3 os-wol-2.4_1

I also observed the problem with earlier versions

2023-01-11T19:56:26 Error   php /usr/local/opnsense/scripts/dhcp/prefixes.php: The command '/sbin/route add -inet6 'xxxx:xxxx:xxxx:xxxx::/62' 'xxxx:xxxx:xxxx:xxxx::xxxx'' returned exit code '1', the output was 'route: writing to routing socket: Network is unreachable add net xxxx:xxxx:xxxx:xxxx::/62: gateway xxxx:xxxx:xxxx:xxxx::xxxx fib 0: Network is unreachable'  
2023-01-11T19:52:05 Error   php /usr/local/opnsense/scripts/dhcp/prefixes.php: The command '/sbin/route add -inet6 'xxxx:xxxx:xxxx:xxxx::/62' 'xxxx:xxxx:xxxx:xxxx::xxxx'' returned exit code '1', the output was 'route: writing to routing socket: Network is unreachable add net xxxx:xxxx:xxxx:xxxx::/62: gateway xxxx:xxxx:xxxx:xxxx::xxxx fib 0: Network is unreachable'  
2023-01-11T19:50:04 Error   php /usr/local/opnsense/scripts/dhcp/prefixes.php: The command '/sbin/route add -inet6 'xxxx:xxxx:xxxx:xxxx::/62' 'xxxx:xxxx:xxxx:xxxx::xxxx'' returned exit code '1', the output was 'route: writing to routing socket: Network is unreachable add net xxxx:xxxx:xxxx:xxxx::/62: gateway xxxx:xxxx:xxxx:xxxx::xxxx fib 0: Network is unreachable'  
2023-01-11T18:07:17 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,wan)) 
2023-01-11T18:07:17 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())  
2023-01-11T18:07:16 Error   opnsense    /usr/local/etc/rc.newwanipv6: warning: ignoring missing default tunable request: debug.pfftpproxy   
2023-01-11T18:07:12 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,wan))    
2023-01-11T18:07:12 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,wan))    
2023-01-11T18:07:12 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())    
2023-01-11T18:07:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())   
2023-01-11T18:07:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())    
2023-01-11T18:07:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,wan) 
2023-01-11T18:07:08 Error   opnsense    /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.    
2023-01-11T18:07:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,wan)) 
2023-01-11T18:07:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,wan))   
2023-01-11T18:07:08 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,wan)  
2023-01-11T18:07:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_run return_gateways_status (execute task : dpinger_status())  
2023-01-11T18:07:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_run return_gateways_status () 
2023-01-11T18:07:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())   
2023-01-11T18:07:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())    
2023-01-11T18:07:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,wan) 
2023-01-11T18:07:03 Error   opnsense    /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.    
2023-01-11T18:07:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,wan)) 
2023-01-11T18:07:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,wan))   
2023-01-11T18:07:03 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,wan)  
2023-01-11T18:07:02 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,wan)) 
2023-01-11T18:07:02 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())  
2023-01-11T18:07:00 Error   opnsense    /usr/local/etc/rc.newwanipv6: warning: ignoring missing default tunable request: debug.pfftpproxy   
2023-01-11T18:06:59 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_run return_gateways_status (execute task : dpinger_status())  
2023-01-11T18:06:59 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_run return_gateways_status () 
2023-01-11T18:06:59 Error   opnsense    /usr/local/etc/rc.newwanipv6: The WANv6 monitor address is empty, skipping. 
2023-01-11T18:06:59 Error   opnsense    /usr/local/etc/rc.newwanipv6: The WANv6 monitor address is empty, skipping. 
2023-01-11T18:06:59 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WANv6))   
2023-01-11T18:06:59 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WANv6)    
2023-01-11T18:06:59 Error   opnsense    /usr/local/etc/rc.newwanipv6: The WAN_PPPOE monitor address is empty, skipping. 
2023-01-11T18:06:59 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WANv6))   
2023-01-11T18:06:59 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WANv6)    
2023-01-11T18:06:59 Error   opnsense    /usr/local/etc/rc.newwanipv6: The WAN_PPPOE monitor address is empty, skipping. 
2023-01-11T18:06:59 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_PPPOE))   
2023-01-11T18:06:59 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_PPPOE)    
2023-01-11T18:06:59 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway 'xxxx::xxxx:xxxx:xxxx:xxxx%pppoe0'   
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv6 default route to xxxx::xxxx:xxxx:xxxx:xxxx  
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: IPv6 default gateway set to wan  
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway 'xx.xxx.xx.xx'   
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv4 default route to xx.xxx.xx.xx   
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: IPv4 default gateway set to wan  
2023-01-11T18:06:58 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_PPPOE))   
2023-01-11T18:06:58 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_PPPOE)    
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway 'xxxx::xxxx:xxxx:xxxx:xxxx%pppoe0'   
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv6 default route to xxxx::xxxx:xxxx:xxxx:xxxx  
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: IPv6 default gateway set to wan  
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway 'xx.xxx.xx.xx'   
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv4 default route to xx.xxx.xx.xx   
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: IPv4 default gateway set to wan  
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'   
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'   
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog' returned exit code '255', the output was ''    
2023-01-11T18:06:58 Error   opnsense    /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 55743'   
2023-01-11T18:06:56 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,wan))    
2023-01-11T18:06:56 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,wan))    
2023-01-11T18:06:56 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())    
2023-01-11T18:06:50 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,wan))   
2023-01-11T18:06:50 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())    
2023-01-11T18:06:48 Error   opnsense    /usr/local/etc/rc.newwanip: warning: ignoring missing default tunable request: debug.pfftpproxy 
2023-01-11T18:06:47 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())   
2023-01-11T18:06:47 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())    
2023-01-11T18:06:47 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,wan) 
2023-01-11T18:06:47 Error   opnsense    /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.    
2023-01-11T18:06:47 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,wan)) 
2023-01-11T18:06:47 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,wan))   
2023-01-11T18:06:47 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,wan)  
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_run return_gateways_status (execute task : dpinger_status())  
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_run return_gateways_status () 
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,wan))  
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,wan))  
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())  
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do()) 
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())  
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure newwanip (,wan)   
2023-01-11T18:06:40 Error   opnsense    /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.  
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,wan))   
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,wan)) 
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure vpn (,wan)    
2023-01-11T18:06:40 Error   opnsense    /usr/local/etc/rc.newwanip: IP address change detected, killing states of old ip xx.xxx.xxx.xxx 
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))  
2023-01-11T18:06:40 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)   
2023-01-11T18:06:40 Error   opnsense    /usr/local/etc/rc.newwanipv6: On (IP address: xxxx:xxxx:xxx:xxxx:xxx:xxxx:xxxx:xxxx) (interface: WAN[wan]) (real interface: pppoe0).    
2023-01-11T18:06:39 Error   opnsense    /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'  
2023-01-11T18:06:39 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_hosts_generate(1))   
2023-01-11T18:06:39 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_resolvconf_generate(1))  
2023-01-11T18:06:39 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (1) 
2023-01-11T18:06:38 Notice  dhcp6c  dhcp6c REQUEST on pppoe0 - running newipv6  
2023-01-11T18:06:38 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_hosts_generate(1))   
2023-01-11T18:06:38 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_resolvconf_generate(1))  
2023-01-11T18:06:38 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (1) 
2023-01-11T18:06:38 Error   opnsense    /usr/local/etc/rc.newwanipv6: The WANv6 monitor address is empty, skipping. 
2023-01-11T18:06:38 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WANv6))   
2023-01-11T18:06:38 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WANv6)    
2023-01-11T18:06:38 Error   opnsense    /usr/local/etc/rc.newwanipv6: The WAN_PPPOE monitor address is empty, skipping. 
2023-01-11T18:06:37 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_PPPOE))   
2023-01-11T18:06:37 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_PPPOE)    
2023-01-11T18:06:37 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway 'xxxx::xxxx:xxxx:xxxx:xxxx%pppoe0'   
2023-01-11T18:06:37 Notice  dhcp6c  dhcp6c RELEASE on pppoe0 - running dns reload   
2023-01-11T18:06:37 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv6 default route to xxxx::xxxx:xxxx:xxxx:xxxx  
2023-01-11T18:06:37 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: IPv6 default gateway set to wan  
2023-01-11T18:06:37 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway 'xx.xxx.xx.xx'   
2023-01-11T18:06:37 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv4 default route to xx.xxx.xx.xx   
2023-01-11T18:06:37 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: IPv4 default gateway set to wan  
2023-01-11T18:06:37 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))  
2023-01-11T18:06:37 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)   
2023-01-11T18:06:37 Error   opnsense    /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'   
2023-01-11T18:06:37 Error   opnsense    /usr/local/etc/rc.newwanipv6: On (IP address: xxxx:xxxx:xxxx:xxxx:xxx:xxxx:xxxx:xxxx) (interface: WAN[wan]) (real interface: pppoe0).   
2023-01-11T18:06:37 Notice  dhcp6c  RTSOLD script - Sending SIGHUP to dhcp6c    
2023-01-11T18:06:37 Error   opnsense    /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'  
2023-01-11T18:06:37 Notice  dhcp6c  dhcp6c RELEASE on pppoe0 - running dns reload   
2023-01-11T18:06:36 Notice  dhcp6c  dhcp6c REQUEST on pppoe0 - running newipv6  
2023-01-11T18:06:36 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_run return_gateways_status (execute task : dpinger_status())    
2023-01-11T18:06:36 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_run return_gateways_status ()   
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: The WANv6 monitor address is empty, skipping.   
2023-01-11T18:06:35 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WANv6)) 
2023-01-11T18:06:35 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (,WANv6)  
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: The WAN_PPPOE monitor address is empty, skipping.   
2023-01-11T18:06:35 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_PPPOE)) 
2023-01-11T18:06:35 Notice  opnsense    /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_PPPOE)  
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: creating /tmp/pppoe0_defaultgwv6 using 'xxxx::xxxx:xxxx:xxxx:xxxx%pppoe0'  
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: removing /tmp/pppoe0_defaultgwv6   
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to xxxx::xxxx:xxxx:xxxx:xxxx    
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to wan    
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'xx.xxx.xx.xx' 
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to xx.xxx.xx.xx 
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan    
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan' 
2023-01-11T18:06:35 Error   dhcp6c  transmit failed: Can't assign requested address 
2023-01-11T18:06:35 Notice  dhcp6c  RTSOLD script - Sending SIGHUP to dhcp6c    
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: On (IP address: xx.xxx.xxx.xxx) (interface: WAN[wan]) (real interface: pppoe0). 
2023-01-11T18:06:35 Error   opnsense    /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'pppoe0'    
2023-01-11T18:06:34 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))  
2023-01-11T18:06:34 Notice  opnsense    /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)   
2023-01-11T18:06:34 Error   opnsense    /usr/local/etc/rc.newwanipv6: On (IP address: xxxx:xxxx:xxxx:xxxx:xx:xxxx:xxxx:xxxx) (interface: WAN[wan]) (real interface: pppoe0).    
2023-01-11T18:06:34 Error   opnsense    /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'  
2023-01-11T18:06:29 Error   dhcp6c  transmit failed: Can't assign requested address 
2023-01-11T18:06:28 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_hosts_generate(1))   
2023-01-11T18:06:28 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_resolvconf_generate(1))  
2023-01-11T18:06:28 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (1) 
2023-01-11T18:06:28 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_hosts_generate(1))   
2023-01-11T18:06:28 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_resolvconf_generate(1))  
2023-01-11T18:06:28 Notice  opnsense    /usr/local/sbin/pluginctl: plugins_configure dns_reload (1)

EuleMitKeule commented 1 year ago

Restarting the firewall or reloading DHCPv6 or PPPoE on the Interfaces->Overview->WAN page does not help for me. The only thing that works is to restart the modem. Also in the overview screen it says that DHCPv6 is up but PPPoE is down while in the faulty state.

fichtner commented 1 year ago

So did someone try the development version of 22.7 to confirm the problem? Because newwanip scripts got rewritten a bit to better detect a real renew event vs. just a new event with the same IP. Also #6204 could play a role. There is a patch to try that I think is not on the latest 22.7.10 development version yet.

Cheers, Franco

EuleMitKeule commented 1 year ago

Could this issue be related to #3240 by any chance? The solution for that person apparently was to turn on Prevent Release in Interface->Settings->IPv6 DHCP. I enabled that now, maybe that fixes my problem. If not I will try the development version.

fichtner commented 1 year ago

Prevent release helps in some cases e.g. when the lease time actually exceeds the ISP mandatory reconnect interval (which is silly really).

EuleMitKeule commented 1 year ago

I also just found this forum thread (german language) that sounds very similar and the solution over there also was to enable prevent release as well as Firewall->Settings->Advanced->Dynamic state reset. Edited

fichtner commented 1 year ago

You probably mean firewall category, not interfaces. Though on 23.1 Dynamic state reset will be removed. It doesn't work the way it was intended and better methods have been introduced which are the default.

Cheers, Franco

fichtner commented 1 year ago

PS: Frankly, to not run into historic quirks it would be useful to try the development version if you can.

EuleMitKeule commented 1 year ago

After switching to development it seems to be working now. I unplugged the DSL from my modem, waited a few seconds and replugged and the connection reestablished itself after a few minutes. I still took OPNsense quite some time after my modem's LED indicated successful DSL connection, but at least I don't have to power cycle or manually intervene anymore. I will observe if this will survive the next ISP reset though. Thank you @fichtner!

Edit: The logs also changed quite a bit:


2023-01-12T09:12:55 | Warning | opnsense-devel | /usr/local/etc/rc.newwanip: Failed to detect IP for WAN[wan]
2023-01-12T09:15:45 | Error | opnsense-devel | /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p  '/var/run/dhcpleases6.pid'  '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3',  the output was 'daemon: process already running, pid: 97789'
2023-01-12T09:15:45 | Error | opnsense-devel | /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p  '/var/run/dhcpleases6.pid'  '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3',  the output was 'daemon: process already running, pid: 97789'
2023-01-12T09:15:45 | Error | opnsense-devel | /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p  '/var/run/dhcpleases6.pid'  '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3',  the output was 'daemon: process already running, pid: 97789'
2023-01-12T09:15:45 | Error | opnsense-devel | /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/radvd -p  /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog' returned exit code  '255', the output was ''
2023-01-12T09:15:46 | Error | opnsense-devel | /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/radvd -p  /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog' returned exit code  '255', the output was ''
2023-01-12T09:15:46 | Error | opnsense-devel | /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/radvd -p  /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog' returned exit code  '255', the output was ''
2023-01-12T09:15:53 | Error | opnsense-devel | /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/ntpd -g -c  '/var/etc/ntpd.conf'' returned exit code '1', the output was ''

fichtner commented 1 year ago

I wouldn't bet on this just yet, but I wholeheartedly thank you for trying the work that was done in the last couple of months on such connectivity issues ❤️

Fingers crossed!

fichtner commented 1 year ago

Yikes, you have 3 tacking interfaces set up perhaps? Related to https://github.com/opnsense/core/issues/5966 I think.

EuleMitKeule commented 1 year ago

Uhm, I don't think so. I have OPT1-3 enabled, but they don't have tracking activated. I also have a VLAN tag 100 interface that I use with my managed switch and the VLAN tag 7 for my WAN/PPPoE. Nothing else. (I don't have much experience with all of this so I am not really sure what tracking even means)

Edit: LAN has tracking enabled of course.

fichtner commented 1 year ago

Yea, I meant "Track interface" IPv6 option being used in three places just because the error repeats three times, which is the amount of times #5966 may force a reload worst case. Not entirely sure from the log given. There should be the rest of it under "notice" log level now as all log messages gained an appropriate level now...

EuleMitKeule commented 1 year ago

system.log Track interface option was definetely only enabled for LAN, but to be sure I completeley disabled OPT1-3 now, since I don't use them anyway. grafik grafik grafik grafik grafik

lhooge commented 1 year ago

hmm, I tried the devel version 23.1.b_151, but after updating I do not get any IPv6 address anymore.

Unfortunately I haven't gained very much knowledge about IPv6 so far... I will try to investigate at the weekend. I've downgraded the version and the IPv6 connection is working.

The system log from 23.1.b_151:

2023-01-12T19:57:15 Error   dhcp6c  transmit failed: Network is down    
2023-01-12T19:57:13 Error   dhcp6c  transmit failed: Network is down    
2023-01-12T19:57:12 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: plugins_run return_gateways_status (execute task : dpinger_status())    
2023-01-12T19:57:12 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: plugins_run return_gateways_status ()   
2023-01-12T19:57:12 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WANv6)) 
2023-01-12T19:57:12 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: plugins_configure monitor (,WANv6)  
2023-01-12T19:57:12 Error   dhcp6c  transmit failed: Network is down    
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_PPPOE)) 
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_PPPOE)  
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route    
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to wan    
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'xx.xxx.xx.xxx'    
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: ROUTING: treating 'xx.xxx.xx.xxx' as far gateway for 'xx.xxx.xx.xx/32'  
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to xx.xxx.xx.xxx
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan    
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan' 
2023-01-12T19:57:11 Notice  dhcp6c  RTSOLD script - Sending SIGHUP to dhcp6c    
2023-01-12T19:57:11 Notice  opnsense-devel  /usr/local/etc/rc.newwanip: IP renwal starting (new: xx.xxx.xx.xx, old: , interface: WAN[wan], device: pppoe0)  
2023-01-12T19:57:10 Warning opnsense-devel  /usr/local/etc/rc.configure_interface: warning: ignoring missing default tunable request: debug.pfftpproxy  
2023-01-12T19:57:08 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure dns (execute task : unbound_configure_do(1))   
2023-01-12T19:57:08 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure dns (execute task : dnsmasq_configure_do(1))   
2023-01-12T19:57:08 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure dns (1)    
2023-01-12T19:57:08 Warning opnsense-devel  /usr/local/etc/rc.configure_interface: dhcpd_radvd_configure(auto) found no suitable IPv6 address on igb2_vlan30    
2023-01-12T19:57:08 Warning opnsense-devel  /usr/local/etc/rc.configure_interface: dhcpd_radvd_configure(auto) found no suitable IPv6 address on igb2_vlan20    
2023-01-12T19:57:08 Warning opnsense-devel  /usr/local/etc/rc.configure_interface: dhcpd_radvd_configure(auto) found no suitable IPv6 address on igb2_vlan10    
2023-01-12T19:57:08 Warning opnsense-devel  /usr/local/etc/rc.configure_interface: dhcpd_radvd_configure(auto) found no suitable IPv6 address on igb2_vlan40    
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))  
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure dhcp (1)   
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure ipsec (execute task : ipsec_configure_do(1,wan))   
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure ipsec (1,wan)  
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WAN_PPPOE)) 
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WAN_PPPOE)  
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WANv6)) 
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WANv6)  
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: ROUTING: skipping IPv6 default route 
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: ROUTING: IPv6 default gateway set to wan 
2023-01-12T19:57:07 Notice  opnsense-devel  /usr/local/etc/rc.configure_interface: ROUTING: entering configure using 'wan'

Earlier I see:

2023-01-12T19:42:33 | Error | opnsense-devel | /usr/local/etc/rc.bootup: The command `/sbin/ifconfig 'igb2_vlan30'  vlandev 'igb2' vlan '30' vlanpcp '0' vlanproto '802.1q'' failed to  execute

Could be resolved by db2bcf7 and it fails to set the correct vlan on wan interface?

EuleMitKeule commented 1 year ago

@fichtner Unfortunately tonight and again this morning I experienced two consecutive internet outages and had to restart the modem. Problem not solved for me. I will try ordering a new Vigor modem, maybe i just got a broken unit (I bought it used).

DerDanilo commented 1 year ago

We also have this problem with multiple setups. ONT --> PPPoe v4 and v6 prefix (/56) via DHCPv6--> OPNsense . We are also forced to reconnect once every 24h and receive a new IPv4 and v6 suffix.

To control when the reconnect should be triggered we setup a cron job to reset the PPPoe and WAN interface at 3 AM. IPv4 is no problem, but the IPv6 prefix is not requested by OPNsense during the interface reset.

Pressing the button "Reload" for DHCPv6 on the Interface Overview for PPPoe immediately requests a new /56 and v6 is working again.

The LAN interface tracking to receive a /64 subnet out of the /56 prefix usually works fine if the PPPoe requests a new /56. Sometimes it also fails and either the interface has to be edited and saved of the FW has to be rebooted.

This leaves us with multiple problems:

complicated setup for pppoe forced reconnect control
DHCPv6 for pppoe not working properly
LAN tracking interface not always working
if DHCPv6 for the pppoe interface can be somehow controlled via cron there is about a minute between the pppoe v4 reconnect and the request for a new v6 suffix (& IP for FW if configured). During that time IPv6 does not work and all devices loose connections to v6 targets for more than a minute (until SLAAC or the local DHCPv6 server kicks in once the new suffix is received)

Why are there no options in the PPPoe configuration page to control when a forced reconnect should occure? I've read multiple threads regarding this isssue and I don't understand why it's not addressed properly? PPPoe is still common, even with fiber providers (for whatever reason). People are litterally switching to other (commercial) products because OPNsense (and apparently pfsense as well) is not handling this problem properly for years. This is very sad.

The following optional options in the pppoe config page make sense in my opinion:

"when to trigger post pppoe reconnect" (both option can be active at the same time):

force pppoe reconnect at MM:HH every day or via cronstyle schedule and trigger "post ppoe reconnect" tasks
watch pppoe v4 IP and trigger "post pppoe reconnect" when changed

The following option make not only sense when the reconnect is triggered but should be also be triggered when pppoe receives a new v4 address.

"post pppoe reconnect / new pppoe ip": (order should be fixed since they depend on each other)

request a new IPv6 prefix (& IP for FW if configured) after pppoe is up again --> DHCP (client) reload
force configured v6 tracking LAN interfaces to track the new v6 prefix (to make sure it knows about the new suffix)
trigger SLAAC/radv restart (to make sure it knows about the new prefix)
trigger local DHCPv6 server restart (to make sure it knows about the new prefix)

Most of these options should be possible to implement quiet easy since nearby all actions are already possible somehow within the GUI. If there is a GUI option there is a command to be triggered via other means. The only one where I couldn't find a button for is to force the LAN interface tracking.

I hope that this somehow makes sense and that we have a solution at hand within version 23.x.

Thanks!

AdSchellevis commented 1 year ago

.... Why are there no options in the PPPoe configuration page to control when a forced reconnect should occure? ....

For the same reason any other interface type doesn't have a planned "reboot" time, you configure how it should connect, broken provider setups are edge cases, which you likely already can deal with using the cron jobs available..

The command most likely to already offer a reconnect is "interface reconfigure" and can be run from the console or scheduled via the cron system integrated in the gui, e.g:

configctl interface reconfigure wan

(replace wan for whatever interface you do want to reconnect)

Which is probably what you're using from the gui now as well:

https://github.com/opnsense/core/blob/b33ed9e20702fef8b95e3816fe367fb2e982ae10/src/www/status_interfaces.php#L38-L40

Unfortunately PPPoE isn't the greatest technology in the world and providers and certainly has downsides on FreeBSD, but complaining about "others" not handling your provider issues sounds a bit silly and sad too to be honest.

DerDanilo commented 1 year ago

Thanks for your reply. I agree that PPPoe is not the greatest technology in the world and also mentioned this kind of:

PPPoe is still common, even with fiber providers (for whatever reason)

The thing is that providers that trigger a PPPoe reconnect are not "broken". It is not an edge case, at least not here in Germany! They do it for privacy reasons to assign a new IPv4 address and a new IPv6 prefix. Yes sure one might declare this as broken but this is quiet common especially in Germany but I guess elsewhere as well. I don't want to discuss about the reasons they do it or if it makes any sense at all but am looking to find a solution for this. I would be happy if they would just assign an IP and a prefix until the connection is interrupted but they don't.

I don't know any reason that other interface types than PPPoe should have a "reboot time" because they don't need them. But PPPoe does require such options.

To control when the reconnect should be triggered we setup a cron job to reset the PPPoe and WAN interface at 3 AM. IPv4 is no problem, but the IPv6 prefix is not requested by OPNsense during the interface reset.

We already use the cron job to reset the interface(s). But unfortunately this is not enough as written in my previous comment. There doesn't seem to be a configctl command to trigger a DHCP client request for the PPPoe IPv6 prefix.

I put time and effort into a lot of testing, thinking about a way to handle this kind of setup as proper as possible. I am sad that this just got more or less the same answer as I've read in multiple posts while searching for a solution. "your provider is broken, we won't implement a solution to this". I repeat, it is not an edge case! I did not call anybody silly but don't understand why there is no way to properly handle pppoe connections who have this "broken" setup as you called it. Other router brands have build in options to control when the reconnect should happen and also have mechanisms to handle the v6 prefix/IP config properly. Hence I don't understand why it seems that there is such a big discussion about wether to implement some options to adress this. People are litterally forced to switch to other router brands (mostly commercial) because this awesome open source project currently has no solution for this. This is sad, as said before!

Either we work together to find a solution for this or many people will keep ending up frustrated because these setups cannot be handled by OPNsense.

Thanks in advance, have a nice weekend!

fichtner commented 1 year ago

Please don’t take my previous engagement here as a reason to complain. As such, I’m going to unsubscribe and give this to the capable hands of the community.

Cheers, Franco

AdSchellevis commented 1 year ago

...Hence I don't understand why it seems that there is such a big discussion about wether to implement some options to adress this....

Probably because there's no clear specification about what the option should do and in which cases. (and maybe most importantly how one should validate this new behavior)

....I don't know any reason that other interface types than PPPoe should have a "reboot time" because they don't need them. But PPPoe does require such options....

According to which RFC?

....Either we work together to find a solution for this or many people will keep ending up frustrated because these setups cannot be handled by OPNsense....

I'll gladly leave this in the capable hands of the community as well. As soon as something matures, just open a PR for discussion.

Have a great weekend too,

Best regards,

Ad

TheDom42 commented 1 year ago

I put time and effort into a lot of testing, thinking about a way to handle this kind of setup as proper as possible. I am sad that this just got more or less the same answer as I've read in multiple posts while searching for a solution. "your provider is broken, we won't implement a solution to this". I repeat, it is not an edge case!

To be honest, I find the tone of this message also quite irritating and I can understand that people might feel like they are being attacked while spending time on investigating the issue. Also I'm not completely convinced if your problem is related to the stuff that was discussed in my inital thread. You keep referring to a v6 suffix, while in my experience it seemed like the problem would be related to prefix assignment and I'm also unsure what the post v6 hooks/Track Interface problems have to do with that - In my experience, when a v6 prefix was assigned on WAN, Track Interface was never a problem. What failed was the prefix assignment from the ISP - and I can't tell if that was on ISP or Opnsense side.

Concerning my initial issue: I have been on 23.1 for a few days now and I believe that I've not seen my initial issue reoccur on my non-important VLANs (for which I kept Track Interface on while I turned it of for my main LAN). This could be a coincidence, but I will try to monitor.

DerDanilo commented 1 year ago

In my experience, when a v6 prefix was assigned on WAN, Track Interface was never a problem. What failed was the prefix assignment from the ISP

Sorry for the confusion. The main issue is that there is no new IPv6 prefix assigned after a PPPoe reconnect. Even with the interface reset it doesn't work. When clicking the "reload" button for the DHCPv6 client on the interface overview it is assigned within seconds. In our case at least this doesn't seem related to the ISP since it's immediately assinged once a DHCPv6 client request is triggered. Because of this I suggested to add hooks/options to the pppoe configuration that will handle these steps.

TheDom42 commented 1 year ago

To be honest, I am still not conviced we are talking about the same issue here. You are referring to problems with DHCPv6 suffixes (as far as I understood it, you are running the DHCPv6 server in Opnsense to assign your GUAs). What I am describing is the fact, that apparently the PREfix from the ISP is missing in certain circumstances. AFAIK, if the prefix is missing, none of the v6 assigments for the clients (behind the FW) can work (so Track Interface as well as DHCPv6 etc.). Therefore, your DHCPv6 issue is probably some kind of symptom of the underlying problem but in my eyes, your hooks/options would change nothing for that root cause.

Maybe I misunderstood you in this case but ranting does not help. Unfortunately, I'm not skilled enough to solve the problem myself with a PR, and now unfortunately @fichtner and @AdSchellevis have withdrawn from this issue.

DerDanilo commented 1 year ago

@TheDom42 Thanks for point out that I was reffering to a suffix. Of course I meant prefix every time I wrote suffix. I just edited my comments to not confuse people reading them. The tracking issues I am reffering to are also related to the fact that there is no new IPv6 prefix assigned/requested during pppoe reconnect/interface reset. We are not using DHCPv6 in any of our OPNsense setups since SLAAC works fine.

It was never my intention to rant or attack anybody, seriously! My only concern was to find a solution to the issues at hand. It was frustating having to figure out that there seems (I might be wrong) to be no feature in OPNsense at this time to handle the current problems. At the same time I was sure that I must have configured it wrong but apparently we are not alone. I am sorry if I offended anybody. So please let's go on and find a solution together.

To test this further I would like to know how can we trigger the DHCPv6 client request from the pppoe interface overview via script or a cron job? This would already help a lot since reloading the DHCPv6 client for the pppoe interface always assigns a new v6 prefix within seconds, at least with our setups.

Pressing the button "Reload" for DHCPv6 on the Interface Overview for PPPoe immediately requests a new /56 and v6 is working again.

fichtner commented 1 year ago

This might be https://github.com/opnsense/dhcp6c/commit/db9f45927 so if anyone wants to test:

# opnsense-revert -z dhcp6c

(I recommend a reboot)

To get back to the current version:

 # opnsense-revert dhcp6c

lhooge commented 1 year ago

@fichtner Thank you for getting back

I've tested it and see this error during the renewal:

But the IPv6 connection is still working.

2023-05-20T14:57:13 | Error | opnsense | /usr/local/etc/rc.newwanipv6: The command '/sbin/mount -r -t nullfs  '/usr/local/lib/python3.9' '/var/unbound/usr/local/lib/python3.9''  returned exit code '1', the output was 'mount_nullfs:  /var/unbound/usr/local/lib/python3.9: Device busy' 

2023-05-20T14:57:12 | Notice | opnsense | /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,opt4))

I think this one is not related to this patch.

TheDom42 commented 1 year ago

This might be opnsense/dhcp6c@db9f45927 so if anyone wants to test:
# opnsense-revert -z dhcp6c
(I recommend a reboot)

To get back to the current version:
 # opnsense-revert dhcp6c

Thank you for reporting back. For me, the connection is/was still buggy under 23.1.8 (I updated to 23.1.9 today). However: after I found this comment - https://forum.opnsense.org/index.php?topic=32259.msg155936#msg155936 a few weeks ago and did as reported

It would probably be beneficial to use Interfaces: Settings: Prevent release

my IPv6 remained stable. Only after I deactivated it, the problem reappeared.

I don't know if this helps troubleshooting but at least for me, this was a stable workaround.

fichtner commented 1 year ago

The state of 23.1.9 seems good for a number of people reporting (different) problems previously. Some of this might need a reboot as well to make use of the last dhcp6c binary update (which is normally not restarted).

It might be good to keep this under observation until 23.7 is out at the end of July and work on remaining issues based on that particular version as there are still a few IPv6 things in the pipeline for it (on the current development version).

Cheers, Franco

Ramalama2 commented 1 year ago

Hey everyone, im on 23.1.11 (Latest Opnsense to this time)

I have basically the same issue, but mine is repeatable and happens every time.

like @DerDanilo said, every Customer in Germany need an nightly reset and 50% of German users use an DSL Connection. The only issue is actually, that not so much German Households are using Opnsense as their FW, since almost any Home User use AVM with their Fritz!Box Product. And Business Contracts don't need a nightly DSL Reset, they have usually an static ip4 adress and a static ipv6 prefix.

Just want to explain why not so much people are reporting this issue. Thats simply because of, it only happens to German housholds that are using Opnsense (that's maybe 1%) and the other issue is, while during resets the ipv6 prefix fails to refresh, the ipv4 connection still works perfectly... That means simply that People don't even recognize it, since the "internet" works xD

Another point is, that the "nightly resets" are not nightly anymore since 1-2 years here in germany. Means it's usually enough to reset weekly these days, before the provider either forces you (does the reset on his end) or disconnects you.

For me it happens every time and is very repeatably, every time the cronjob triggers a wan reset, every time i loose the prefix-delegation and it doesn't get renewed.

Here is the Cronjob:
Here is the missing Prefix Delegation, after the Cronjob gots executed:
As you see, the LAN Interface fails to get an IPv6 Adress, due to the missing Prefix delegation on the WAN Interface.

If i press on the WAN interface -> The Orange "Reload" Button near "DHCPv6 up" -> I get a proper Prefix Delegation again, the Lan interface gets the IPv6 Adress again and basically everything is fixed and working:

My question is actually very simple, isn't there a way, i could simply trigger that Orange "Reload" button, with an cronjob or Monit?

Thanks & Cheers

Ramalama2 commented 1 year ago

The actual solution to this issue is, simply rebooting Opnsense with a Cronjob... Thats the only way here that everything works as expected and i get every night a new ip.

asmcc commented 9 months ago

Do we have still this issue and can someone report, that it works now with these German dynamic IPV6? The diskussion under https://forum.opnsense.org/index.php?topic=26832.0 is also stopped in summer last year. And is the reboot really only the singular possible solution? A shell script and a patch was proposed in forum and if I understood it correct without a really reboot. Unfortunatelly I can not register me on the forum due to "SPAM IP address". I do not know, why a German IP is a SPAM, but that is other discussion with the admin of opnsense forum. I use OPNsence since 2-3 weeks and have such issues with tracked interfaces. As WAN interface an AVM Fritz!Box is used. My local provider changes the IPv6 address very seldom, but this address is not static. Therefore and unfortunatelly I can not "generate" such an event for test purposes. Today I replaced my old Box with I new one. And I assume, that the new mac address on DSL interface provoke my provider to give me a new IPv6 address. In my case I use my own ULA prefix (fd90-xxxx address) on the WAN side. That means, that the FritzBox "give" me two addresses: global and local one. I read somewhere, that such a practice can generate issues, since it is not guaranted, which address of these both addresses is "the first" and therefore the ULA (fd90-xxxx) is tracked instead of global address. And yes, I can confirm, that after reboot of OPNsence GUAs are correct. But is the reboot really the unique solution?

dannykorpan commented 6 months ago

Hi, the problem still exists. Are there any updates or workarounds for this?

TheDom42 commented 5 months ago

As I posted above, my workaround (which has been stable for more than a year now) was this: https://forum.opnsense.org/index.php?topic=32259.msg155936#msg155936

It would probably be beneficial to use Interfaces: Settings: Prevent release

Tbh, as there were no issues for me after I switched this setting on, I haven't paid attention to the underlying bug afterward. Apparently, it is still there based on your reports, but for me, this is rock solid.

opnsense / core

IPv6 Track Interface sometimes fails to assign GUAs after PPPoE reconnect #6223