os-wireguard 2.4_1 | missing route to/from to virtual IPs of type Proxy ARP | after upgrading OPNsense to 23.7.7

[relume]

Important notices Before you add a new report, we ask you kindly to acknowledge the following:

• [X] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
• [X] I have searched the existing issues, open and closed, and I'm convinced that mine is new.
• [X] The title contains the plugin to which this issue belongs

Describe the bug

• After upgrading OPNsense from version 23.7.5 to OPNsense to 23.7.6 and 23.7.7 (23.7.7_1/2/3) routing to/from Wireguard interfaces/networks to virtual IPs of type Proxy ARP (public IPs 1:1 to internal IPs) do not work any more. It worked on OPNsense 23.7.5 and before, even after reboots. This is the case for all configured Wireguard interfaces/networks (1 mobile/road warrior, 2 site to site). Routing to defined LAN networks do work.

• Restarting wireguard by WebGUI or cli does not enable routing again.

• We configured for testing a single route (System/Routes/Configuration) entry and adding the routing configuration form one virtual IP Proxy ARP (mypublicIP/32) and as gateway one automatically defined wireguard gateway. By applying this routing configuration, routing to/from all defined wireguard interfaces/networks come up to all defined Proxy ARP virtual IPs (not only the single Proxy ARP virtual IP just defined.

• But routing routing to/from all defined wireguard interfaces/networks to defined Proxy ARP virtual IPs does not survive any rebooting. Only reapplying the routing configuration brings up the routing again. Thus I am not shure if the issue is concerning the os-wireguard plugin or more a general routing/gatway problem.

To Reproduce

• at the moment only configuring the above described routing configuration form one virtual IP Proxy ARP (mypublicIP/32) and as gateway one automatically defined wireguard gateway and reapplying it after reboot is reproducible on our configuration.

Expected behavior

• routing routing to/from Wireguard interfaces/networks to virtual IPs of type Proxy ARP should work/come up at every start/restart of OPNsense as it was on version 23.7.5.

Screenshots

Relevant log files

• wireguard log : Error | wireguard | /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: not a valid interface gateway address: '
• general log : Error | opnsense | /usr/local/etc/rc.routing_configure: ROUTING: not a valid interface gateway address: ''
• routing log : no concerning errors

Additional context

Environment

OPNsense 23.7.7_3-amd64 FreeBSD 13.2-RELEASE-p3 OpenSSL 1.1.1w 11 Sep 2023 CPU type | QEMU Virtual CPU version 2.5+ (6 cores, 6 threads)

[OPNsense-bot]

Thank you for creating an issue. Since the ticket doesn't seem to be using one of our templates, we're marking this issue as low priority until further notice.

For more information about the policies for this repository, please read https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md for further details.

The easiest option to gain traction is to close this ticket and open a new one using one of our templates.

[fichtner]

Might work with the latest changes for 23.7.8 but first time I’m hearing this use case so not sure.

[relume]

Thank you. 23.7.8 is actually not yet available on the "normal" firmware update channel. So I can not test.

Just to clarify why our public addresses defined as virtual IP Proxy ARP should be also accessible over wireguard VPN (when connected by wireguard) and not directly over the public route: only few services are enabled to be public, other services as ssh etc. are only accessible within LAN and to our defined wireguard VPNs. Thus those single public IPs are declared in the wireguard peer configurations as "AllowedIPs".

[fichtner]

It should be https://github.com/opnsense/plugins/commit/af80514ad

feel free to test:

# opnsense-patch -c plugins af80514ad

[relume]

I applied the patch af80514ad and rebooted.

Unfortunately the patch has no effect to the described issue. Again only reapplying (entered into edit mode then clicking ok without any changes) the route entry mentioned above brought up the correct routing.

best

[AdSchellevis]

It might be better to share the full settings (replacing private data)

[relume]

Thanks. Which parts/aspects of the settings are of concrete interest ( as screenshots from WebGUI or as snippets from the actual config file)?

[AdSchellevis]

snippets from the config for the relevant parts are likely best.

[relume]

hoping that obfuscation of private data did not harm clearness. many thanks. opnsense_2.7.7_3_config_snippets_20231107.xml.zip

[AdSchellevis]

I don't think this has anything todo with wireguard to be honest, your wan is xxx.yyy.zzz.242/29, the next hop is xxx.yyy.zzz.241 according to the gateways, but has a "far gateway" property set and then fails at:

https://github.com/opnsense/core/blob/b787a35c8e05bdff3df9fcbf098a6e5f8964a3e2/src/etc/inc/system.inc#L556-L559

Which is odd for two reasons, the file supplied has a "gateway" set and likely doesn't need a host route at all. The proxyarp entries live on the wan interface, which appears to be a vtnet, also not really related to Wireguard.

[relume]

Thanks for your reponse and sorry to "bother".

• In front of our OPNsense Firewall we have an other OPNsense only routing instance with the IP xxx.yyy.zzz.241/29 for our public subnet xxx.yyy.zzz.240/29 on the same vtnet as our OPNsense Firewall - that is correct.

• When I configurated our OPNsense Firewall instance (with WAN-IP xxx.yyy.zzz.242/29) first in august 2023 (version from 2023.07.31), the only way to get the Proxy ARP virtual IPs to work, was to set for the upstream default Gateway the option "far gateway" (even it did not convince me).

• I disabled now this setting (option "far gateway") and accessibility/routing to the Proxy ARP virtual IPs came down immediately from wiregurad networks. A reboot did not bring up again routing to/from Proxy ARP virtual IPs. Before toggling the "fat gateway" option every reboot (on Version 23.7.5) routing to/from Proxy ARP virtual IPs was brougth up again for either point of origin: public, LANs and for wireguard networks (as reported).

• Reenabling the "fat gateway" option and rebooting does not bring up routing to/from Proxy ARP virtual IPs and the wireguard networks.

• Actually/now the only possibility to bring up routing to/from Proxy ARP virtual IPs and the wireguard again after a reboot is to enter in the WebGUI in the default upstream gateway settings and reapplying it without any changes. Or the same procedure (again withou any changes) with the "dummy" single routing entry to one of the Proxy ARP virtual IPs (mentioned above) even if this routing entry is actually disabled.

• Honestly my OPNsense proficiency is to "fractional" to understand why those manual WebGUI interactions that should have no effect as there are no setting changes bring up the routing to/from Proxy ARP virtual IPs and the wireguard networks, but a reboot does not (after the updates from 23.7.5 to 23.7.6 and 23.7.7...).

• I would appreciate any hint how to solve and/or to narrow down this issue. And it seems as you wrote not to be related to wireguard dirctly but affects routing to/from wireguard networks for the defined Proxy ARP virtual IPs.

[AdSchellevis]

This lies somewhat beyond community support scope for me, the use of proxy arp entries is not very common to be honest, when these proxy entries actually lie within the same network as the primary interface (as the config suggests) a normal virtual ip might be more suitable.

The forum (https://forum.opnsense.org) might be a better place to discuss network design choices.

[relume]

Thank you - I can understand your point about the support question. I thought it coud be a kind of bug as it worked up to 23.7.5 flawless until the following udates to 23.7.6 etc. By the way, my first configuration approach in august was with a "normal" virtual IP and I was not able to get it to work at all. In the forum I was told to use proxy arp virtual IPs as the only way to make work this type of configuration and it worked immediately by using proxy arp virtual IPs. best regards

[AdSchellevis]

which forum topic was that? just interested if I see something obvious in relation to your issue

[relume]

it was on the forum post Firewall | 1:1 / One-to-One NAT single IPs for multiple public single IPs where I asked about the 1:1 mapping.

[AdSchellevis]

well, if xxx.yyy.zzz.244 lies inside the wan network, an alias is more logical in my humble opinion. After adding an alias, I would try to ping from that source address using the firewall to some external address to validate if the address works. If that works, move on to the nat rules.

[relume]

thanks again.

1. I gave it a try with alias virtual IP for an additional public IP xxx.yyy.zzz.246 with an NAT One-to-One mapping to an LAN IP as the other settings mentioned previously .
2. A route was inserted automatically for this public IP xxx.yyy.zzz.246 and ping from wireguard networks started immediately to work (setting this additional public IP beforhand in the wireguard client/peer Allowed IPs setting.). Thus I changed the other proxy arp virtual IPs also to alias virtual IPs. Automatic created routes were inserted an ping from wireguard networksworked now also for those public IPs. But tcp access from wireguard networks to those public IPs xxx.yyy.zzz.244 - 246 did not work, even as I opened for testing the incoming Firewall rules fully.
3. I swichted from Manual outbound NAT rule generation to Automatic outbound NAT rule generation but no automatic rules were generated. Normal LAN to Internet connection went down, but tcp access from wireguard networks to the public IPs xxx.yyy.zzz.244 - 246 started. I switched back to Manual outbound NAT rule generation and tcp access from wireguard networks still worked.
4. After rebooting OPNsense tcp access from wireguard networks to the public IPs (244 - 246) was down again. I tried to repeat this effect from point 3. only by oppening (toggling) the default NAT outbound rule for the WAN interface with Interface-address as NAT-address and saving the rule without any change. TCP access from wireguard networks to the public IPs (244 - 246) started immediately.
5. Updated to OPNsense version 23.7.8 with reboot and repeated point 4. to get TCP access from wireguard networks.
6. As no additional or changed autogenerated firewall rules were shown/after "toggling" the default NAT outbound rule I had a look at the cli level with the command pfctl -sa to show possible differences direct after reboot and before togglin the default NAT outbound rule and also after having toggled the default NAT rule.
7. The differences before/after toggling are obvious:

by "toggling" (after reboot) OPNsense inserted the following additional NAT rules (those rules do not survive a reboot):

10.92.1.10 - 12 : internalt IPs mapped One-To-One to public IPs (alias) xxx.yyy.zzz.244/32 - 246/32

TRANSLATION RULES:
no nat proto carp all
...
nat on wg2 inet from (wg2:network) to 10.92.1.10-> (wg2) port 1024:65535 round-robin
nat on wg1 inet from (wg1:network) to 10.92.1.10 -> (wg1) port 1024:65535 round-robin
nat on wg3 inet from (wg3:network) to 10.92.1.10 -> (wg3) port 1024:65535 round-robin
...
nat on wg2 inet from (wg2:network) to 10.92.1.11-> (wg2) port 1024:65535 round-robin
nat on wg1 inet from (wg1:network) to 10.92.1.11 -> (wg1) port 1024:65535 round-robin
nat on wg3 inet from (wg3:network) to 10.92.1.11 -> (wg3) port 1024:65535 round-robin
...
etc.
...
rdr on wg2 inet from any to xxx.yyy.zzz.244 -> 10.92.1.10 bitmask
rdr on wg1 inet from any to xxx.yyy.zzz.244 -> 10.92.1.10 bitmask
rdr on wg3 inet from any to xxx.yyy.zzz.244 -> 10.92.1.10 bitmask
...
etc.
...

8. Thus I am wondering, why "toggling" the default NAT outbound rule, which has nothing to do with the One-to-One mapping and the wireguard networks, has the effect that the missing rules for the wireguard networks are inserted by OPNsense but not by default at startup and thus do not survive a reboot?

[AdSchellevis]

The missing outbound nat might be a bug introduced by https://github.com/opnsense/core/commit/6ea9d216e2a717c594f2a95568bda7366dfea4d2 , we're looking into alternatives as this is likely a side affect and the reload shouldn't be needed at that point.

[relume]

Thank you for reporting.

[relume]

Updated today to 23.7.8_1-amd64 with os-wireguard 2.5_1. Above nat and rdr rules for wg interfaces survived the installation of the update but not the following reboot.

[fichtner]

I swichted from Manual outbound NAT rule generation to Automatic outbound NAT rule generation but no automatic rules were generated.

This is expected. WireGuard VPN doesn't offer automatic rule generation because it doesn't have an (assigned) interface IPv4 configuration and that's the same for all tunnel types.

As far as #6852 goes this pertains to rc.linkdown, but shouldn't affect bootup. If after reboot this doesn't work try running:

# /usr/local/etc/rc.filter_configure

to see if that fixes your issue. The thing I'm wondering, however is why it's concluding there is no address on WireGuard interface, because that comes from the early configuration through WireGuard options. Can you provide the full wireguard log and the system log for "ifconfig" errors after bootup and probable fix through filter configure script?

[fichtner]

PS: What is your "Tunnel address" for that particular WireGuard instance?

[relume]

Thank you.

This is expected. WireGuard VPN doesn't offer automatic rule generation because it doesn't have an (assigned) interface IPv4 configuration and that's the same for all tunnel types.

I intended all types of automatic rule generation such standard outbound rules from LAN to WAN. But hopped also Wireguard VPN outbound rules would be generated automatically. Good to know.

# /usr/local/etc/rc.filter_configure

Executing manually above command brings up/adds the missing nat and rdr rules etc. after a reboot.

wireguard log | 2023.11.16 | no entries before reboot, log contains only entries after reboot (2023-11-16T16:35:49)
xxx.yyy.zzz.242 = fw.mydomain.net

<37>1 2023-11-16T16:36:11+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="1"] wireguard instance wg_mobile (wg1) can not reconfigure without stopping it first.
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="2"] wireguard instance wg_mobile (wg1) stopped
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="3"] wireguard instance wg_mobile (wg1) started
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="4"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: entering configure using 'opt2'
<35>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="5"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: not a valid interface gateway address: ''
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="6"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: plugins_configure monitor (,WG_MOBILE_GW)
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="7"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: plugins_configure monitor (execute task : dpinger_configure_do(,WG_MOBILE_GW))
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="8"] wireguard instance wg_hb_campora (wg2) can not reconfigure without stopping it first.
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="9"] wireguard instance wg_hb_campora (wg2) stopped
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="10"] wireguard instance wg_hb_campora (wg2) started
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="11"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: entering configure using 'opt4'
<35>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="12"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: not a valid interface gateway address: ''
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="13"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: plugins_configure monitor (,WG_CAMPORA_GW)
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="14"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: plugins_configure monitor (execute task : dpinger_configure_do(,WG_CAMPORA_GW))
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="15"] wireguard instance wg_prosana (wg3) can not reconfigure without stopping it first.
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="16"] wireguard instance wg_prosana (wg3) stopped
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="17"] wireguard instance wg_prosana (wg3) started
<37>1 2023-11-16T16:36:12+01:00 fw.mydomain.net wireguard 31443 - [meta sequenceId="18"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: entering configure using 'opt3'

I could not find any entry for "ifconfig" in the system log neither messages for/related to executing manually the command rc.filter_configure after login (at 2023-11-16T16:37:10) as admin by ssh :

system log | 2023.11.16 | no entries before reboot, log contains only entries after reboot (2023-11-16T16:35:49)

xxx.yyy.zzz.242 = fw.mydomain.net

<13>1 2023-11-16T16:36:18+01:00 fw.mydomain.net kernel - - [meta sequenceId="18"] <118> wg_campora (wg2) -> v4: 10.242.81.80/32
<13>1 2023-11-16T16:36:18+01:00 fw.mydomain.net kernel - - [meta sequenceId="19"] <118> wg_mobile (wg1) -> v4: 10.242.10.1/25
<13>1 2023-11-16T16:36:18+01:00 fw.mydomain.net kernel - - [meta sequenceId="20"] <118> wg_prosana (wg3) -> v4: 10.242.70.80/32
<13>1 2023-11-16T16:36:18+01:00 fw.mydomain.net kernel - - [meta sequenceId="21"] <118>
<13>1 2023-11-16T16:36:18+01:00 fw.mydomain.net kernel - - [meta sequenceId="22"] <118> HTTPS: SHA256 xxxx
<13>1 2023-11-16T16:36:18+01:00 fw.mydomain.net kernel - - [meta sequenceId="23"] <118>               xxxx
<13>1 2023-11-16T16:36:18+01:00 fw.mydomain.net kernel - - [meta sequenceId="24"] <118> SSH:   SHA256 xxxx (ECDSA)
<13>1 2023-11-16T16:36:18+01:00 fw.mydomain.net kernel - - [meta sequenceId="25"] <118> SSH:   SHA256 xxxx (ED25519)
<13>1 2023-11-16T16:36:18+01:00 fw.mydomain.net kernel - - [meta sequenceId="26"] <118> SSH:   SHA256 xxxx (RSA)
<85>1 2023-11-16T16:37:10+01:00 fw.mydomain.net sudo 47138 - [meta sequenceId="27"]    admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/bin/su

actual boot log:

2023-11-16T16:36:12+01:00 dpinger_configure_do[31443] done.
2023-11-16T16:36:12+01:00 system_routing_configure[31443] Setting up routes for opt3...
2023-11-16T16:36:12+01:00 system_routing_configure[31443] done.
2023-11-16T16:36:12+01:00 wireguard_configure_do[269] done.
2023-11-16T16:36:12+01:00 ntpd_configure_do[269] Starting NTP service...
2023-11-16T16:36:13+01:00 ntpd_configure_do[269] done.
2023-11-16T16:36:13+01:00 unbound_configure_do[269] Starting Unbound DNS...
2023-11-16T16:36:14+01:00 unbound_configure_do[269] done.
2023-11-16T16:36:14+01:00 rrd_configure[269] Generating RRD graphs...
2023-11-16T16:36:14+01:00 rrd_configure[269] done.

wireguard interfaces tunnel addresses (wg1, wg2, wg3: all wg intances have the same problem):

wg1 | wg_mobile | 10.242.10.1/25, client peers 10.242.10.10 - 10.242.10.126/32
wg2 | wg_campora | 10.242.81.80/32, client network peer 10.242.81.81/32, remote network 10.81.0.1/16
wg3 | wg_prosana | 10.242.70.80/32, client network peer 10.242.70.70/32, remote network 10.70.0.1/16

As far I can see there is no log entry that shows the difference before and after manually executing the command rc.filter_configure. May I enable some additional log option?

[relume]

I had a look also at the configd log. I seems that at boot up OPNsense/Filter generated is generated before WG instances init. But after WG init OPNsense/Unbound/core is recalled again. After manual execution of command rc.filter_configure (at 2023-11-16T16:39:08) only OPNsense/Filter is regenerated.

configd log | 2023.11.16 | reboot (2023-11-16T16:35:49)

xxx.yyy.zzz.242 = fw.mydomain.net

<13>1 2023-11-16T16:35:10+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="38257"] [2b98bad0-58d7-443f-94b4-68f9f288f002] Reboot system
<13>1 2023-11-16T16:36:07+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="1"] [e11bf910-6faf-4abe-82b5-15a30b572867] request pf current overall table record count and table-entries limit
<13>1 2023-11-16T16:36:07+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="2"] [3a4173ed-2b5a-42b9-b755-0a9009477af7] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="3"] [6a554a56-2c03-4090-8dca-4947fa4dbd5f] Linkup starting vtnet1
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="4"] [f6601ff7-c7a0-4aa8-b194-f2151ad2b69b] Linkup starting vtnet2
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="5"] [f6a85012-ebfc-42b0-b67c-d9418e0021c7] generate template OPNsense/Filter
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="6"] generate template container OPNsense/Filter
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="7"] [a7ddc1c2-a265-41fd-b7fa-d08369846657] Linkup starting vtnet0
<15>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="8"]  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
<15>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="9"]  OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="10"] [2d5263f2-2a03-49b2-acfb-0ecf2d67c664] refresh url table aliases
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="11"] [7049b0fa-ad09-459c-9a5d-e3de065aa1e4] Unbound cache flush
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="12"] [8cfd7659-50af-4462-8ad7-73552cf44e89] generate template OPNsense/WebGui
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="13"] generate template container OPNsense/WebGui
<15>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="14"]  OPNsense/WebGui generated //usr/local/etc/php.ini
<15>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="15"]  OPNsense/WebGui generated //usr/local/lib/php.ini
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="16"] [e65e4a09-9ae0-4bd9-90f4-502952a53b30] show system routing table
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="17"] [acef4191-eee0-4632-85b1-c77644964758] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:36:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="18"] [4ed8fb74-fa4a-4733-a772-30659235104b] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="19"] [4087c5a4-59f9-4ce8-bf0c-c97edc17f47c] generate template OPNsense/Unbound/*
<13>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="20"] generate template container OPNsense/Unbound/core
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="21"]  OPNsense/Unbound/* generated //usr/local/etc/unbound.opnsense.d/access_lists.conf
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="22"]  OPNsense/Unbound/* generated //var/unbound/advanced.conf
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="23"]  OPNsense/Unbound/* generated //usr/local/etc/unbound/unbound-blocklists.conf
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="24"]  OPNsense/Unbound/* generated //usr/local/etc/unbound.opnsense.d/safesearch.conf
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="25"]  OPNsense/Unbound/* generated //usr/local/etc/unbound.opnsense.d/dot.conf
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="26"]  OPNsense/Unbound/* generated //var/unbound/private_domains.conf
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="27"]  OPNsense/Unbound/* generated //usr/local/etc/unbound.opnsense.d/domainoverrides.conf
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="28"]  OPNsense/Unbound/* generated //var/unbound/root.hints
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="29"]  OPNsense/Unbound/* generated //usr/local/etc/unbound_dhcpd.conf
<15>1 2023-11-16T16:36:09+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="30"]  OPNsense/Unbound/* generated //var/unbound/dnsbl_module.py
<13>1 2023-11-16T16:36:11+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="31"] [cddd9e9e-98f3-48dc-b99a-649de27e709e] generate template OPNsense/Filter
<13>1 2023-11-16T16:36:11+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="32"] generate template container OPNsense/Filter
<15>1 2023-11-16T16:36:11+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="33"]  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
<15>1 2023-11-16T16:36:11+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="34"]  OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
<13>1 2023-11-16T16:36:11+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="35"] [0e9697d9-6148-466b-81eb-626aa7cb0f1e] refresh url table aliases
<14>1 2023-11-16T16:36:11+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="36"] message 0e9697d9-6148-466b-81eb-626aa7cb0f1e [] returned   
<13>1 2023-11-16T16:36:11+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="37"] [a0361de1-cb44-43fa-bf57-d6413e34c639] IPsec list legacy VirtualTunnelInterfaces
<14>1 2023-11-16T16:36:11+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="38"] message 2d5263f2-2a03-49b2-acfb-0ecf2d67c664 [] returned {"status": "ok"}   
<13>1 2023-11-16T16:36:11+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="39"] [2c7a036e-940d-487f-9dd9-9c11840cf46e] configure wireguard instances ()
<13>1 2023-11-16T16:36:12+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="40"] [0495c88c-16bb-4a84-89f9-5f7353de3e9d] show system routing table
<13>1 2023-11-16T16:36:12+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="41"] [18dc7bca-deba-4124-97ad-71b3dc091eb9] show system routing table
<13>1 2023-11-16T16:36:12+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="42"] [642cc27d-dd07-4b42-9c0a-36cf7ed59991] show system routing table
<13>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="43"] [023f26ff-cb4d-4fe3-a265-e25d36044bf5] Unbound cache dump
<13>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="44"] [e0dcd01c-21c7-4b8a-8e63-77135d220381] generate template OPNsense/Unbound/*
<13>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="45"] generate template container OPNsense/Unbound/core
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="46"]  OPNsense/Unbound/* generated //usr/local/etc/unbound.opnsense.d/access_lists.conf
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="47"]  OPNsense/Unbound/* generated //var/unbound/advanced.conf
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="48"]  OPNsense/Unbound/* generated //usr/local/etc/unbound/unbound-blocklists.conf
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="49"]  OPNsense/Unbound/* generated //usr/local/etc/unbound.opnsense.d/safesearch.conf
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="50"]  OPNsense/Unbound/* generated //usr/local/etc/unbound.opnsense.d/dot.conf
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="51"]  OPNsense/Unbound/* generated //var/unbound/private_domains.conf
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="52"]  OPNsense/Unbound/* generated //usr/local/etc/unbound.opnsense.d/domainoverrides.conf
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="53"]  OPNsense/Unbound/* generated //var/unbound/root.hints
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="54"]  OPNsense/Unbound/* generated //usr/local/etc/unbound_dhcpd.conf
<15>1 2023-11-16T16:36:13+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="55"]  OPNsense/Unbound/* generated //var/unbound/dnsbl_module.py
<13>1 2023-11-16T16:36:15+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="56"] [22a64f46-7276-480d-a0a3-226f50ef0dae] Restarting syslog
<13>1 2023-11-16T16:36:15+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="57"] [20f7c2b6-4db6-445c-a9ed-a77a4ba055c2] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:36:15+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="58"] [b3fa2865-8d7c-4e63-86e9-dab1407cfdf9] restarting cron
<13>1 2023-11-16T16:36:15+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="59"] [5fa58af9-ca2c-484f-80b2-84eb0d73ec37] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:36:15+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="60"] [3cbf6242-bf5f-423e-a19d-0de95c412820] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:36:15+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="61"] [e49bfd97-a950-4d7a-884b-71bde3c2e3e5] generate template OPNsense/Syslog
<13>1 2023-11-16T16:36:15+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="62"] generate template container OPNsense/Syslog
<13>1 2023-11-16T16:36:15+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="63"] [175dbdde-2dce-41a5-879a-01b6443852f4] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="64"] [dcb28898-aff2-40d1-8734-ac9bad8db9dc] generate template OPNsense/Cron
<15>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="65"]  OPNsense/Syslog generated //etc/rc.conf.d/syslog_ng
<15>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="66"]  OPNsense/Syslog generated //etc/newsyslog.conf
<15>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="67"]  OPNsense/Syslog generated //usr/local/etc/syslog-ng.conf
<15>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="68"]  OPNsense/Syslog generated //usr/local/etc/syslog-ng.conf.d/syslog-ng-destinations.conf
<15>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="69"]  OPNsense/Syslog generated //usr/local/etc/syslog-ng.conf.d/syslog-ng-local.conf
<15>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="70"]  OPNsense/Syslog generated //usr/local/etc/syslog-ng.conf.d/syslog-ng-lockout.conf
<15>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="71"]  OPNsense/Syslog generated //usr/local/etc/syslog-ng.conf.d/syslog-ng-config-events.conf
<13>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="72"] generate template container OPNsense/Cron
<15>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="73"]  OPNsense/Cron generated //var/cron/tabs/nobody
<13>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="74"] [36a5ed01-502d-4b13-874a-70e41e6dd0f6] List syslog applications
<13>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="75"] [53c7e945-8f3e-436e-8047-8ba36220894e] Archive syslog files
<13>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="1"] [b8e96b0b-2f88-4223-b6cb-346260d8f282] configure openvpn instances
<14>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="2"] message 53c7e945-8f3e-436e-8047-8ba36220894e [] returned OK  
<13>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="3"] [98e3d671-e526-416b-9577-c2706ceaf40f] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="4"] [50bed38e-505f-46b6-81b7-638bdf1ea993] IPsec list legacy VirtualTunnelInterfaces
<14>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="5"] message b8e96b0b-2f88-4223-b6cb-346260d8f282 [] returned OK  
<14>1 2023-11-16T16:36:16+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="6"] message 22a64f46-7276-480d-a0a3-226f50ef0dae [] returned OK  
<13>1 2023-11-16T16:36:30+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="7"] [35e8d9c1-5c5f-4a5d-b777-81f33a4dd2cb] Fetching service list ( )
<13>1 2023-11-16T16:36:30+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="8"] [7ac5a76d-d0c5-4d06-bc41-63c16970c587] request traffic stats
<13>1 2023-11-16T16:36:30+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="9"] [76f8cea2-de30-4929-baed-090283099dc3] list gateway status
<13>1 2023-11-16T16:36:30+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="10"] [58c4f987-a278-4123-8de9-8f9e2b1ac58a] Retrieve firmware product info
<13>1 2023-11-16T16:36:30+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="11"] [aa1a8340-2d8b-46a1-816e-9eadfc8499ca] request traffic stats
<13>1 2023-11-16T16:36:30+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="12"] [3d4ce535-4f22-46d4-aa06-28ef03d77ca7] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:36:30+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="13"] [558a1a7e-b497-44db-a83f-6ae40005d5c5] system status
<13>1 2023-11-16T16:36:35+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="14"] [51e191ae-58df-42ca-8459-e80b8c2df3b0] list gateway status
<13>1 2023-11-16T16:36:35+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="15"] [f48b9d02-fbee-47a8-b0f1-d036100b3596] request traffic stats
<13>1 2023-11-16T16:36:35+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="16"] [e63f9d73-5c8b-4fd4-8eb3-228ade223fe6] Fetching service list ( )

...

<13>1 2023-11-16T16:39:02+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="154"] [443eef50-271f-48bc-9242-0edc12049058] IPsec list legacy VirtualTunnelInterfaces
<13>1 2023-11-16T16:39:03+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="155"] [933c7ef8-30cf-4430-b068-749481a74f04] Retrieve firmware product info
<13>1 2023-11-16T16:39:05+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="156"] [0397d3d8-2bd3-48ac-93b2-69f41fcf7920] request traffic stats
<13>1 2023-11-16T16:39:07+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="157"] [a4600f27-e91d-435e-91e1-a61695121613] list gateway status
<13>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="158"] [515ac427-f97e-40ed-8161-b0121dae9383] request pf current overall table record count and 

should be the moment when executing manually command "manually executing the command rc.filter_configure"

<13>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="158"] [515ac427-f97e-40ed-8161-b0121dae9383] request pf current overall table record count and table-entries limit
<13>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="159"] [db4ebdf7-e2f6-4da7-beea-d0d6f917dde5] generate template OPNsense/Filter
<13>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="160"] generate template container OPNsense/Filter
<15>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="161"]  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
<15>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="162"]  OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
<13>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="163"] [adff69f7-b48f-47a8-bfde-bde7e58f4805] refresh url table aliases
<13>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="164"] [64b3d0d9-684f-4e9e-a280-595ce4e0f70b] Fetching service list ( )
<13>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="165"] [f4a5a7a3-4a9b-46b3-805c-fcb2a13896d9] Retrieve firmware product info
<13>1 2023-11-16T16:39:08+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="166"] [15d624e4-d0c9-4774-8d4f-47ba0d55a12d] IPsec list legacy VirtualTunnelInterfaces
<14>1 2023-11-16T16:39:10+01:00 fw.mydomain.net configd.py 236 - [meta sequenceId="167"] message adff69f7-b48f-47a8-bfde-bde7e58f4805 [] returned {"status": "ok"}   

[fichtner]

Sorry for so many question. This came up in our meeting today and I want to specifically ask:

1. Do you have an interface assigned to each wireguard instance?
2. If yes on 1. do you have a gateway created as well?
3. If ye son 2. do you have the interface IPv4 mode set to static?
4. Does this actually only happen with the automatic NAT for WAN or do you have manual NAT rules that do not work.

The background here is that interface IPv4 mode setting was never intended for Wireguard and also doesn't work anymore on the latest versions. The real spot for setting WireGuard addresses is "tunnel address" under the respective WireGuard instance settings. The automatic nat rules were byproduct of this wrong IPv4 interface setting, too.

Cheers, Franco

[relume]

1. Do you have an interface assigned to each wireguard instance?

wg1 for mobile (road warrior) wg seems to have been created automatically. In Interfacs:Assignments the wg1 entry can not be deleted. wg2 and wg3 for Site-to-Site wg VPN can be deleted, but I cannot remember if I assigned them manually. I gues not. For all wg interface the option "Dynamic gateway policy" is enabled, thus "This interface does not require an intermediate system to act as a gateway" is enabled.

2. If yes on 1. do you have a gateway created as well?

Gateways (System:Gateway:Single) were created automatically, also for all wg interfaces.

3. If ye son 2. do you have the interface IPv4 mode set to static?

For all wg interfaces the option "IPv4 Configuration Type" (interfaces:wg...) is set to "None" as by default.

4. Does this actually only happen with the automatic NAT for WAN or do you have manual NAT rules that do not work

All NAT rules are working (only wg NAT rules do not work after reboot as discussed/reported previously).

• We have no "NAT: Port Forward" rules.
• We have 3 manual "NAT: Onte-to-One" rules from public subnet addresses (xxx.yyy.zzz.244/32, 245/32, 246/32) to the corresponding 3 internal LAN IPs.
• We have 1 manual "NAT: Outbound" rule as Source "any" on "WAN" interface, Destination "*" with "NAT address: WAN address". "Automatic outbound NAT rule generation" does not generate any rule at all. (for wg interfaces they would not be generated anyway, as you mentioned earylier).

The background here is that interface IPv4 mode setting was never intended for Wireguard and also doesn't work anymore on the latest versions. The real spot for setting WireGuard addresses is "tunnel address" under the respective WireGuard instance settings. The automatic nat rules were byproduct of this wrong IPv4 interface setting, too.

I have no option to select/set tunnel address in the IPv4 Configuration Type option list nor elsewhere.

best

[relume]

Updated today to OPNsense 23.7.9-amd64.

The issue still persists after reboot and I have to run # /usr/local/etc/rc.filter_configure manually in order that nat and rdr rules for wg interfaces are created to get access from wg networks to internal IPs mapped One-To-One to virtual public IPs (alias).

best

[fichtner]

So I worked on a root cause today and if you could test the following patch that would be helpful: https://github.com/opnsense/core/commit/64e0867a4

# opnsense-patch 64e0867a4

I'm not sure this will immediately fix it but it would be good to exclude this exact case for your missing rules problem.

[fichtner]

wg1 for mobile (road warrior) wg seems to have been created automatically. In Interfacs:Assignments the wg1 entry can not be deleted.

This isn't automatic. You simply locked the interface (which is ok). Same for the other assigned interfaces. The process is not automatic.

I have no option to select/set tunnel address in the IPv4 Configuration Type option list nor elsewhere.

That option is under WireGuard Instances "Tunnel Address" :)

Cheers, Franco

[relume]

Many thanks! I can report the following:

patch 64e0867a4:

1. rebooted > no nat, rdr rules for wg interfaces to virtual public IPs (alias) as expected.
2. Installed the patch 64e0867a4 > after patch installation same situation as on 1.
3. rebooted > same situatioin as on 1.

wg interfaces

That option is under WireGuard Instances "Tunnel Address"

I am sorry I misunderstood your statement about the wg interfaces and IPv4 interfaces:

The background here is that interface IPv4 mode setting was never intended for Wireguard and also doesn't work anymore on the latest versions. The real spot for setting WireGuard addresses is "tunnel address" under the respective WireGuard instance settings. The automatic nat rules were byproduct of this wrong IPv4 interface setting, too.

I looked for an explicit option "tunnel address" in the interface config and wg instance forms. As I have set right from the start the appropriate working tunnel address in CIDR notation I did not expect that you may assume this field. :-)

But nevertheless I can not interpret what are the meaning/consequences of

The automatic nat rules were byproduct of this wrong IPv4 interface setting, too.

So to avoid some "strange" interferences from the already existings wg interfaces I did the following test:

1. created a new wg instance (wg4) and enabled new wg instance(tunnel address 10.242.11.1/24, no overlapping port, no peers), applied wg setting, checked with # pfctl -sa > no nat, rdr rules for wg4 and all others wg interfaces, neither wg1-3 clients can access virtual public IPs.
2. created/assigned in WebGUI interfaces new wg4 interface as proposed from the WebGUI, leaving all defaults for the new created interface (not enabled), saved the form, applied the changes, checked with # pfctl -sa > nat, rdr rules for wg1-3 were created at this point, wg1-3clients can access virtual public IPs.
3. enabled wg4 interface, saved, checked with # pfctl -sa > now also nat, rdr rules for wg4 to all definied virtual public IPs (alias) are present.
4. rebooted, checked with # pfctl -sa > no nat, rdr rules for all wg interfaces are present.
5. instead of running manually # /usr/local/etc/rc.filter_configure after reboot, just disabled wg4 interface, saved and applied changes in the form, checked with # pfctl -sa > now nat, rdr rules for wg1-3 are present (rules for wg4 are not present as expected). Thus it does not matter if the command # /usr/local/etc/rc.filter_configure is run manually or some changes on the WebGUI interface are applied (assuming that later calls /usr/local/etc/rc.filter_configure in some deppending circumstances).

best regards

[fichtner]

Things are become more weird as we try to fix this. Ok, so patch not working is totally unexpected. It appears that timing does matter as a later filter reload is ok, but the inline reload right after wireguard configuration still is not. Another user also said automating the filter reload "some time" after boot seems to fix it too.

I've prepared a separate debug patch with https://github.com/opnsense/core/commit/6e938bd0c7

# opnsense-patch 6e938bd0c7

Can you apply this one as well and let me have the following output again right after something you did fixed the rules?

# diff -u /tmp/rules.debug{.old,}
# diff -u /tmp/ifconfig.debug{.old,}

Thanks, Franco

[relume]

hello - thanks again and hopping I can help. I did the following:

1. enabled test wg4 interface, saved, applied changes > nat, rdr rules for wg4 are added to existing rules for others wg1-3 interfaces (added manually last day with filter reload (rc.filter_configure).
2. installed new patch # opnsense-patch 6e938bd0c7 rebooted > no nat, rdr rules for wg1-4.
3. run diff -u /tmp/rules.debug{.old,} and diff -u /tmp/ifconfig.debug{.old,} > no output either.
4. opened on WebGUI interfaces edit form for test wg4 interface (allready enabled on reboot, IPv4 Configuration Type = None), enabled interface option Lock/Prevent interface removal assuming that this option setting should have no direct effect to interfaces "basic" settings, saved form, run diff -u /tmp/rules.debug{.old,} and diff -u /tmp/ifconfig.debug{.old,} > no output either.
5. applied changes to the wg4 interface WebGUI form, run diff -u /tmp/rules.debug{.old,} and diff -u /tmp/ifconfig.debug{.old,} > nat, rdr rules for wg1-4 were created as expected and got the following outputs:

/home/admin # diff -u /tmp/rules.debug{.old,}
--- /tmp/rules.debug.old    2023-11-30 18:01:38.296918000 +0100
+++ /tmp/rules.debug    2023-11-30 18:01:38.301007000 +0100
@@ -101,25 +101,49 @@
 rdr on lo0 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
 rdr on vtnet0 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
 rdr on vtnet2 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
+rdr on wg2 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
+rdr on wg1 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
+rdr on wg3 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
+rdr on wg4 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
 nat on lo0 inet from (lo0:network) to {10.92.1.10} -> (lo0) port 1024:65535 # atlas
 nat on vtnet0 inet from (vtnet0:network) to {10.92.1.10} -> (vtnet0) port 1024:65535 # atlas
 nat on vtnet2 inet from (vtnet2:network) to {10.92.1.10} -> (vtnet2) port 1024:65535 # atlas
+nat on wg2 inet from (wg2:network) to {10.92.1.10} -> (wg2) port 1024:65535 # atlas
+nat on wg1 inet from (wg1:network) to {10.92.1.10} -> (wg1) port 1024:65535 # atlas
+nat on wg3 inet from (wg3:network) to {10.92.1.10} -> (wg3) port 1024:65535 # atlas
+nat on wg4 inet from (wg4:network) to {10.92.1.10} -> (wg4) port 1024:65535 # atlas
 nat on vtnet1 inet from (vtnet1:network) to {10.92.1.10} -> (vtnet1) port 1024:65535 # atlas
 binat on vtnet1 from 10.92.1.11 to any -> xxx.yyy.zzz.245/32 # helios
 rdr on lo0 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
 rdr on vtnet0 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
 rdr on vtnet2 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
+rdr on wg2 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
+rdr on wg1 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
+rdr on wg3 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
+rdr on wg4 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
 nat on lo0 inet from (lo0:network) to {10.92.1.11} -> (lo0) port 1024:65535 # helios
 nat on vtnet0 inet from (vtnet0:network) to {10.92.1.11} -> (vtnet0) port 1024:65535 # helios
 nat on vtnet2 inet from (vtnet2:network) to {10.92.1.11} -> (vtnet2) port 1024:65535 # helios
+nat on wg2 inet from (wg2:network) to {10.92.1.11} -> (wg2) port 1024:65535 # helios
+nat on wg1 inet from (wg1:network) to {10.92.1.11} -> (wg1) port 1024:65535 # helios
+nat on wg3 inet from (wg3:network) to {10.92.1.11} -> (wg3) port 1024:65535 # helios
+nat on wg4 inet from (wg4:network) to {10.92.1.11} -> (wg4) port 1024:65535 # helios
 nat on vtnet1 inet from (vtnet1:network) to {10.92.1.11} -> (vtnet1) port 1024:65535 # helios
 binat on vtnet1 from 10.92.1.12 to any -> xxx.yyy.zzz.246/32 # test
 rdr on lo0 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
 rdr on vtnet0 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
 rdr on vtnet2 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
+rdr on wg2 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
+rdr on wg1 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
+rdr on wg3 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
+rdr on wg4 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
 nat on lo0 inet from (lo0:network) to {10.92.1.12} -> (lo0) port 1024:65535 # test
 nat on vtnet0 inet from (vtnet0:network) to {10.92.1.12} -> (vtnet0) port 1024:65535 # test
 nat on vtnet2 inet from (vtnet2:network) to {10.92.1.12} -> (vtnet2) port 1024:65535 # test
+nat on wg2 inet from (wg2:network) to {10.92.1.12} -> (wg2) port 1024:65535 # test
+nat on wg1 inet from (wg1:network) to {10.92.1.12} -> (wg1) port 1024:65535 # test
+nat on wg3 inet from (wg3:network) to {10.92.1.12} -> (wg3) port 1024:65535 # test
+nat on wg4 inet from (wg4:network) to {10.92.1.12} -> (wg4) port 1024:65535 # test
 nat on vtnet1 inet from (vtnet1:network) to {10.92.1.12} -> (vtnet1) port 1024:65535 # test

 antispoof log for vtnet1 

and

/home/admin # diff -u /tmp/ifconfig.debug{.old,}
--- /tmp/ifconfig.debug.old 2023-11-30 18:01:38.297002000 +0100
+++ /tmp/ifconfig.debug 2023-11-30 18:01:38.300714000 +0100

@@ -41,23 +41,24 @@
    groups: pfsync
 pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
    groups: pflog
-wg2: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1380
-   description: wg_campora (opt4)
+wg1: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
+   inet 10.242.10.1 netmask 0xffffff80
    groups: wg wireguard
-   nd6 options=9<PERFORMNUD,IFDISABLED>
-wg1: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
-   description: wg_mobile (opt2)
+   nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
+wg2: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
+   inet 10.242.81.80 netmask 0xffffffff
    groups: wg wireguard
-   nd6 options=9<PERFORMNUD,IFDISABLED>
+   nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
 wg3: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
-   description: wg_prosana (opt3)
    options=80000<LINKSTATE>
+   inet 10.242.70.80 netmask 0xffffffff
    groups: wg wireguard
-   nd6 options=9<PERFORMNUD,IFDISABLED>
+   nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
 wg4: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
    description: wg_test (opt5)
    options=80000<LINKSTATE>
+   inet 10.242.11.1 netmask 0xffffff00
    groups: wg wireguard
    nd6 options=9<PERFORMNUD,IFDISABLED>

so I did an other test analog to the steps above as follwing:

1. rebooted again > no nat, rdr rules for wg1-4.
2. run diff -u /tmp/rules.debug{.old,} and diff -u /tmp/ifconfig.debug{.old,} > no output either.
3. opened on WebGUI interfaces edit form for vtnet0 interface (WAN interface, IPv4 Configuration Type = static IPv4 ), enabled interface option Lock/Prevent interface removal, saved form, run diff -u /tmp/rules.debug{.old,} and diff -u /tmp/ifconfig.debug{.old,} > no output either.
4. applied changes to the vtnet0 interface WebGUI form, run diff -u /tmp/rules.debug{.old,} and diff -u /tmp/ifconfig.debug{.old,} > nat, rdr rules for wg1-4 were created as expected and got the following outputs:

diff -u /tmp/rules.debug{.old,}
--- /tmp/rules.debug.old    2023-11-30 18:30:16.895980000 +0100
+++ /tmp/rules.debug    2023-11-30 18:30:16.900970000 +0100
@@ -101,25 +101,49 @@
 rdr on lo0 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
 rdr on vtnet0 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
 rdr on vtnet2 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
+rdr on wg2 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
+rdr on wg1 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
+rdr on wg3 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
+rdr on wg4 from {any} to {xxx.yyy.zzz.244/32} -> {10.92.1.10} bitmask # atlas
 nat on lo0 inet from (lo0:network) to {10.92.1.10} -> (lo0) port 1024:65535 # atlas
 nat on vtnet0 inet from (vtnet0:network) to {10.92.1.10} -> (vtnet0) port 1024:65535 # atlas
 nat on vtnet2 inet from (vtnet2:network) to {10.92.1.10} -> (vtnet2) port 1024:65535 # atlas
+nat on wg2 inet from (wg2:network) to {10.92.1.10} -> (wg2) port 1024:65535 # atlas
+nat on wg1 inet from (wg1:network) to {10.92.1.10} -> (wg1) port 1024:65535 # atlas
+nat on wg3 inet from (wg3:network) to {10.92.1.10} -> (wg3) port 1024:65535 # atlas
+nat on wg4 inet from (wg4:network) to {10.92.1.10} -> (wg4) port 1024:65535 # atlas
 nat on vtnet1 inet from (vtnet1:network) to {10.92.1.10} -> (vtnet1) port 1024:65535 # atlas
 binat on vtnet1 from 10.92.1.11 to any -> xxx.yyy.zzz.245/32 # helios
 rdr on lo0 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
 rdr on vtnet0 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
 rdr on vtnet2 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
+rdr on wg2 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
+rdr on wg1 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
+rdr on wg3 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
+rdr on wg4 from {any} to {xxx.yyy.zzz.245/32} -> {10.92.1.11} bitmask # helios
 nat on lo0 inet from (lo0:network) to {10.92.1.11} -> (lo0) port 1024:65535 # helios
 nat on vtnet0 inet from (vtnet0:network) to {10.92.1.11} -> (vtnet0) port 1024:65535 # helios
 nat on vtnet2 inet from (vtnet2:network) to {10.92.1.11} -> (vtnet2) port 1024:65535 # helios
+nat on wg2 inet from (wg2:network) to {10.92.1.11} -> (wg2) port 1024:65535 # helios
+nat on wg1 inet from (wg1:network) to {10.92.1.11} -> (wg1) port 1024:65535 # helios
+nat on wg3 inet from (wg3:network) to {10.92.1.11} -> (wg3) port 1024:65535 # helios
+nat on wg4 inet from (wg4:network) to {10.92.1.11} -> (wg4) port 1024:65535 # helios
 nat on vtnet1 inet from (vtnet1:network) to {10.92.1.11} -> (vtnet1) port 1024:65535 # helios
 binat on vtnet1 from 10.92.1.12 to any -> xxx.yyy.zzz.246/32 # test
 rdr on lo0 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
 rdr on vtnet0 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
 rdr on vtnet2 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
+rdr on wg2 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
+rdr on wg1 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
+rdr on wg3 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
+rdr on wg4 from {any} to {xxx.yyy.zzz.246/32} -> {10.92.1.12} bitmask # test
 nat on lo0 inet from (lo0:network) to {10.92.1.12} -> (lo0) port 1024:65535 # test
 nat on vtnet0 inet from (vtnet0:network) to {10.92.1.12} -> (vtnet0) port 1024:65535 # test
 nat on vtnet2 inet from (vtnet2:network) to {10.92.1.12} -> (vtnet2) port 1024:65535 # test
+nat on wg2 inet from (wg2:network) to {10.92.1.12} -> (wg2) port 1024:65535 # test
+nat on wg1 inet from (wg1:network) to {10.92.1.12} -> (wg1) port 1024:65535 # test
+nat on wg3 inet from (wg3:network) to {10.92.1.12} -> (wg3) port 1024:65535 # test
+nat on wg4 inet from (wg4:network) to {10.92.1.12} -> (wg4) port 1024:65535 # test
 nat on vtnet1 inet from (vtnet1:network) to {10.92.1.12} -> (vtnet1) port 1024:65535 # test

 antispoof log for vtnet1 

and

diff -u /tmp/ifconfig.debug{.old,}
--- /tmp/ifconfig.debug.old 2023-11-30 18:30:16.896089000 +0100
+++ /tmp/ifconfig.debug 2023-11-30 18:30:16.900626000 +0100
@@ -41,23 +41,23 @@
    groups: pfsync
 pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
    groups: pflog
-wg2: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1380
-   description: wg_campora (opt4)
+wg1: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
+   inet 10.242.10.1 netmask 0xffffff80
    groups: wg wireguard
-   nd6 options=9<PERFORMNUD,IFDISABLED>
-wg1: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
-   description: wg_mobile (opt2)
+   nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
+wg2: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
+   inet 10.242.81.80 netmask 0xffffffff
    groups: wg wireguard
-   nd6 options=9<PERFORMNUD,IFDISABLED>
+   nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
 wg3: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
-   description: wg_prosana (opt3)
    options=80000<LINKSTATE>
+   inet 10.242.70.80 netmask 0xffffffff
    groups: wg wireguard
-   nd6 options=9<PERFORMNUD,IFDISABLED>
+   nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
 wg4: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
-   description: wg_test (opt5)
    options=80000<LINKSTATE>
+   inet 10.242.11.1 netmask 0xffffff00
    groups: wg wireguard
-   nd6 options=9<PERFORMNUD,IFDISABLED>
+   nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>

so I seems that applying changes to whatever WebGUI interface edit form will reloading interface and filter in general.

best and thank you again

[fichtner]

Ok this is the expected reason although totally unexpected. Can you check what dmesg outputs? And what the wireguard log says specifically?

[fichtner]

Somehow I suspect this has to do with proxy ARP.. specifically VIPs changes in 23.7.7. Is e.g. 10.242.10.1 a tunnel address or proxy arp address or both?

[relume]

hello, appreciating very much your effort. please find the report:

Ok this is the expected reason although totally unexpected. Can you check what dmesg outputs? And what the wireguard log says specifically?

after reboot I get this log entries:

wireguard/latest.log
<37>1 2023-12-01T09:24:51+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="1"] wireguard instance wg_mobile (wg1) can not reconfigure without stopping it first.
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="2"] wireguard instance wg_mobile (wg1) stopped
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="3"] wireguard instance wg_mobile (wg1) started
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="4"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: entering configure using 'opt2'
<35>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="5"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: not a valid interface gateway address: ''
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="6"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: plugins_configure monitor (,WG_MOBILE_GW)
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="7"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: plugins_configure monitor (execute task : dpinger_configure_do(,WG_MOBILE_GW))
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="8"] wireguard instance wg_hb_campora (wg2) can not reconfigure without stopping it first.
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="9"] wireguard instance wg_hb_campora (wg2) stopped
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="10"] wireguard instance wg_hb_campora (wg2) started
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="11"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: entering configure using 'opt4'
<35>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="12"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: not a valid interface gateway address: ''
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="13"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: plugins_configure monitor (,WG_CAMPORA_GW)
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="14"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: plugins_configure monitor (execute task : dpinger_configure_do(,WG_CAMPORA_GW))
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="15"] wireguard instance wg_prosana (wg3) can not reconfigure without stopping it first.
<37>1 2023-12-01T09:24:52+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="16"] wireguard instance wg_prosana (wg3) stopped
<37>1 2023-12-01T09:24:53+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="17"] wireguard instance wg_prosana (wg3) started
<37>1 2023-12-01T09:24:53+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="18"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: entering configure using 'opt3'
<37>1 2023-12-01T09:24:53+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="19"] wireguard instance wg_test (wg4) can not reconfigure without stopping it first.
<37>1 2023-12-01T09:24:53+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="20"] wireguard instance wg_test (wg4) stopped
<37>1 2023-12-01T09:24:53+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="21"] wireguard instance wg_test (wg4) started
<37>1 2023-12-01T09:24:53+01:00 myopnsense.net wireguard 30075 - [meta sequenceId="22"] /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: ROUTING: entering configure using 'opt5'

After editing WebGUI interface enabling/disabling option Lock/Prevent interface removal as "dummy" action, saving form, applying changes nat, rdr rules for wg interfaces are created as expected but there are no additional log entries in the wireguard log.

dmesg (entries not very clear as timestamp is missing):

---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.2-RELEASE-p5 stable/23.7-n254837-8806e8fefb1 SMP amd64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
VT(vga): text 80x25
CPU: QEMU Virtual CPU version 2.5+ (2266.85-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x60fb1  Family=0xf  Model=0x6b  Stepping=1
  Features=0x1783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x80202001<SSE3,CX16,x2APIC,HV>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
Hypervisor: Origin = "KVMKVMKVM"
real memory  = 9663676416 (9216 MB)
avail memory = 8274354176 (7891 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <BOCHS  BXPC    >
FreeBSD/SMP: Multiprocessor System Detected: 6 CPUs
FreeBSD/SMP: 1 package(s) x 6 core(s)
random: unblocking device.
ioapic0 <Version 1.1> irqs 0-23
Launching APs: 1 2 5 3 4
wlan: mac acl policy registered
random: entropy device external interface
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
vtvga0: <VT VGA driver>
kvmclock0: <KVM paravirtual clock>
Timecounter "kvmclock" frequency 1000000000 Hz quality 975
kvmclock0: registered as a time-of-day clock, resolution 0.000001s
aesni0: No AES or SHA support.
acpi0: <BOCHS BXPC>
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: could not evaluate _ADR - AE_NOT_FOUND
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf140-0xf14f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
uhci0: <Intel 82371SB (PIIX3) USB controller> port 0xf0c0-0xf0df irq 11 at device 1.2 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
pci0: <bridge> at device 1.3 (no driver attached)
virtio_pci0: <VirtIO PCI (modern) GPU adapter> mem 0xfd000000-0xfd7fffff,0xfde00000-0xfde03fff,0xfead0000-0xfead0fff irq 10 at device 2.0 on pci0
virtio_pci1: <VirtIO PCI (legacy) Balloon adapter> port 0xf000-0xf03f mem 0xfde04000-0xfde07fff irq 11 at device 3.0 on pci0
vtballoon0: <VirtIO Balloon Adapter> on virtio_pci1
pcib1: <ACPI PCI-PCI bridge> mem 0xfead1000-0xfead10ff irq 10 at device 5.0 on pci0
pci1: <ACPI PCI bus> on pcib1
virtio_pci2: <VirtIO PCI (legacy) SCSI adapter> port 0xe000-0xe03f mem 0xfe800000-0xfe800fff,0xfdc00000-0xfdc03fff irq 10 at device 1.0 on pci1
vtscsi0: <VirtIO SCSI Adapter> on virtio_pci2
virtio_pci3: <VirtIO PCI (legacy) Console adapter> port 0xf040-0xf07f mem 0xfead2000-0xfead2fff,0xfde08000-0xfde0bfff irq 11 at device 8.0 on pci0
vtcon0: <VirtIO Console Adapter> on virtio_pci3
virtio_pci4: <VirtIO PCI (legacy) Console adapter> port 0xf080-0xf0bf mem 0xfead3000-0xfead3fff,0xfde0c000-0xfde0ffff irq 10 at device 9.0 on pci0
vtcon1: <VirtIO Console Adapter> on virtio_pci4
virtio_pci5: <VirtIO PCI (legacy) Network adapter> port 0xf0e0-0xf0ff mem 0xfead4000-0xfead4fff,0xfde10000-0xfde13fff irq 10 at device 18.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci5
vtnet0: Ethernet address: 00:50:56:00:10:00
vtnet0: netmap queues/slots: TX 1/256, RX 1/512
000.000769 [ 449] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=512
virtio_pci6: <VirtIO PCI (legacy) Network adapter> port 0xf100-0xf11f mem 0xfead5000-0xfead5fff,0xfde14000-0xfde17fff irq 11 at device 19.0 on pci0
vtnet1: <VirtIO Networking Adapter> on virtio_pci6
vtnet1: Ethernet address: 00:50:56:00:10:90
vtnet1: netmap queues/slots: TX 1/256, RX 1/512
000.000770 [ 449] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=512
virtio_pci7: <VirtIO PCI (legacy) Network adapter> port 0xf120-0xf13f mem 0xfead6000-0xfead6fff,0xfde18000-0xfde1bfff irq 11 at device 20.0 on pci0
vtnet2: <VirtIO Networking Adapter> on virtio_pci7
vtnet2: Ethernet address: 00:50:56:00:10:10
vtnet2: netmap queues/slots: TX 1/256, RX 1/512
000.000771 [ 449] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=512
pcib2: <ACPI PCI-PCI bridge> mem 0xfead7000-0xfead70ff irq 10 at device 30.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> mem 0xfead8000-0xfead80ff irq 11 at device 31.0 on pci0
pci3: <ACPI PCI bus> on pcib3
acpi_syscontainer0: <System Container> on acpi0
vmgenc0: <VM Generation Counter> on acpi0
acpi_syscontainer1: <System Container> port 0xaf00-0xaf0b on acpi0
acpi_syscontainer2: <System Container> port 0xafe0-0xafe3 on acpi0
acpi_syscontainer3: <System Container> port 0xae00-0xae17 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
orm0: <ISA Option ROM> at iomem 0xe8000-0xeffff pnpid ORM0000 on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
fdc0: No FDOUT register!
Timecounters tick every 10.000 msec
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
ugen0.1: <Intel UHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
da0 at vtscsi0 bus 0 scbus2 target 0 lun 0
da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 61440MB (125829120 512 byte sectors)
cd0 at ata1 bus 0 scbus1 target 0 lun 0
cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: Serial Number QM00003
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
uhub0: 2 ports with 2 removable, self powered
ugen0.2: <QEMU QEMU USB Tablet> at usbus0
intsmb0: <Intel PIIX4 SMBUS Interface> irq 9 at device 1.3 on pci0
intsmb0: intr IRQ 9 enabled revision 0
smbus0: <System Management Bus> on intsmb0
uhid0 on uhub0
uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 2.00/0.00, addr 2> on usbus0
lo0: link state changed to UP
pflog0: permanently promiscuous mode enabled
vtnet1: link state changed to UP
vtnet2: link state changed to UP
wg0: changing name to 'wg2'
wg2: link state changed to UP
wg1: link state changed to UP
wg3: link state changed to UP
wg4: link state changed to UP
[fib_algo] inet.0 (bsearch4#16) rebuild_fd_flm: switching algo to radix4_lockless
vtnet0: link state changed to UP
wg1: link state changed to DOWN
wg1: link state changed to UP
wg2: link state changed to DOWN
wg0: changing name to 'wg2'
wg2: link state changed to UP
wg3: link state changed to DOWN
wg3: link state changed to UP
wg4: link state changed to DOWN
wg4: link state changed to UP
wg4: link state changed to DOWN
wg4: link state changed to UP

Somehow I suspect this has to do with proxy ARP.. specifically VIPs changes in 23.7.7.

Not sure what you mean. The issue (that nat, rdr rules for wg interfaces are not created after reboot) started with update 23.7.6, until and with 23.7.5 the issue was not present. Until this comment in this thread the virtual public IPswere of type (Mode) Proxy ARP, since there they are configured as "IP Alias".

Is e.g. 10.242.10.1 a tunnel address or proxy arp address or both?

All 10.242.X.X IPs present in the above output from ifconfig.debug for wg interfaces are tunnel addresses and only defined in the wg instance configuration.

For the moment as an "easygoing" workaround I followed the hint you mentioned from an other user. I put an entry in /usr/local/etc/rc.syshook.d/start to reload filter configuration at the end of the boot process:

/usr/local/etc/rc.syshook.d/start/ 
10-newwanip
20-freebsd
25-syslog
90-carp
90-cron
90-openvpn
90-sysctl
95-beep
99-filter_configure_workaround

the entry/file 99-filter_configure_workaround contains:

#!/bin/sh

/usr/local/etc/rc.filter_configure

So this workaround works and missing nat, rdr rules for wg interfaces to the virtual public IPs (IP Alias) are "immediately" present "automatically" after reboot. Of course not a fix to the deeper cause, but just a working workaround and unfortunately not represented in the configuration backups. :-)

Many thanks again.

[OPNsense-bot]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.