search

opnsense / core

OPNsense GUI, API and systems backend
https://opnsense.org/
BSD 2-Clause "Simplified" License
3.36k stars 753 forks source link

Port forwarding via wireguard interface not working #4389

Closed goofus closed 3 years ago

goofus commented 4 years ago

Important notices Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md

[x] I have searched the existing issues and I'm convinced that mine is new.

Describe the bug Port forwarding doesn't work through wireguard interface to lan.

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce Steps to reproduce the behavior:

  1. Establish wireguard connection
  2. Forward a tcp port from the wireguard(WAN) network to LAN network
  3. Open port with ncat on host in LAN
  4. Try to connect to forwarded port from WAN
  5. Follow packets with tcpdump on OPNsense firewall

Expected behavior Establishing a tcp connection between WAN host (xxx.de) via wireguard interface (wg0) to LAN host.

Screenshots

https://i.imgur.com/x9hBagG.png https://i.imgur.com/dJs9l38.png https://i.imgur.com/Ylx9J3L.png

Relevant log files tcpdump on LAN interface on OPNsense

19:28:57.774733 IP xxx.de.23023 > 192.168.20.101.11526: Flags [S], seq 1802740233, win 64240, options [mss 1380,sackOK,TS val 79282081 ecr 0,nop,wscale 7], length 0
19:28:57.775498 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133438719 ecr 79282081,nop,wscale 7], length 0
19:28:58.776502 IP xxx.de.23023 > 192.168.20.101.11526: Flags [S], seq 1802740233, win 64240, options [mss 1380,sackOK,TS val 79283084 ecr 0,nop,wscale 7], length 0
19:28:58.777248 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133439721 ecr 79282081,nop,wscale 7], length 0
19:28:59.779374 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133440723 ecr 79282081,nop,wscale 7], length 0
19:29:00.792129 IP xxx.de.23023 > 192.168.20.101.11526: Flags [S], seq 1802740233, win 64240, options [mss 1380,sackOK,TS val 79285100 ecr 0,nop,wscale 7], length 0
19:29:00.792870 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133441737 ecr 79282081,nop,wscale 7], length 0
19:29:02.979382 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133443923 ecr 79282081,nop,wscale 7], length 0
19:29:04.888799 IP xxx.de.23023 > 192.168.20.101.11526: Flags [S], seq 1802740233, win 64240, options [mss 1380,sackOK,TS val 79289196 ecr 0,nop,wscale 7], length 0
19:29:04.888967 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133445833 ecr 79282081,nop,wscale 7], length 0
19:29:08.952726 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133449897 ecr 79282081,nop,wscale 7], length 00

tcpdump on wg0 interface on OPNsense

19:34:19.395412 IP xxx.de.23023 > 10.65.68.45.11526: Flags [S], seq 2533116111, win 64240, options [mss 1380,sackOK,TS val 79603694 ecr 0,nop,wscale 7], length 0
19:34:20.408152 IP xxx.de.23023 > 10.65.68.45.11526: Flags [S], seq 2533116111, win 64240, options [mss 1380,sackOK,TS val 79604707 ecr 0,nop,wscale 7], length 0
19:34:22.424482 IP xxx.de.23023 > 10.65.68.45.11526: Flags [S], seq 2533116111, win 64240, options [mss 1380,sackOK,TS val 79606723 ecr 0,nop,wscale 7], length 0
19:34:26.682679 IP xxx.de.23023 > 10.65.68.45.11526: Flags [S], seq 2533116111, win 64240, options [mss 1380,sackOK,TS val 79610980 ecr 0,nop,wscale 7], length 0`

SYN ACK doesn't get forwarded

Environment

OPNsense 20.7.3-amd64

Forum Report https://forum.opnsense.org/index.php?topic=18013 https://forum.opnsense.org/index.php?topic=18062 https://forum.opnsense.org/index.php?topic=19409 https://forum.opnsense.org/index.php?topic=17973

Tuxist commented 3 years ago

same issue on my system

Optic00 commented 3 years ago

my 2 cents: its not just port forwarding via WireGuard, its generally all VIP that don't behave like a physical interface. The same problem is with GRE and OpenVPN Tunnels. Last time I researched this I suspected reply-to to be the problem. What I had working was NAT through WireGuard tunnel after hours fiddling around with this. I would like to use a WireGuard Interface with a public ipv4 and run HA proxy or just NAT it but it doesn't work.

rt-outofservice commented 3 years ago

Same issue on the latest 20.7.5. Port forwarding through OpenVPN tunnel in exactly same configuration works just fine, so this is Wireguard specific.

mimugmail commented 3 years ago

You can play around with reply-to Rules to figure it out. I dont have a lab to test

amonhk commented 3 years ago

Same issue on WireGuard Port forwarding, in my opinion the problem is in the routes, the traffic does not return for the same input route. Forcing a static route works but doesn't fix the problem.

mimugmail commented 3 years ago

Correct, there is a mismatch in the reply-to rules somewhere

calvinbui commented 3 years ago

Port forwarding works for me on OpenVPN but not WireGuard.

TheLinuxGuy commented 3 years ago

I am also trying to use wireguard + port forwarding and a public IP from the other endpoint and it is not working. Seems related to routing table.

I wrote about it here before I found this bug: https://forum.opnsense.org/index.php?topic=21006.0

TheLinuxGuy commented 3 years ago

You can play around with reply-to Rules to figure it out. I dont have a lab to test

Any links / references on how to do these test or play around / sample commands etc?

H4R0 commented 3 years ago

@TheLinuxGuy don't waste time on it, I tried everything already.

pfsense-devel now has wireguard using the kernel implementation, opnsense will probably follow with 21.1.x/21.7

The kernel implementation will boost performance again and hopefully fix the routing problem.

fichtner commented 3 years ago

I'm curious why should it solve routing issues? Kernel WireGuard has everyone giddy, but it won't live up to the hype for sure.

akront commented 3 years ago

not sure if this is relevant, but managed to make outbound NAT and port forwarding work on WG after weeks of trial and error

the key component missing was WG only works on default WAN IP not WAN VIPs that you may have, and also you need outbound NAT rules for the port forward traffic coming in and going over the tunnel and also manual MSS value set for all to work properly

more references here https://forum.opnsense.org/index.php?topic=21445.15

OPNsense-bot commented 3 years ago

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.

H4R0 commented 3 years ago

From the forum discussion port forwarding works with the kernel implementation.

The package is based on the freebsd rework by jason and not the horrible netgate implementation.

Install with (experimental and still in work): pkg install wireguard-kmod

fichtner commented 3 years ago

Please only install wireguard-kmod instead to avoid future side effects with the wireguard meta package.

frankyw commented 3 years ago

Can still not get this to work, no matter what I try.

H4R0 commented 3 years ago

Can still not get this to work, no matter what I try.

I did a quick test and it indeed still routes over wan.

trunet commented 3 years ago

I'm also unable to make port forwarding works. This ticket should be reopened, if not I can open a new one.

In my case, I installed wireguard-kmod, rebooted. tcpdumps below, you can see packet arriving fine on wg1 but being replied back on wan directly.

OpnSense wg1 tcpdump:

13:12:46.987457 IP [REDACTED_PUBLIC_IP].46256 > 10.13.128.89.10000: Flags [S], seq 3380801657, win 29200, options [mss 1380,sackOK,TS val 3306454498 ecr 0,nop,wscale 7], length 0

OpnSense ix1_vlan34 tcpdump (my WAN interface):

13:12:46.987713 IP 10.13.128.89.10000 > [REDACTED_PUBLIC_IP].46256: Flags [S.], seq 3870681174, ack 3380801658, win 65160, options [mss 1460,sackOK,TS val 3841074193 ecr 3306454498,nop,wscale 7], length 0
13:12:46.987814 IP 10.13.128.89.10000 > [REDACTED_PUBLIC_IP].46256: Flags [S.], seq 3870681174, ack 3380801658, win 65160, options [mss 1460,sackOK,TS val 3841074193 ecr 3306454498,nop,wscale 7], length 0
...... more TCP SYN/ACK retries
fichtner commented 3 years ago

It would be better to take this to the forum where people can actually tell you it works and help diagnose. Also, kmod is unsupported at this point.

Cheers, Franco

amonhk commented 3 years ago

I noticed this commit 286000d

Firewall - allow manual reply-to configuration

Refactor firewall edit page to allow selection of a reply-to gateway in stead of the single "disablereplyto" option. Since underscores aren't valid for gateway names, we should be able to use disable here to mark the previous "disablereplyto" setting which we can unravel when saving settings again.

just tested 21.7.b with the new reply-to option and it looks like WORK immagine

trunet commented 3 years ago

@amonhk I think this is still not available for general public, right? But when we create a NAT port-forward, the firewall rule is added automatically and associated, and we don't have access to change this. If this makes it to work, the NAT rule should handle the firewall rule automatically as well.

frankyw commented 3 years ago

It would be better to take this to the forum where people can actually tell you it works and help diagnose. Also, kmod is unsupported at this point.

Already tried that, still does not work: https://forum.opnsense.org/index.php?topic=22856.0

trunet commented 3 years ago

Is this a misconfig by all of us, or this is a bug? if it's a bug, or this issue needs to be reopened, or we need to open a new one.

H4R0 commented 3 years ago

Is this a misconfig by all of us, or this is a bug? if it's a bug, or this issue needs to be reopened, or we need to open a new one.

Since the exact same setup works with openvpn and ipsec I don't see how this is a misconfig from us.

Site-to-Site works, it's just port forwarding where replies get routed over wan instead of the wg interface.

Disabling reply-to and several other settings have no effect on this. The only workaround is to masquerade on the client side.

It would be interesting if pfsense has the same issue though.

amonhk commented 3 years ago

@trunet I also don't understand why the same openvpn logic applied to wireguard doesn't work (I think it's a bug) @fraenki I made it work that way: with the 21.7.b version (type development) immagine firewall ingress rule on the wireguard interface immagine Filter rule association set to None immagine

I hope it will be useful to someone

mimugmail commented 3 years ago

Thx for testing, I plan to write an Advanced example section for WG docs

trunet commented 3 years ago

same port forward rule for both openvpn and wireguard interfaces. for openvpn it adds reply-to automatically, for wireguard it doesn't.

pass in quick on ovpnc4 reply-to (ovpnc4 10.XX.X.XX) inet proto tcp from any to <myserver> port = 10000 flags S/SA keep state label "514698ad49eba95b7d5b82502e1c3e65"
pass in quick on ovpnc4 reply-to (ovpnc4 10.XX.X.XX) inet proto udp from any to <myserver> port = 10000 keep state label "514698ad49eba95b7d5b82502e1c3e65"
pass in quick on wg1 inet proto tcp from any to <myserver> port = 10000 flags S/SA keep state label "514698ad49eba95b7d5b82502e1c3e65"
pass in quick on wg1 inet proto udp from any to <myserver> port = 10000 keep state label "514698ad49eba95b7d5b82502e1c3e65"
jortan commented 3 years ago

It would be interesting if pfsense has the same issue though.

pfsense doesn't (or at least didn't previously) have the same issue. I migrated from pfsense not too long after they introduced wireguard (due to the wg implementation drama / pfsense announcing they were going to remove wireguard temporarily)

I'm certain I had port forwarding working on wireguard interfaces in pfsense, but I also can't get them working in opnsense.

trunet commented 3 years ago

I did some $this->log to debug the problem.

wireguard interface doesn't contain a gateway on the interfacemapping, although I have a gateway defined on "Single"... openvpn contains a gateway there.

(
    [if] => wg1
    [descr] => [MY INTERFACE]
    [enable] => 1
    [lock] => 1
    [spoofmac] =>
    [ipaddrv6] =>
    [ipaddr] =>
    [gateway] =>
    [gatewayv6] =>
    [ifconfig] => Array
        (
            [ipv4] => Array
                (
                    [0] => Array
                        (
                            [ipaddr] => [MY VPN IP]
                            [subnetbits] => 24
                        )

                )

            [ipv6] => Array
                (
                )

        )

)

Therefore, https://github.com/opnsense/core/blob/master/src/opnsense/mvc/app/library/OPNsense/Firewall/FilterRule.php#L150 is false and it doesn't add reply-to wireguard interfaces.

Nikotine1 commented 3 years ago

@trunet I also don't understand why the same openvpn logic applied to wireguard doesn't work (I think it's a bug) @fraenki I made it work that way: with the 21.7.b version (type development) immagine firewall ingress rule on the wireguard interface immagine Filter rule association set to None immagine

I hope it will be useful to someone

I can confirm that manually creating the NAT port forward firewall rule, including the reply-to, fixed the port forwarding issue via Wireguard!

I could not figure out why Transmission kept saying my listening port was closed. It worked via WAN, but stayed closed via the WG interface. I even used Wireshark and could see Transmission communicating back an forth with 87.98.162.88. There was just some traffic that didn't happen when using the WG interface. I'm so lucky I stumbled upon this github issue!

hazarjast commented 3 years ago

@trunet I also don't understand why the same openvpn logic applied to wireguard doesn't work (I think it's a bug) @fraenki I made it work that way: with the 21.7.b version (type development) ... I hope it will be useful to someone

I can confirm that manually creating the NAT port forward firewall rule, including the reply-to, fixed the port forwarding issue via Wireguard!

I could not figure out why Transmission kept saying my listening port was closed. It worked via WAN, but stayed closed via the WG interface. I even used Wireshark and could see Transmission communicating back an forth with 87.98.162.88. There was just some traffic that didn't happen when using the WG interface. I'm so lucky I stumbled upon this github issue!

I would agree with @amonhk that this has to be a bug/oversight for wireguard interfaces when it comes to auto-associated Port Forward NAT firewall rules. I cannot think of any good reason why wireguard should behave differently from openvpn in this regard. The fact that it does behave differently under OPNSense is maddening. I am on 21.7.3_3 community release with os-wireguard 1.7 and this issue is still present. The workaround of manually adding the firewall rule with 'reply-to' set as the wireguard gateway is confirmed working for me. If not for finding the replies here, I would have become completely bald from pulling my own hair out troubleshooting this issue XD

Thanks to @amonhk and @Nikotine1 for taking the time to post and confirm the 'bandaid' for this. Hopefully this behavior can be corrected in OPNSense releases going forward. Since this issue auto-closed and appears to no longer be receiving attention since ~June I Have opened a new bug report here: https://github.com/opnsense/core/issues/5307 to summarize the issue and hopefully draw fresh attention to it.

Cheers!