search

opnsense / core

OPNsense GUI, API and systems backend
https://opnsense.org/
BSD 2-Clause "Simplified" License
3.38k stars 758 forks source link

NAT reflection problem #1417

Closed thomasnilsen closed 7 years ago

thomasnilsen commented 7 years ago

Take this example;

(22.33.44.55) WAN -- OPNsense -- LAN (10.0.0.0/24)

I forward TCP port 10000 from WAN to LAN 10.0.0.10 port 10000 I forward TCP port 10001 from WAN to LAN 10.0.0.11 port 10000 I forward TCP port 10002 from WAN to LAN 10.0.0.12 port 10000

This works great when I access my services from remote, but when trying to access from LAN subnet it does not work. I have enabled "Enabled (Pure NAT)" in reflection on the NAT rule. I have also used "Pass" as rule to make sure its passed. Tried "Disabled" as well of course, and nothing works.

I see in the firewall that there is a "Pass" logged "OUT on LAN", but it clearly does not work.

I just moved from pfsense where this worked well... Very happy with the migration except from this little bugger..! 👍

Not sure if this is relevant, but im running both WAN and LAN as vlan interfaces on two different interfaces.

I'm more than willing to provide tcpdump or perform other test tasks if needed.

I'm running "17.1.1 (installed)"

regards Thomas

AdSchellevis commented 7 years ago

Hi Thomas,

Can you grep the generated ruleset so we can check what's in there?

grep rdr /tmp/rules.debug

Best regards,

Ad

thomasnilsen commented 7 years ago

Hi,

I have changed/masked the public ip.

no rdr proto carp rdr on re0_vlan102 inet proto { tcp udp } from any to 15.157.232.18 port 3671 -> $homeLYnk rdr on { re1_vlan202 re1 } inet proto { tcp udp } from any to 15.157.232.18 port 3671 -> $homeLYnk rdr on re0_vlan102 inet proto tcp from any to 55.175.222.16 port 43443 -> $homeLYnk port 443 rdr on { re1_vlan202 re1 } inet proto tcp from any to 15.157.232.18 port 43443 -> $homeLYnk port 443 rdr pass on re0_vlan102 inet proto tcp from any to 15.157.232.18 port 8000 -> 10.0.202.201 rdr pass on { re1_vlan202 re1 } inet proto tcp from any to 15.157.232.18 port 8000 -> 10.0.202.201 rdr on re0_vlan102 inet proto tcp from any to 15.157.232.18 port 8001 -> 10.0.202.202 port 8000 rdr on { re1_vlan202 re1 } inet proto tcp from any to 15.157.232.18 port 8001 -> 10.0.202.202 port 8000 rdr on re0_vlan102 inet proto tcp from any to 15.157.232.18 port 8002 -> 10.0.202.203 port 8000 rdr on { re1_vlan202 re1 } inet proto tcp from any to 15.157.232.18 port 8002 -> 10.0.202.203 port 8000 rdr on re0_vlan102 inet proto tcp from any to 15.157.232.18 port 8003 -> 10.0.202.204 port 8000 rdr on { re1_vlan202 re1 } inet proto tcp from any to 15.157.232.18 port 8003 -> 10.0.202.204 port 8000

PM me and I'll provide you with direct ssh login if you like.

best regards Thomas

AdSchellevis commented 7 years ago

I'm not at the office at the moment, but looking at the above rules, I seem to be missing the ones you mentioned in your issue (port 10000).

Are those addresses in aliases or defined on the rule? if they're in a rule, can you grep for the port number and see what pops-up?

Next thing I notice is that there are two interfaces referenced in a rule, but one of them doesn't seem to be in a vlan (re1). What is defined on re1?

thomasnilsen commented 7 years ago

In my initial post i said 10000, but it's 8000/8001/8002/8003

$homeLYnk is an alias to 10.0.202.200

The other addresses are "hardcoded" and not aliases. (But i have tried with aliases as well).

the IP address (15.157.232.18) is the WAN IP.

re0 is not in configured re0_vlan102 is WAN (15.157.232.18) re1_vlan202 is LAN (10.0.202.0/24) re1 is MGMT (10.0.1.0/24)

AdSchellevis commented 7 years ago

ok, tested this on my end, it seems to be missing an outbound nat rule, which confuses the receiving party. A simple workaround for now is to add an outbound nat rule for your lan network to nat the packages with your firewall's lan ip.

Goto Firewall -> NAT -> Outbound, add a rule containing the following: Source : your LAN network Destination : your LAN network Nat address: LAN address

thomasnilsen commented 7 years ago

That works! Thanks.

thomasnilsen commented 7 years ago

Found this under Firewall - Settings - Advanced.

Automatic outbound NAT for Reflection
Automatically create outbound NAT rules which assist inbound NAT rules that direct traffic back out to the same subnet it originated from. Required for full functionality of the pure NAT mode of NAT Reflection for port forwards or NAT Reflection for 1:1 NAT. Note: This only works for assigned interfaces. Other interfaces require manually creating the outbound NAT rules that direct the reply packets back through the router.

Should this be enabled? Can this be the cause of the bug, me not enabling this? It was disabled by default.

thomasnilsen commented 7 years ago

I tried to disable the "workaround" outbound nat rule now while enable this option. And it works.. 👍

AdSchellevis commented 7 years ago

@thomasnilsen I totally missed that piece of code :) let's close the issue.

githubatf2f10 commented 6 years ago

HI Guys, Can I get some help here? I have a similiar issue with opnsense... Here's forum post, https://forum.opnsense.org/index.php?topic=6696.0

here's details another forum, https://discourse.mailinabox.email/t/letsencrypt-expired-and-dns-errors/2704

Thanks in advance.....

githubatf2f10 commented 6 years ago

@AdSchellevis AdSchellevis

githubatf2f10 commented 6 years ago

configured options here. Network Address Translation Reflection for port forwards Enable (pure nat) Reflection for 1:1 Enable Automatic outbound NAT for Reflection Enable

NAT->Port Foward : NAT reflection use system default Filter rule association Rule NAT

Firewall: NAT: Outbound Mode Tried both Manual and Hybrid....

githubatf2f10 commented 6 years ago

These two are critical in doing reflection nat.... @TorWrt# iptables-save | grep NAT -A zone_dmz_prerouting -s 192.168.140.0/24 -d 76.10.176.225/32 -p tcp -m tcp --dport 53 -m comment --comment "dns (reflection)" -j DNAT --to-destination 192.168.140.253:53 -A zone_dmz_prerouting -s 192.168.140.0/24 -d 76.10.176.225/32 -p udp -m udp --dport 53 -m comment --comment "dns (reflection)" -j DNAT --to-destination 192.168.140.253:53

-A zone_dmz_postrouting -s 192.168.140.0/24 -d 192.168.140.253/32 -p tcp -m tcp --dport 53 -m comment --comment "dns (reflection)" -j SNAT --to-source 192.168.140.1 -A zone_dmz_postrouting -s 192.168.140.0/24 -d 192.168.140.253/32 -p udp -m udp --dport 53 -m comment --comment "dns (reflection)" -j SNAT --to-source 192.168.140.1

On openSense, we are missing this, like that on openWRT ?????

-A zonedmzprerouting -s 192.168.140.0/24 -d 76.10.176.225/32 -p udp -m udp --dport 53 -m comment --comment "dns (reflection)" -j DNAT --to-destination 192.168.140.253:53

@trumpwall:~ # pfctl -sn no nat on em0_vlan140 inet proto tcp from 192.168.140.1 to 192.168.140.253 port = domain no nat on em0_vlan140 inet proto udp from 192.168.140.1 to 192.168.140.253 port = domain nat on em0_vlan140 inet proto tcp from 192.168.140.0/24 to 192.168.140.253 port = domain -> 192.168.140.1 port 1024:65535 nat on em0_vlan140 inet proto udp from 192.168.140.0/24 to 192.168.140.253 port = domain -> 192.168.140.1 port 1024:65535

AdSchellevis commented 6 years ago

@githubatf2f10 If the automatic rules won't fit your needs, you can always try to use manual rules (like stated earlier in this issue). We are working on a rewrite of the rule engine, which has more priority at the moment.

One small tip when trying to get issues solved, try to describe your network situation step by step, as in where is your traffic coming from, where should it be going, etc, etc. Pointing to different resources and rules from very different products usually won't help moving things forward.

githubatf2f10 commented 6 years ago

@AdSchellevis Thanks a lot for taking a loot at this!

I have updated with new info on the other links.. anyway, I will post some briefly here. That screen capture is on the other link as well...

On openSense, we are missing this, like that on openWRT ?????

-A zone_dmz_prerouting -s 192.168.140.0/24 -d 76.10.176.225/32 -p udp -m udp --dport 53 -m comment --comment "dns (reflection)" -j DNAT --to-destination 192.168.140.253:53

@trumpwall:~ # pfctl -sn no nat on em0_vlan140 inet proto tcp from 192.168.140.1 to 192.168.140.253 port = domain no nat on em0_vlan140 inet proto udp from 192.168.140.1 to 192.168.140.253 port = domain nat on em0_vlan140 inet proto tcp from 192.168.140.0/24 to 192.168.140.253 port = domain -> 192.168.140.1 port 1024:65535 nat on em0_vlan140 inet proto udp from 192.168.140.0/24 to 192.168.140.253 port = domain -> 192.168.140.1 port 1024:65535

It looks like it has this Pre-routing on OpenSense as well...???!!! Then, this didnt' catch or function???

rdr on pppoe0 inet proto tcp from any to (pppoe0) port = domain -> 192.168.140.253 rdr on pppoe0 inet proto udp from any to (pppoe0) port = domain -> 192.168.140.253 rdr on em0_vlan140 inet proto tcp from any to (pppoe0) port = domain -> 192.168.140.253 rdr on em0_vlan140 inet proto udp from any to (pppoe0) port = domain -> 192.168.140.253

githubatf2f10 commented 6 years ago

I did a capture of what I have with opnsense as well. It looks like it has its Nat Reflection setting in the finalized rule set as well... Just don't understand why it didn't function like OpenWRT did. It seemed not being able to capture that packet and modify DNAT+SNAT and send that packet back; instead, it send out to PPPoE with DNAT only and that ships packet out to be discarded by ISP...

Here's what I see on PPPoE tcpdump on opnSense. SNAT is not done and the packet is out to WAN....

@trumpwall:~ # tcpdump -ni pppoe0 port 53

07:39:13.173835 IP 76.10.176.225.56965> 192.168.140.253.53: Flags [S], seq 333284484, win 29200, options [mss 1460,sackOK,TS val 1144622951 ecr 0,nop,wscale 7], length 0

on opnWRT, you don't see this traffic leaking out to WAN, as it's been properly reflected back on DMZ already.. @TorWrt:# tcpdump -ni pppoe-wan port 53 07:07:17.556020 IP 76.10.176.225.27726 > 192.203.230.10.53: 40886% [1au] NS? . (28) 07:07:17.556147 IP 76.10.176.225.53311 > 192.203.230.10.53: 20231% [1au] A? aspmx.l.google.com. (47) 07:07:17.556244 IP 76.10.176.225.10260 > 192.203.230.10.53: 57585% [1au] AAAA? aspmx.l.google.com. (47)

githubatf2f10 commented 6 years ago

my set up is like this..

Internet ---WAN(pppoe)--opnsense--several internal netowrks (one of them is DMZ). On DMZ, I have this email Server (MAIB).

There will be traffic leaving MAIB to it's own public IP(which is WAN ip) as this Mail server need to be accessed from outside. This requires NAT Reflection to help it to work,as in OpenWRT router's NAT Loopback. OpnSense has this NAT Reflection and it has in its rule set. However, the packet still leaked outward through PPPoE without an opportunity of Reflecting back out with DMZ interface ip. The DNAT part seemed being done, but SNAT part is not happening. That 's why we see packet with WANIP to DNAT private Mail server ip is shipped out to Internet ...

githubatf2f10 commented 6 years ago

hope this helps...this link has updated screen capture and diagnose details... https://discourse.mailinabox.email/t/letsencrypt-expired-and-dns-errors/2704

githubatf2f10 commented 6 years ago

here's an output of the rules, rdr seems working as it stated here... the post NAT part might not fuction as it should.../??? It looks like that OpnSense did DNAT and shipped packet directly to WAN, without redirecting packet back on DMZ with that 192.168.140.253.Therefore, Post NAT rule on DMZ (NAT Reflect) did not do SNAT to change it's source ip to DMZ ip......??? Is this a possible reason?

root@trumpwall:~ # grep rdr /tmp/rules.debug no rdr proto carp no rdr on em0_vlan110 proto tcp from any to ( em0_vlan110 ) port { 443 80 22 } rdr on pppoe0 inet proto tcp from any to (pppoe0) port 80 -> 192.168.140.253 rdr on { em0_vlan120 em0_vlan140 em0_vlan130 em0_vlan110 openvpn } inet proto tcp from any to (pppoe0) port 80 -> 192.168.140.253 rdr on pppoe0 inet proto tcp from any to (pppoe0) port 25 -> 192.168.140.253 rdr on { em0_vlan120 em0_vlan140 em0_vlan130 em0_vlan110 openvpn } inet proto tcp from any to (pppoe0) port 25 -> 192.168.140.253 rdr on pppoe0 inet proto tcp from any to (pppoe0) port 443 -> 192.168.140.253 rdr on { em0_vlan120 em0_vlan140 em0_vlan130 em0_vlan110 openvpn } inet proto tcp from any to (pppoe0) port 443 -> 192.168.140.253

rdr on pppoe0 inet proto { tcp udp } from any to (pppoe0) port 53 -> 192.168.140.253 rdr on { em0_vlan120 em0_vlan140 em0_vlan130 em0_vlan110 openvpn } inet proto { tcp udp } from any to (pppoe0) port 53 -> 192.168.140.253 rdr on pppoe0 inet proto tcp from any to (pppoe0) port 993 -> 192.168.140.253 rdr on { em0_vlan120 em0_vlan140 em0_vlan130 em0_vlan110 openvpn } inet proto tcp from any to (pppoe0) port 993 -> 192.168.140.253 rdr on pppoe0 inet proto tcp from any to (pppoe0) port 587 -> 192.168.140.253 rdr on { em0_vlan120 em0_vlan140 em0_vlan130 em0_vlan110 openvpn } inet proto tcp from any to (pppoe0) port 587 -> 192.168.140.253

githubatf2f10 commented 6 years ago

I tried manually add rules on DMZ, it seems not working as well.......?? maybe i didn't do it correct.. if you can help provide a detailed one for me to try...it would be appreciated...thanks

githubatf2f10 commented 6 years ago

@thomasnilsen Is your setting still working? Any thoughts of any setting that I could try? thanks...

thomasnilsen commented 6 years ago

@githubatf2f10

Hi, I found that all things worked after by changing some settings. See post from 21st of Feb this year.

githubatf2f10 commented 6 years ago

screenshot_20171226_081252 screenshot_20171226_081337 screenshot_20171226_081425 screenshot_20171226_081453 screenshot_20171226_081522 screenshot_20171226_081549 screenshot_20171226_081603

githubatf2f10 commented 6 years ago

@thomasnilsen @AdSchellevis

Here's my configuration screen captures.... It seems to me that Packet is DNATed from WAN IP to my intended Private IP;

rdr on pppoe0 inet proto { tcp udp } from any to (pppoe0) port 53 -> 192.168.140.253 rdr on { em0_vlan120 em0_vlan140 em0_vlan130 em0_vlan110 openvpn } inet proto { tcp udp } from any to (pppoe0) port 53 -> 192.168.140.253

At this moment, the packet 192.168.140.253--->192.168.140.253:53, for example, should follow Routing Table to be sent back to DMZ zone with 192.168.140.1 as DMZ interface....to be processed by SNAT into a packet like, 192.168.140.1-->192.168.140.253....

However, it is still sent out through WAN, with public ip as source, not being redirected back through DMZ to my Private IP.

@trumpwall:~ # tcpdump -ni pppoe0 port 53

07:39:13.173835 IP 76.10.176.225.56965> 192.168.140.253.53: Flags [S], seq 333284484, win 29200, options [mss 1460,sackOK,TS val 1144622951 ecr 0,nop,wscale 7], length 0 07:39:13.173835 IP 76.10.176.225.56965> 192.168.140.253.53: Flags [S], seq 333284484, win 29200, options [mss 1460,sackOK,TS val 1144622951 ecr 0,nop,wscale 7], length 0

Hence, the SNAT has never been applied, although I can see in the rule set, we have correct NAT Reflection rules and correct RDR rules...

@trumpwall:~ # pfctl -sn no nat on em0_vlan140 inet proto tcp from 192.168.140.1 to 192.168.140.253 port = domain no nat on em0_vlan140 inet proto udp from 192.168.140.1 to 192.168.140.253 port = domain nat on em0_vlan140 inet proto tcp from 192.168.140.0/24 to 192.168.140.253 port = domain -> 192.168.140.1 port 1024:65535 nat on em0_vlan140 inet proto udp from 192.168.140.0/24 to 192.168.140.253 port = domain -> 192.168.140.1 port 1024:65535

githubatf2f10 commented 6 years ago

Do we have a way of seeing how that packet is processed at each stage of it coming to OpnSense and leaving OpnSense, including routing decisions? I used tcpdump for interface capture to find out these issues, however, it's not clear still how each decision was made in opsense.

For example, I would like to see...

  1. Packet with src.ip->dst.ip coming to DMZ, now being processed by either SNAT rules or other rules,
  2. Packet resides in OpnSense, now being processed by NAT Reflection rules, and/or routing rules
  3. Packet leaves OpnSense at interface, now being processed by DNAT rules.
  4. if being dropped,denied,then by which rules/reason...etc.

That way, I can clearly see which rules/setting is doing what to that packet at each stage of its life traversing OpnSense. That will help us to clearly find out exact which step is the root of problem.

thanks in advance.

githubatf2f10 commented 6 years ago

@thomasnilsen @AdSchellevis

Any thoughts? do I need to open a new bug request? Please kindly let me know once you get some time.

thanks

AdSchellevis commented 6 years ago

@githubatf2f10 you better try the forum for support discussions

githubatf2f10 commented 6 years ago

@AdSchellevis I was asked to come to here from forum. ... _:( https://forum.opnsense.org/index.php?topic=6696.0

githubatf2f10 commented 6 years ago

Just reload OpnSense after trying pfsense; I noticed that one of my rules was using ICMP, instead of IPv4. Once it got fixed, Mailinabox checks everything fine. So, it's not the fault of Pure NAT, it's my fault of configuration.

That makes me wonder whether you tested it properly when you tried OpnSense. It looks configuration almost exactly with PfSense.

Thanks peng screenshot_20180218_055734