search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
4.02k stars 3.49k forks source link

mwan3: port forwarding not working on wan1 but working on wan2 #14266

Closed thomaslunze closed 3 years ago

thomaslunze commented 3 years ago

I am running mwan3 with dual wan (wan1 / wan2) and have tried to eliminate the issue that port forwarding to an internal lan ip is not working as expected.

e.g. port forwarding to my internal TS3 server is working on wan2 but not on wan1. Its not related to a particular service (both are running on the same provider (telekom) with a public ipv4 APN and different DDNS)

i can attach to my secure ftp server on port tcp 22 and also to my TS3 server on UDP 9987 using wan2. I have also tried with webmin on tcp 11000 to wan2 without issues. but on wan1 none of the services do respond. 👎 must be a firewall/routing issue with mwan3

while troubleshooting I have swapped routers, interfaces , firewall rule etc.. and I finally giving up and seek for some valuable help.

below please find the most common config files you would need. How would you start from scratch on this?

firewall.txt mwan3.txt network.txt

Software-Version

OpenWrt - OpenWrt 19.07.5 r11257-5090152ae3 LuCI - git-20.348.38488-caae7ad

aaronjg commented 3 years ago

It may be a conntrack issue and related to #13909. Can you use tcpdump to see if the connection is initiated to the internal service, and if so see how the response is routed?

Does this work if mwan3 is disabled?

jamesmacwhite commented 3 years ago

I have recently added a specific note about conntrack. The option has been missing entirely from the docs for a while, but I think a few inbound traffic issues are likely related to it. Certainly inbound ICMP, I wouldn't be surprised if port forwarding is an issue as well.

https://openwrt.org/docs/guide-user/network/wan/multiwan/mwan3#flushing_conntrack

The one thing to try is when doing port forwarding and having multiple external IPs is make sure in your port forward rules for IPv4, you target the specific external IPv4 address in the rule. I couldn't see any src_dip reference in your DNAT rules. That might help.

As @aaronjg has said though, using tcpdump on your WAN interface and filtering it to the port of your port forward rules will tell you if the traffic is being received and where the reply is going over. The working theory would be, it might be hitting the right WAN interface but the reply then goes over the wrong interface.

thomaslunze commented 3 years ago

I am not good enough in reading the tcpdumps to identify the problem but have tried to run it on each wan interface and for the example of webmin on tcp 11000 for now.

i do see traffic on both wan interfaces coming. attaching the tcpdumps from wan1 and wan2 for tcp 11000 forwarding (wan2 responding fine. wan1 not responding)

also attaching tcpumps for mwan3 being disabled. then no forwarding does work at all.

wan1__not_working_11000_mwan3_enabled.txt wan1_11000_mwan3_disabled.txt wan2_11000_mwan3_disabled.txt wan2_working_11000_mwan3_enabled.txt

thomaslunze commented 3 years ago

if you need anything more just let me know. would be great if you then could give specific commands/optione you would like to see 👍

aaronjg commented 3 years ago

I just looked through the config files you posted. It appears you don't have a gateway or netmask set for WAN1. Is this deliberate? Is WAN1 working as an outgoing interface for router initiated traffic?

You can try: mwan3 use wan1 ping 1.1.1.1 to test.

thomaslunze commented 3 years ago

network interfaces do both have mask and gw. Both Interfaces are working as expected for incoming / outgoing traffic. Its just that portforwarding is only working on wan2 but not on wan1 regardsless if mwan3 is enabled /disabled.

config interface 'wan2' option proto 'static' option ifname 'eth0.3' option ipaddr '192.168.3.2' option gateway '192.168.3.1' option metric '20' option delegate '0' option macaddr '16:91:82:C2:82:CC' option netmask '255.255.255.0'

config interface 'wan1' option ifname 'eth0.2' option proto 'static' option metric '10' option delegate '0' option ipaddr '192.168.8.2' option macaddr '15:91:82:C2:82:CB' option gateway '192.168.8.1' option netmask '255.255.255.0'

i have run this: root@OpenWrt:/etc/config# ping -I eth0.2 1.1.1.1 PING 1.1.1.1 (1.1.1.1): 56 data bytes 64 bytes from 1.1.1.1: seq=0 ttl=51 time=37.807 ms 64 bytes from 1.1.1.1: seq=1 ttl=51 time=38.771 ms 64 bytes from 1.1.1.1: seq=2 ttl=51 time=39.037 ms ^C --- 1.1.1.1 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 37.807/38.538/39.037 ms

root@OpenWrt:/etc/config# ping -I eth0.3 1.1.1.1 PING 1.1.1.1 (1.1.1.1): 56 data bytes 64 bytes from 1.1.1.1: seq=0 ttl=51 time=45.238 ms 64 bytes from 1.1.1.1: seq=1 ttl=51 time=45.052 ms 64 bytes from 1.1.1.1: seq=2 ttl=51 time=44.953 ms ^C --- 1.1.1.1 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 44.953/45.081/45.238 ms

mwan3 use wan1 ping 1.1.1.1 is not a valid command

aaronjg commented 3 years ago

I see. It looks like part of the config had been cut off in the file you posted earlier.

mwan3 use is a new feature, and not present on the 19.07 version of mwan3. However, there have not been major changes to the configs in regards to this, so I'd be surprised if the issue is resolved in the latest version.

Your configurations all look reasonable. I'll look into this more and see if I can reproduce the problem or come up with some more diagnostic commands for you to run.

thomaslunze commented 3 years ago

yeah that would be great. I do remember that my setup was working as expected for quite some time but some months (maybe even years) back it stopped working on wan1. I thought this would be an issue with the DMZ mode not working on my primary router connect to wan1 but as I have swapped devices now I figured out that its not the router but openwrt/mwan3 :-(

just let me know if you need something. I was also using unstable builds previously but also having the same issues even with newer versions of openwrt + mwan3 :-(

cheers and thanks for your help

ptpt52 commented 3 years ago

@thomaslunze could you test some firmware in x-wrt (https://github.com/x-wrt/build-release/releases) I use an custom fork of mwan3, which is very different implementation (https://github.com/x-wrt/com.x-wrt/tree/master/net/mwan3plus)

just test only as a comparison to determine if this problem exists?

feckert commented 3 years ago

@thomaslunze

Its just that portforwarding is only working on wan2 but not on wan1 regardsless if mwan3 is enabled /disabled.

Is this really a mwan3 problem then?

thomaslunze commented 3 years ago

@thomaslunze could you test some firmware in x-wrt (https://github.com/x-wrt/build-release/releases) I use an custom fork of mwan3, which is very different implementation (https://github.com/x-wrt/com.x-wrt/tree/master/net/mwan3plus)

just test only as a comparison to determine if this problem exists?

I guess your version uses the new switch config right? So I would need to reconfig everything again. Just tried to install the factory.img but did not come up so had to boot alterbnative partition.

@thomaslunze

Its just that portforwarding is only working on wan2 but not on wan1 regardsless if mwan3 is enabled /disabled.

Is this really a mwan3 problem then?

not sure :-D I have just tried to reset one of the boot partitions to default but seems i cannot get the firewall working as expected. will try with a stock firmware now without any addons...

aaronjg commented 3 years ago

Is this really a mwan3 problem then?

not sure :-D I have just tried to reset one of the boot partitions to default but seems i cannot get the firewall working as expected. will try with a stock firmware now without any addons...

Either way, seems like a reasonable feature to have. Just a question of if this is a bug report or feature request :)

thomaslunze commented 3 years ago

Is this really a mwan3 problem then?

not sure :-D I have just tried to reset one of the boot partitions to default but seems i cannot get the firewall working as expected. will try with a stock firmware now without any addons...

Either way, seems like a reasonable feature to have. Just a question of if this is a bug report or feature request :)

It was working before as expected for years. Not sure if there is something wrong on my vlan switch config or anything else. Cannot get firewall on wan working even without mwan3 on stock openwrt... out of ideas.

@ptpt52 frustrated on your firmware as it resets to 192.168.15.1 rather than the default and also several luci items are just not there e.g. software ... sorry but this is useless and took me around an hour to figure out whats wrong as I have been expecting 192.168.1.1 as default LAN IP.... really? com on

ptpt52 commented 3 years ago

@thomaslunze sorry I forget to tell you the default lan ip is 192.168.15.1 and default login password is admin

BTW, does the mwan3 works to this issue?

thomaslunze commented 3 years ago

@thomaslunze sorry I forget to tell you the default lan ip is 192.168.15.1 and default login password is admin

BTW, does the mwan3 works to this issue?

guess what... after unbricking my device I was not happy enough to continue on this. sorry

ptpt52 commented 3 years ago

@thomaslunze

Please keep calm, you are only one step away from the truth This is just a misunderstanding because of the difficulty of communication x-wrt just uses a different default setting If you can, please test again

thomaslunze commented 3 years ago

Is this really a mwan3 problem then?

not sure :-D I have just tried to reset one of the boot partitions to default but seems i cannot get the firewall working as expected. will try with a stock firmware now without any addons...

Either way, seems like a reasonable feature to have. Just a question of if this is a bug report or feature request :)

I have checked some mails and it was definitely working on both wan interfaces in november 2019. I have swapped internet modems several times now and its always related to wan1 not responding to any firewall forwardings. ping is working on both interfaces from external but only wan2 r4esponds to forwardings to services on the openwrt device itself or internal lan ip services. will try to check on some more default releases tomorrow.

ideas are highly welcome...

aaronjg commented 3 years ago

ideas are highly welcome...

Here are some more things to look at to help diagnose.

1) When it used to work, did it ever work without mwan3, or was mwan3 always required? 2) I am a bit surprised that it is working on wan2 and not wan1 rather than vice-versa, since wan1 has a lower metric. Can you disable wan2 and see if traffic is then handled correctly on wan1 3) Looking at the tpcdump, I'm trying to make sense of all the ip address, could you help me fill in the missing links:

thomaslunze commented 3 years ago
  1. Not sure as I never ran it without mwan3
  2. Either interfaces can be down and the other one routes traffic fine. e.g. wan2 down - wan1 serves the lan with the internet connection

  3. 192.168.2.2 - wan2 internal 192.168.3.2 - wan1 - obsolete 192.168.8.2 - wan1 internal 37.83.150.99 - wan 1 public interface with ddns 37.82.89.209 - wan 2 public interface with ddns

my setup is like: public wan1 (37.83.150.99) --> LTE modem (192.168.8.1) --> wan 1 internal (192.168.8.2) --> LAN (192.168.1.0) public wan2 (37.82.89.209) --> LTE modem (192.168.2.1) --> wan 1 internal (192.168.2.2) --> LAN (192.168.1.0)

  1. .will check on this later.
  2. will check on this later.
  3. --> will check on this later.

I am still trying to get a stock firmware running with only wan1 and port forwarding

And a BIG thank you for all your support on this

thomaslunze commented 3 years ago

So finally it turns out to be a NAT loopback/reflection issue.

When testing my firewall port forwardings while being connected to a public VPN I can connect fine to both DDNS / wan interfaces on specified services.

When connected to LAN only then I can only connect to wan2 ddns ip successfully.

Question now is if there could be anything done to allow NAT loopback/hairpinning to both WAN Interfaces? I'm really sure this worked fine a year back with all this.

Or... is there any way to flip DDNS entries in case wan1 goes offline?

something like: 1) wan2 is mapped to DDNS wan2.dyndns.org 2) wan2 goes offline 3) wan1 is mapped to DDNS wan2.dyndns.org 4) so users could still connect from outside to the same ddns

Thanks again for all your patience

jamesmacwhite commented 3 years ago

Realistically. You shouldn't need to have anything happen with DDNS. You are wanting a DNS round robin, where by you have two port forward rules over both WANs exposing whatever service, your DDNS record has your two external IP addresses and then each DNS lookup for your DDNS hostname will return either external WAN IP randomly.

Why not change the DDNS name internally to respond with the LAN IPs, rather than external for LAN access. Saving you having to query the external WAN IPs, you could use dnsmasq to make the request inside the LAN return the LAN IPs.

Not sure this is a mwan3 problem at this point. More general setup.

thomaslunze commented 3 years ago

Yeah makes sense. By using the ddns on both wan interfaces I would test the real usecase like coming from external. But I guess I am fine now with just enabling vpn for those test cases.

Thanks a ton and enjoy your holidays everyone

aaronjg commented 3 years ago

Question now is if there could be anything done to allow NAT loopback/hairpinning to both WAN Interfaces?

Perhaps. If I understand correctly, your router is not aware of the mapping between the LTE public IPs and the local IPs, so when you try to access public ip of wan2, it actually leaves the router through wan1, and then comes back in through wan2.

Does reflection work when you try to access via 192.168.8.2 and 192.168.2.2? That is what the reflection rule is really set up to do. You likely need to add some rules to make your router aware of the public IPs and handle reflection properly on those.

If you take a look at iptables -t nat -S, you can see how the existing reflection rules are set up, and can serve as a model for which additional rules you need to add.

Or... is there any way to flip DDNS entries in case wan1 goes offline?

Yes, take a look at /etc/mwan3.user. There you can set custom commands to run when interfaces come online or go offline.

thomaslunze commented 3 years ago

Reflection works fine on both Interfaces via 192.168.8.2 and 192.168.2.2. Will check the iptables and see how to adjust. Thanks so much

thomaslunze commented 3 years ago

I just figured out that both my LTE/5G Modems are capable of running in Bridge Mode which fully handsoff the public IP to the wan Interfaces :-D Now reflection knows about the public IP's and has no issues at all :-D Thanks a ton

deejayexe commented 2 years ago

@thomaslunze I know this thread is closed, but could you tell how you solved it, since I am in a similar situation with port forwarding in the wanb and explain how you put ddns for both wan