LAN access works, but port forwarding on router does not

[templedf]

I am launching the Docker container on my Mac as:

% docker run -d -it -e EULA=TRUE -p 19132:19132/udp --name mc itzg/minecraft-bedrock-server

From my LAN I can enter the world using my Android device no problem. I tested will all of the machine's IP addresses, and they all work.

If I enable port forwarding on my router for port 80, I can connect to a web server running locally on my Mac using the public IP address of the router.

If I enable port forwarding on my router for port 19132, I cannot connect to the Minecraft server running on my Mac using the public IP address of the router. I have tried it with both UDP and TCP+UDP forwarded on the router and on the container, with no luck.

Port forwarding on the router is working to my Mac, and the Minecraft server is accessible from the LAN. The issue is only when I try to connect to the Docker container through the router. I don't see any reason why double port forwarding would fundamentally be a problem.

The next thing for me to try will be grabbing a container with a simple web server to see if I can access port 80 through the router. In the meantime, any other suggestions would be appreciated!

Thank you for all the work you put into this project!

[templedf]

I just tested with a simple web server, and it works, but it's extremely slow. It takes 7-12 seconds to get a response. The interesting thing is that as soon as the web server logs the HTTP request, I see the response, so the issue is that it's taking forever to route the request. I do not see that delay when forwarding to a web server running directly on my Mac.

The Minecraft client is giving up in what seems like exactly 7 seconds, so it's very likely that the issue is that routing delay. That routing delay only exists when I'm port forwarding at the router and the container. I'll keep digging, but this is sounding like a Docker issue. Any insights would be much appreciated!

[itzg]

It does sound like it's something about Docker and/or the router interaction. I'll do some debugging myself with my router. I mainly use Java edition myself and port forwarding for that works great...but then it's a TCP protocol rather than UDP.

[itzg]

...FWIW port forwarding via my public IP address worked fine for me. My router is an Asus RT-AC66U B1 and Docker is running on Ubuntu 20.04 with the version info:

Server:
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.12
  Git commit:       77e06fd
  Built:            Mon Jun  8 20:24:59 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Port forwarding config in my router looks like

[itzg]

...column headers would help

[templedf]

After a bit more digging, it looks like the issue is in the router. If I watch with tcpdump, I can see that the packets aren't making it to my host. I also noticed that there's a pattern. The first packet goes through fine, but all subsequent packets get delayed. If there's at least a 20 second pause between packets, the next packet will go through fine again. It doesn't matter whether the server is in a container or running directly on the host. I'm going to raise a ticket with my ISP, and I'll let you know if I learn anything useful.

[templedf]

I've learned a couple more things. First, the delay between packets is 21 seconds. The reason I was seeing variance was that it's 21 seconds from the first packet until the next packet is allowed through. If I send a second packet t seconds after the first, it will be delayed by 21 - t seconds. Second, if I start an HTTP server in a container and enable port forwarding, others who are outside my LAN are able to access it without a problem. It's just me seeing the delay when I'm coming from inside my LAN. I was also able to verify that using my smart phone disconnected from the LAN. Third, If I try that same trick with my smart phone and Minecraft, the client tells me I can't access a dedicated server over a mobile data connection. I tried to trick it by having my smart phone host a hot spot (while disconnected from the LAN) and using a second smart phone to access Minecraft via the hot spot, but the client just told me it cannot connect to the world. I can't tell if that means the problem still exists for Minecraft but not HTTP, or that having a mobile data network in the middle was an issue. I'm looking for a friend with Minecraft who can help me debug. I should have further progress on that this week.

[maxfield-allison]

Having similar issues, following and up to assist

[templedf]

Here's my latest update. I now have my modem running in IP pass-through mode, where it passes all inbound traffic directly to my mesh, and the mesh has port forwarding configured. In this configuration, no packets that originate from my network make it through, though requests that originate from outside of my LAN can connect without issue. I'm still trying to line up a friend with Minecraft to test with me. (The challenge is that all of my "friends" with Minecraft are actually my son's friends, which makes them 8-10 years old and highly unreliable! 😄)

I've spoken with tech support for my ISP, who actually did try to help me work through it. They'd never heard of an issue like this and ultimately didn't have a better solution than trying IP pass-through mode. I think it's very safe to say, at this point, that this has nothing to do with the Docker image.

[itzg]

As a data point, when I was using the wifi router provided by Spectrum, I was having similar issues doing any routing from my computer on my LAN towards my public IP address and back again to a Docker container running on an Ubuntu server also on my LAN. I later realized they were charging a monthly fee for that router, turned it in, and with the Asus router mentioned above it has worked ever since. The problem in my case seems to be called "hairpin routing" or rather the lack of support thereof. Basically the NAT routing of some routers isn't able to out through the NAT and loop right back through the NAT.

So, IP pass-through mode seems like a reasonable solution for a similar sounding problem. From the name, I'm guessing that bypasses NAT processing.

[templedf]

The hairpin routing thing is interesting. I'll have to look up to see if that's a known issue with my router or modem. Thanks for that pointer!

I just got someone to try connecting to my world from outside my LAN, and it failed. I have the modem in IP pass-through mode, forwarding to my mesh. The mesh is forwarding 19132 to my laptop for all protocols. The laptop is running the container with 19132/udp forwarded. I was watching on tcpdump, and didn't see any traffic go by during the trial.

After the test, I tried logging into my server from within my LAN using my laptop's internal IP address, and it worked fine. I was able to see the traffic on tcpdump. Note that if I repeat that same test using a web server, it works, which means it's something specific to the port range or the protocol. I'm pretty sure it has nothing to do with Minecraft or the container because I didn't see any traffic during the trial.

Next I think I'll try having the modem forward directly to laptop plugged into ethernet rather than to the mesh. I was really hoping that it was only the hairpin routing that was in the way. Now that I know it's not working from outside the LAN, I'm running out of ideas. It did just occur to me that maybe the modem isn't forwarding UDP traffic. I'll have to try exploring that, though I'm not sure how yet without writing code.

[itzg]

Yeah, I wonder if your particular router is just not handling UDP as well as it should. It's a relatively less common transport protocol, so could see that happening.

[templedf]

Good news! I fiddled around some more with IP pass-through on the router and was able to get it working. Previous the modem was in "default server" pass-through mode, which forwards all TCP and UDP traffic to a given LAN IP. I tried changing it to "passthrough" pass-through, which actually assigns the public IP address to a node on the network via DHCP. I used my phone as a hotspot and connected a second device to it, and I was able to log into my server. Yay!

Armed with my new knowledge of how things work, I decided to go back and try regular port forwarding. I disabled IP pass-through and re-enabled port forwarding. Then using a second device connecting through my phone's hotspot, I was able to connect.

It appears that the original issue really was the hairpin routing. It just took a long and winding journey to discover and prove it. Hopefully this adventure will be useful to someone else in the future.

[templedf]

Two little addendums. First, I just tried having port 80 on the modem forward to the mesh, and the mesh forward port 80 to my laptop, and it did not work. Looks like I'm going to have to have my laptop plugged into the DMZ to to allow public access. I can live with that.

Second, I want to give a plug for Sonic.net. They're my ISP, and their tech support is unreal. The tech I spoke with today did his best to help me with this issue within bounds of the what reasonably relates to my Internet service, and then offered to help me debug my out-of-bounds Minecraft issues this evening when his shift ends. I am almost always blown away by the quality of Sonic's tech support people and their willingness to get into the technical details. If you're in California, they're totally worth considering.

[itzg]

That's amazing customer support! Such a relief when you find someone helpful like that.

[itzg]

Good to close out the issue? And thanks for documenting all your findings along the way. Like you said, that should help any others that encounter similar problems.