tailscale doesnt forward all exposed ports on host machine

[machinsk]

Tailscale has issues displaying hosted web UI from other chart services, even the TrueNAS UI by default. I had to check 'HostNetwork' to reach the TrueNAS UI from inside the tailnet, other services seem to be random if the port they are hosted on will be forwarded to the tailnet, as if there is a limit or something.. here is my nmap from inside and outside the tailnet for the trueNAS machine:

inside tailnet:

~ % nmap -p1-65535 truenas
Starting Nmap 7.94 ( https://nmap.org ) at 2024-02-20 01:03 MST
Nmap scan report for truenas (100.85.83.77)
Host is up (0.00080s latency).
rDNS record for 100.85.83.77: truenas.tailfa906.ts.net
Not shown: 65510 closed tcp ports (conn-refused)
PORT      STATE SERVICE
53/tcp    open  domain
80/tcp    open  http
111/tcp   open  rpcbind
139/tcp   open  netbios-ssn
443/tcp   open  https
445/tcp   open  microsoft-ds
4711/tcp  open  trinity-dist
6000/tcp  open  X11
6443/tcp  open  sun-sr-https
6444/tcp  open  sge_qmaster
6999/tcp  open  iatp-normalpri
10010/tcp open  rxapi
10248/tcp open  unknown
10250/tcp open  unknown
10257/tcp open  unknown
10259/tcp open  unknown
20244/tcp open  unknown
20720/tcp open  unknown
29642/tcp open  unknown
29643/tcp open  unknown
29644/tcp open  unknown
29652/tcp open  unknown
29653/tcp open  unknown
34547/tcp open  unknown
50051/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 9.42 seconds

outside tailnet:

~ % nmap -p1-65535 192.168.10.142
Starting Nmap 7.94 ( https://nmap.org ) at 2024-02-20 01:04 MST
Nmap scan report for 192.168.10.142
Host is up (0.00032s latency).
Not shown: 65512 closed tcp ports (conn-refused)
PORT      STATE    SERVICE
53/tcp    open     domain
80/tcp    open     http
111/tcp   open     rpcbind
139/tcp   open     netbios-ssn
179/tcp   open     bgp
443/tcp   open     https
445/tcp   open     microsoft-ds
5357/tcp  open     wsdapi
6000/tcp  open     X11
6443/tcp  filtered sun-sr-https
10250/tcp open     unknown
20244/tcp open     unknown
20720/tcp open     unknown
20810/tcp open     crtech-nlm
29642/tcp open     unknown
29643/tcp open     unknown
29644/tcp open     unknown
29652/tcp open     unknown
29653/tcp open     unknown
30041/tcp open     unknown
32400/tcp open     plex
34547/tcp open     unknown
50051/tcp open     unknown

Nmap done: 1 IP address (1 host up) scanned in 6.37 seconds

[stavros-k]

The reason that some apps work and some others not (without hostnetwork), is that some apps allow us to change the container port and map it to the same number as the NodePort (host's port), while others not.

This port is also reflected in the cluster's network,

For example. App X allows us to change container port.

• You select port 30000 as NodePort, and the same port gets defined as container port.
• In-cluster requests (eg tailscale to app X) on port 30000 will work

App Y does not allow us to change the container port

• You select port 30000 as NodePort but the container port remains unchanged (eg port 80).
• In-cluster requests (eg tailscale to app X) on port 30000 will NOT work, but if you use port 80 it will.

[machinsk]

@stavros-k is there a way to unify the port allocations?

My understanding is that: App X is ran in docker container with port A, which is inside a Kubernetes node with port B, if A does not map to B inside the node, Tailscale does not expose the correct port since Tailscale would be mapping port B.

If the TrueNAS UI is only changing port B, could I attach to the pod console and change the mapping of the docker container to the pod? I've have a good amount of experience with docker but I'm new to kubernetes.

[stavros-k]

@stavros-k is there a way to unify the port allocations?

My understanding is that: App X is ran in docker container with port A, which is inside a Kubernetes node with port B, if a A does not map to B, Tailscale does not expose the correct port since Tailscale would be mapping port B.

If the TrueNAS UI is only changing port B, could I attach to the pod console and change the mapping of the docker container to the pod? I've have a good amount of experience with docker but I'm new to kubernetes.

Hmm, now I re-read what I typed before, I realize that while most of it is correct (the part that not all containers allows us to change the internal port). It shouldn't affect connectivity. Internal container port should not matter in any case.

The whole communication happens at the Service level. This should and always match the port you define on the web-ui.

Can you please give me 2 examples (1 app that works and 1 that does not). Please also specify which ports and ip or dns names you tried to access them!

---

There are 3 ports into play here. Container port <-> Service Port <-> Node Port

Anything inside the cluster trying to access an something inside the cluster, will use the Service Port. Anything outside the cluster trying to access something inside the cluster will use the Node Port

Only the service it self will access the Container Port

Photo for example

[machinsk]

@stavros-k Okay, there was a couple abstractions I didn't know about, this is helpful.

So pihole works, on port 20720 and Immich does not work on 9001, it has another port by default but it didn't work either.

Tailscale's current settings: pihole's settings: Immich settings: Apps running: Current nmap mappings (red appear in both):

[stavros-k]

Yea pihole runs with hostNet enabled, and since you have the host net enabled on TS too, this works.

For Immich, can you try exec 'ing into the tailscale container and try curl`'ing immich?

[machinsk]

the container shell doesn't have curl, nor apt, and I wouldn't know what ip to curl in respect to the container..

[stavros-k]

Try wget, use the IP that the magicDNS resolves to, (I assume it is the NAS IP?)

[machinsk]

Its reachable. I guess I'm just kinda dumb. Most people would get around this issue with a subnet. I didn't realize that my server, although it is hosting the subnet, its local ip address is also accessable on the subnet and therefore those services are acceessable on the tailnet.. although it would be nice to use the MagicDNS name for the server to access the services, no big deal though. I'm sure a DNS could be manually setup and added.

[stavros-k]

I'll try to setup something as close as possible, locally (a vm or something), to isolate changes from truenas, see if the issue is on infra setup or somewhere else.

I'll report back once I have something.

[machinsk]

Immich port vs the nas default tailnet ip vs subnet ip of nas this is basically showing what we saw with nmap

[sync-by-unito[bot]]

➤ Stavros Kois commented:

Hello, please see if you can reproduce it on TrueNAS 24.10 or later. Feel free to open an issue if it still exists.

Thanks.