Some charts appear limited to only listen on primary interface

[Zaephor]

Use Case:

I've configure TrueNAS into an overlay network, and trying to run the "official" minio app to provide S3 services to the overlay network. My host essentially has 2 interfaces, enp6s18 and mesh-overlay. From limited testing, looks like the details of the overlay network don't matter.

Issue:

The chart seems to offer 9000 and 9002 on the enp6s18 interface only, it's not listening on lo or my mesh-overlay. Tried using nginx-proxy-manager hoping to have it listen on mesh-overlay to bridge back to minio, but it turns out to have the same issue.

Reproduction:

Simply deploy the minio app, there's no networking configuration options.

SSH to the host, or try from GUI shell:

curl http://<enp6s18-ip>:9002 - Sucess, returns HTML
curl http://127.0.0.1:9002 - Failed to connect, "Couldn't connect to server"

Extra test, using a dummy interface:

ip link add dummy0 type dummy
ip addr add 169.254.1.1/32 dev dummy0
ip link set dummy0 up
curl http://169.254.1.1:9002 - Failed to connect, "Couldn't connect to server"

Comparison/testing:

I spot checked Plex, Jellyfin, Minio and nginx-proxy-manager. Plex/32400 and Jellyfin/8096 respond on all network interfaces - lo, enp6s18, mesh-overlay and dummy0 minio/9000,9002 and nginx-proxy-manager/30020,30021,30022 - only responds on enp6s18

---

I have Plex and Jellyfin configured with "host networking" enabled so that local discovery works properly, this option appears to be missing from Minio and nginx-proxy-manager. I would assume other charts are also demonstrating similar limitations.

[stavros-k]

Hello, some apps are on purpose not exposing option for host networking. This is because their internal ports (which will be used when host network is enabled) are hardcoded and non changeable.

For example, nignx proxy manager uses 80,81 and 443. 80 and 443 are by default consumed by the TrueNAS WebUI. Allowing hostNetwork for this app, means that a user could be potentially locked out of the webui.

There also apps that bind to a single interface, even with host networking turned on. AFAIK MinIO is not one of them.

I quickly looked over MinIO app and I see host network being enabled automatically when distributed mode is enabled. I'll leave this issue open and I can take a look if we can also expose a toggle to enable it without distributed mode too.

[Zaephor]

Thanks for replying, the safety concern makes sense to me.

nginx-proxy-manager was mostly chosen as the first tool I found that could probably relay traffic within the host, before heading down the rabbit hole of looking at truecharts/nginx-proxy-manager or truecharts/traefik. I didn't realize nginx-proxy-manager's configuration appears to have ports 80/81/443 hardcoded by default(never used it before). And I mostly mention the host networking for reproduction/comparison purposes, but I generally hate enabling host networking in my other environments.

Is host networking the only way to bind to multiple interfaces in a truenas scale environment? I mean, I'd prefer that hostnetworking not be the main solution, but I assume it would be the quick/easy choice. My understanding is stuck somewhere between docker-compose and knowing just enough k8s to recognize some CRD names in a vocabulary test, so I probably just don't understand enough k3s to recognize why it's behaving as it is.

[stavros-k]

There is a way to attach one or more interfaces to a pod, but I'm not sure if your overlay interface will be available.

Can you check on "Launch Docker Image" (Bluefin) or "Custom App" (Cobia), if under Networking, you can see the interface listed?

[Zaephor]

@stavros-k yep, it's there, and my "dummy" interfaces from playing around. Looks like it's filtering out the lo and any docker/k3s interfaces correctly.

I guess in hindsight I could/should've simply launched the official minio container directly, so sorry for adding to anyone's to-do list. After reading the couple release notes about the "S3 Service" being deprecated(never used it in the past), and instructions telling users to migrate to the minio chart, I think I just assumed it was behavior parity with the existing storage protocols(SMB/NFS).

[stavros-k]

@stavros-k yep, it's there, and my "dummy" interfaces from playing around. Looks like it's filtering out the lo and any docker/k3s interfaces correctly.

I guess in hindsight I could/should've simply launched the official minio container directly, so sorry for adding to anyone's to-do list. After reading the couple release notes about the "S3 Service" being deprecated(never used it in the past), and instructions telling users to migrate to the minio chart, I think I just assumed it was behavior parity with the existing storage protocols(SMB/NFS).

Okay, in this case we can look into adding interface selection on per chart basis. I'll leave this open until its resolved.

[T-One]

Okay, in this case we can look into adding interface selection on per chart basis. I'll leave this open until its resolved.

This is something i need too, like the nginx proxy manager can only listen on the main interface but my second interface has a vlan for external access, it the moment i cannot route external traffic to the proxy manager.

[Matthew-Jenkins]

Something hinky with nginx-proxy-manager. Since the admin page is not secure I was trying to proxy it. However it just hangs if I try.

[stavros-k]

With TrueNAS SCALE 24.10, running docker, it should listen to all interfaces by default. Later will also be added options to select specific interfaces to listen.