Allow listening on an address that is different from the externally reachable address

[jdom]

It would be nice if Orleans supports listening in an address that is not necessarily the same one that is externally addressable. A very common example of this is when running in containers, where the Silo could be configured to bind to localhost and a fixed port, but then this is mapped into a virtual network and the host acts like a NAT.

So basically allow all the silos in their containers to listen on localhost:22222 and localhost:30000 but in their membership register with a distinct IP address and/or a different port that is passed in by the host (it should support changing ports, not just IP, since there might be many silos/containers running in the same host, and to avoid port conflicts with the host).

This might be a good candidate for doing when we replace the legacy ClusterConfiguration with the new IOptions-based configuration.

[benjaminpetit]

I think it is a good idea to have an option to listen to ANY, and make the IP address used in the membership table more configurable (fixed IP or better subnet configuration than today)

to bind to localhost and a fixed port

If you do that, the silo would be only accessible from inside the container. I guess you wanted to say listen to any (0.0.0.0)?

this is mapped into a virtual network and the host acts like a NAT

I don't think you want to use NAT here... port forwarding maybe, but it seems easy enough to just change the port used by the silo so it is easier to configure.

I am sure we can find use case where the container would not be aware of the "external" IP address, but I think in a lot of cases it would be simpler if all the container share the same network (or at least are all reachable from a network to another).

I would like also to remind that client-to-silo and silo-to-silo should NOT be exposed on a non-secure network/

[tudordid]

We are looking currently at running Orleans across active-active k8s clusters today, where each cluster has its own VNET for the nodes. For either a single Orleans cluster or a multi cluster deployment to work in this scenario, we plan to bridge the 2 VNETs and pass in the node ip (assigned from the VNET) vs the default kube proxy ip into each container so that the silo can register with that and be able to see silos in the "sister" VNET. Julian told me today about Orleans' current requirement for the silo IP to be one from an in-container network interface, so the plan for now is to look at creating a virtual network interface (as a copy of the kube proxy network interface) in the Docker containers with the passed in VNET ip and have Orleans use that.

[galvesribeiro]

Let me drop my 2 cents here...

When I started talking about containers and Orleans months ago in another issue, I was discussing with @reubenbond exactly about this situation where I suggested to change the gateway and the membership tables to store not just the internal IP, but the exposed as well.

After some days, debating that, we came to a conclusion that the major complication factor is that Orleans use IP to identify and route messages to a silo at message level and that was a big change todo when all our efforts were on 2.0.

This situation will happen not only on containers. It will happen on any kind of non-trivial networking which requires routing/NAT or something different than everything(client and silo) on the same vNET(remember, I know Orleans wasn’t designed for public networks but internal networks may require routing/nat/firewall and that will fail with current implementation).

With all that in mind, and considering that nowadays nobody name servers as pets(everything is randomly generated with a prefix), I would change Orleans on both membership and message addressing to use host names instead of IPs, listen to 0.0.0.0 (by default, and let the user to set a particular interface if they want to) and get free of IP addressing business. I mean, come on... we have DNS for ages, why we do use IP addresses nowadays?

I hope it help on the discussion.

[ReubenBond]

@galvesribeiro as long as the silo has the same identifier locally as it does remotely, then everything is fine. So the silo's SiloAddress has to be consistent and it has to be a single IP:Port combo which is used to connect to it. However, the silo doesn't necessarily need to be listening only on that address. The silo could advertise its identity as 171.22.0.1:45123 even though it's actually listening on 0.0.0.0:33500. I think that works with the container example - it need to know which address it's accessible on and which address to bind to, but those might be different addresses because of NAT/routing.

[veikkoeeva]

Related, so cross-referening: How to configure multi-homed servers? (see also links contained in the cross-referenced issue).