Router creation doesn't configure hostPort

[simon3z]

When running:

oadm router management-metrics \
  -n default \
  --credentials=/etc/origin/master/openshift-router.kubeconfig \
  --service-account=router \
  --ports='443:5000' \
  --selector='kubernetes.io/hostname=vm-oshift01.vn1.bytenix.com' \
  --stats-port=1937 \
  --host-network=false

The resulting dc is not configured with the hostPort (5000):

# oc get dc management-metrics -o yaml
 ....
        name: router
        ports:
        - containerPort: 443
          protocol: TCP

Version

origin-1.1.4-0.git.0.9d8230b.el7.centos.x86_64 oc v1.1.4 kubernetes v1.2.0-origin-41-g91d3e75

Steps To Reproduce

Provided above.

Current Result

The hostPort is not configured (it was in the past).

Expected Result

The hostPort should be configured:

        name: router
        ports:
        - containerPort: 443
          hostPort: 5000
          protocol: TCP

cc @deads2k

[liggitt]

Don't you need to set --host-network=true?

[simon3z]

Don't you need to set --host-network=true?

No, there's not need to use the host network (otherwise you can't do the redirection at all). Editing the dc manually (adding the hostPort) solves the issue.

[bmeng]

The --ports does not seem to open a specified host port, but is mapping a specified port to service.

oadm router --replicas=1 --service-account=router routerc --host-network=false --ports="80:10080,443:10443"

oc get svc routerc
NAME      CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
routerc   172.30.99.118   <none>        10080/TCP,10443/TCP,1936/TCP   2m

[0xmichalis]

Host ports default to container ports only when you use host network.

[knobunc]

@ramr Can you just validate that this is behaving as expected?

[simon3z]

For clarity --host-network=true is not a workaround or a viable alternative for three reasons:

1. you would need additional specific firewall rules to reach the port
2. you can't do a port redirection as we require: 5000 (of the host) => 443 (of the router pod)
3. exposing the full host network to the pod is not a best practice unless you really need it

The real (working) workaround is to manually editing the deploymentConfig of the router adding the relevant hostPort value.

This is a regression from the behavior we had in 3.1 (where hostPort was correctly added).

cc @deads2k

[ramr]

@simon3z this was specifically changed in commit: https://github.com/openshift/origin/commit/241d02b8e89d492ad680559e56bb1549c9985c34 to not expose host ports and run with lesser privileges (just hostnetwork scc for admin routers and just something that allows etcd view/list access to routes/endpoints for user routers]).

So in the case of running the router with host networking and the hostnetwork scc, you can already bind to those ports on the host network stack, so that one's a given. In container networking mode, we want to be able to run with a lesser set of privileges (aka as a regular and not admin user) and so the rationale out here is to not expose host ports as it requires a higher permission set.

So for your specific case, you can either do this via:

1. As you did you did - edit the router DC to use host ports. Of course the SCC that the router service account belongs to needs the permissions for that. Do note, for most users/service accounts, this may not be a viable operation (not permitted).
2. You could also achieve this by creating a router in container network mode and then exposing it via a NodePort service (port range is a bit restrictive 30000-32767 however).

Example:

oadm router --latest-images --host-network=false 
oc create -f
https://gist.githubusercontent.com/ramr/43f55948461df51cc8ffe01530af0ce7/raw/7ebab70c691692a2ecb5ad39db3aa092a65d35be/nodeport-router-yaml 

And in both cases, whether its port 5000 or as the above example goes 30405, you would still need to punch a hole in firewall for that.

[simon3z]

@ramr the oadm router help states:

--ports='80:80,443:443': A comma delimited list of ports or port pairs to expose on the router pod. The default is set for HAProxy

So, if I am specifying a hostPort I expect that to be honored (as it was in the past).

The fact that by default you want to use a service account that can't use an hostPort is unrelated, because the user can specify a different service account (--service-account) that has that permission.

So in the case of running the router with host networking and the hostnetwork scc, you can already bind to those ports on the host network stack, so that one's a given.

With --host-network=true you can't do redirection:

# oadm router test1 --host-network=true --ports='5000:443'
error: For host networking mode, please ensure that the container [5000] and host [443] ports match

And in both cases, whether its port 5000 or as the above example goes 30405, you would still need to punch a hole in firewall for that.

With --host-network=false you don't have to punch a hole in firewall because the nat table redirection takes place before filtering.

Current situation is:

• using --host-network=true I can't specify a different port with --ports='5000:443 (see above)
• using --host-network=false the ports I am specifying with --ports='5000:443 are not taken in account (even when using a service account that has access to hostPort).

For us this is a serious regression.

[deads2k]

@ramr @smarterclayton @knobunc This looks like a significant and breaking change to our CLI that makes previously simple flows harder. We've just invalidated the instructions for installing CloudForms without giving any easy replacement.

I see these choices:

1. leave it like it is in master. This breaks anyone who wrote directions against our old CLI and makes it impossible to write a single set of directions that works across multiple versions of openshift.
2. restore the old behavior and make new behavior work on different flags. This wouldn't break anyone. Was the main argument against that the flag names become confusing?

If we choose option 1, is it even possible to express the changes that are needed in terms of our oc patch CLI command or would we be forcing people down into the interactive oc edit mode?

[smarterclayton]

We intended to drop support for hostPorts on routers. Firewall punching seems one argument - can you describe why you need firewall punching?

[smarterclayton]

Actually, my bigger question is - how are you installing OpenShift? Is this for dev mode?

[deads2k]

We intended to drop support for hostPorts on routers. Firewall punching seems one argument - can you describe why you need firewall punching?

Previously, he was able to successfully install his router using the openshift API without having to make corresponding changes on the nodes themselves. Requiring a firewall update requires him to have the authority and the tooling to make that change as well.

[deads2k]

NodePort service (port range is a bit restrictive 30000-32767 however).

I'm pretty sure that a port other than 5000 defeats the purpose of the remapping that he's attempting to perform.

[smarterclayton]

Remapping routers to arbitrary ports has never been support as part of the product. We'll keep backwards compatibility on the behavior, but users should know this is not a supported path for other reasons (no way to know what ports the resources are on). Host ports are not a recommended way to do anything, although they're a useful tool in dev / test clusters. Routers running on host-ports is not supported officially, just within the bounds of "if you do things, they work".