openshift: cannot access (remote) service using portal address from node

[ibotty]

Using the multitenant plugin on atomic hosts (using openvswitch on docker), openshift v1.1, I cannot access a service portal address (172.30.183.139:5000) on the host.

curl 172.30.183.139:5000

times out.

curl 172.30.183.139:5000 --interface tun0

works.

as does using the endpoint (10.1.2.13) directly.

When adding a route to the portal net, all requests are working.

ip r add 172.30.0.0/16 dev tun0

The relevant part of the (generated) nat table looks like that.

-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -s 10.1.0.0/16 ! -d 10.1.0.0/16 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4d415351 -j MASQUERADE
-A KUBE-SEP-FIQ6M4FASARUD5ON -s 10.1.2.13/32 -m comment --comment "default/docker-registry:5000-tcp" -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-FIQ6M4FASARUD5ON -p tcp -m comment --comment "default/docker-registry:5000-tcp" -m tcp -j DNAT --to-destination 10.1.2.13:5000
-A KUBE-SVC-ECTPRXTXBM34L34Q -m comment --comment "default/docker-registry:5000-tcp" -j KUBE-SEP-FIQ6M4FASARUD5ON
-A KUBE-SERVICES -d 172.30.183.139/32 -p tcp -m comment --comment "default/docker-registry:5000-tcp cluster IP" -m tcp --dport 5000 -j KUBE-SVC-ECTPRXTXBM34L34Q

[danwinship]

This is by design; the service IP addresses are for use by pods, not nodes. Why do you need to connect to a service IP address from a node?

[sdodson]

@ncdc ^^ comment seems like it'd mean we can't use the kube service ip as a resolver, no?

@danwinship is that limitation multitenant only?

[danwinship]

We can certainly change this; as the bug report says, it's just a matter of ip r add 172.30.0.0/16 dev tun0. I just didn't think there was any reason to...

[ibotty]

My openshift cluster provides services also to the nodes (e.g. rpm-ostree remote, centralized logging, etc). Of course I could expose some of these services via routes. It also makes debugging easier ;).

[ibotty]

Also: how do pods with host networking (e.g. the router) work? They can't access portal ips either, right?

[ncdc]

@sdodson

comment seems like it'd mean we can't use the kube service ip as a resolver, no?

garrrrrrr. @danwinship we are hoping to have the nodes' /etc/resolv.conf point at the kube service ip for DNS

[danwinship]

OK, so I'll make this work again...

[danwinship]

(It didn't intentionally get broken, it just hadn't occurred to me before that anyone expected it to work.)