How to expose service with Cloud DNS?

kitwalker12 commented 9 years ago

hi. major n00b here. I setup the heat stack on my rackspace cloud. I see overlord, master, discovery and 3 minions are setup. I've also verified overlord setup. I deployed the guestbook example. I see that the service is running:

kubernetes-master ~ # /opt/bin/kubectl get services
NAME           LABELS                                    SELECTOR            IP(S)          PORT(S)
kubernetes     component=apiserver,provider=kubernetes   <none>              10.1.0.1       443/TCP
redis-master   name=redis-master                         name=redis-master   10.1.254.206   6379/TCP
redis-slave    name=redis-slave                          name=redis-slave    10.1.231.118   6379/TCP

How do I go about exposing the frontend service to reach it from a browser?

kitwalker12 commented 9 years ago

Also. What would be the process of adding new nodes to the cluster?

metral commented 9 years ago

to expose the frontend service you'll have to NAT the public IP to the k8s network. - unfortunatly, corekube is not intended to handle this, and there is no integration with Cloud DNS, so you'll have to do it yourself

metral commented 9 years ago

theoretically the process of adding more minions would just require the minion part of the template to be extracted and allow for receiving of the overlord's IP as a parameter to connect to, as the overlord just loops waiting for more minions to deploy, but, this feature too has not been implemented. this assumes that all minions are in the same region and cloud network. PR's are welcome

kitwalker12 commented 9 years ago

are there any guides on how to setup the NAT?

metral commented 9 years ago

there are not as this is specific & unique to each environment / policy you want to enforce

kitwalker12 commented 9 years ago

do you think just setting the publicIPs fields for the service wouldn't work. is kube-proxy part of this setup?

metral commented 9 years ago

the publicIP would not work off the bat without NAT manipulation of some sort as the whole K8s cluster runs on its own 10.244.0.0/15 subnet via Flannel, which is overlayed on top of a RAX Cloud Network / Isolated network (192.168.3.0/24) off of interface eth2, where the public interface / IP is on interface eth0. this can get messy and is not dynamically adjustable with just iptables.

yes, kube-proxy is installed on each worker/minion node.

your best bet is to check this out to see how google does it, but again, you'll have to implement it yourself as it is not built into Corekube: https://github.com/GoogleCloudPlatform/kubernetes/blob/v1.0.0/examples/guestbook/README.md#accessing-the-guestbook-site-externally

ChiragMoradiya commented 8 years ago

Is there a plan to support exposing kubernetes services via Openstack Load balancer as described in the following article?

http://docs.openstack.org/developer/magnum/dev/dev-kubernetes-load-balancer.html

metral commented 8 years ago

Note: This is in regards to the current/latest k8s stable release v1.0.6 which is being used for new deployments from the master branch of Corekube

@ChiragMoradiya The LoadBalancer type in k8s is still not supported for Rackspace environments even though some boilerplate code for it seems to exist, see https://github.com/kubernetes/kubernetes/blob/v1.0.6/pkg/cloudprovider/rackspace/rackspace.go#L423-L425

As far as it being supported on native OpenStack, it seems to be enabled but this requires & assumes that your openstack deployment has access to DNSaaS (i.e. Designate) and it is configured accordingly within your org's infrastructure, see https://github.com/kubernetes/kubernetes/blob/v1.0.6/pkg/cloudprovider/openstack/openstack.go#L430-L453

Once the LoadBalancer type hits a future stable release for the Rackspace provider pkg, then it will natively work and be available on Corekube once I update it to deploy that version of k8s. As far as supporting it on OpenStack natively once it is available, it will not be supported through Corekube due to the limitation or lack of insight into how your particular org manages DNS and would just be too difficult to support. (note: networking is hard in general and it gets even more complex as you start to dive into enabling it for containers - corekube is aimed at helping you get past the major stuff around infrastructure setup, but it's certainly not perfect and it can't adapt itself to any and all environments)

As a workaround for Rackspace specific environments (i.e. public cloud), you must:

Use a NodePort type for your k8s service and let it create a random port or supply your own - it'll state what the port is if you don't specify one i.e. 32306
Manually create a load balancer for port 80 and add the ServiceNet IP's of each minion host as well as the port created (i.e. 32306) by the NodePort type to the nodes the loadbalancer manages
Configure iptables rules on each minion host to do port forwarding from port 80 to the NodePort type port created, i.e:

iptables -t nat -A OUTPUT -d 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to-port 32306

Then hit the loadbalancer IP and it should forward to the minions accordingly

The workaround is not optimal, but until the LoadBalancer type is fully supported on Rackspace this is the only way per Google's suggestion: https://github.com/kubernetes/kubernetes/blob/v1.0.6/docs/user-guide/services.md#type-nodeport

metral / corekube

How to expose service with Cloud DNS? #18