search

mbruzek / layer-k8s

A repository for the Kubernetes charm implmeneted in the reactive framework.
9 stars 3 forks source link

Performance evaluation of kubernetes cluster #28

Closed eddytruyen closed 8 years ago

eddytruyen commented 8 years ago

We performed some performance measurements to compare a mongod instance in kubernetes against native mongod on ubuntu trusty. The performance overhead of kubernetes using the YCSB benchmark was on the high side. It was so high we believed there is something wrong with how I set up the kubernetes bundle using juju quickstart

First I noticed that I am not running the latest version, but version 1.7.1. I don't think that is the problem.

I think the problem is that each kubernetes unit runs an entire kubernetes master with controller/scheduler/etc... Is it possible to configure only one node as master and the other nodes only with the essential components kubelet, cadvisor and proxy?

Disclaimer: this first performance test is not realistic, as only one mongod instance is deployed. In fact it would make more sense to deploy a single docker container directly using docker run).

More info about the raw data of the performance test can be found here: https://github.com/eddytruyen/kubernetes_on_openstack/wiki/Performance-evaluation-of-single-mongod-pod-vs-a-native-mongod-instance

mbruzek commented 8 years ago

The reason we chose to run all the kubernetes services on each node was to increase availability of the master node. Since we are in the cloud any instance could be eliminated. We designed the kubernetes cluster to survive the loss of any node and still have a master available. As you pointed out this is not the optimal setup for performance testing. With the current charm code there is not a way to run in the mode you requested. We do plan to split the master and node out, once we can decide on the best way to maintain high availability of the master units. That work has yet to be scheduled, so if you are interested in helping we could collaborate on the design and implementation of this operation mode.

eddytruyen commented 8 years ago

I agree that this is a very nice feature of your work. In the scenario with automated shutdowns and restarts, itis really wonderful to see how kubectl runs and observes a consistent state. But making all nodes master replicas is overdoing it a bit, isn't it. How does your architecture scale towards a high number of nodex.

Kubernetes has some guidelines for high availability cluster setups: http://kubernetes.io/docs/admin/high-availability/

Thanks for inviting me to collaborate, but I don't have the time for diving into the internals of kubernetes and juju. I am evaluating it for its suitability for SaaS.

eddytruyen commented 8 years ago

The strange thing of our performance evaluation was that the hyperkube process on the follower nodes ate 20 % of the cpu while the performance test was a set of write request to a service of type NodePort, with one pod placed at such follower node.

eddytruyen commented 8 years ago

It seems that the largest part of the overhead comes from the docker container and not Kubernetes.

Please find the data of the performance evaluation here: https://github.com/eddytruyen/kubernetes_on_openstack/wiki/Summary:-Performance-evaluation-of-Kubernetes This seems quite reasonable for a highly-available kubernetes cluster. I therefore would like to close this issue.