Run OpenWhisk on Kubernetes

[ghost]

This issue can be used to track the progress for running OpenWhisk on Kubernetes. For this issue, we plan on running a simple OpenWhisk deployment on Kube and learn what would be an easy way to achieve this goal.

Initial tasks that need to be accomplished:

• [x] Add a README to Document how to deploy OpenWhisk to Kubernetes and keep track of our thought process about the deployment strategy.
• [x] Deploy CouchDB within a Kubernets Pod
• [x] Deploy Consul within a Kubernets Pod
• [x] Deploy Controller within a Kubernets Pod
• [x] Deploy Kafka within a Kubernets Pod
• [x] Deploy Nginx within a Kubernets Pod
• [x] Deploy Invoker within a Kubernets Pod

Validation for OpenWhisk on Kube: As we go along, it would be nice to validate these deployments using travis, but I do not have any experience with setting up Kubernets on Travis. This will require some investigation on how to set up.

• [x] Set up Kubernetes on Travis.

For each check box above, we will submit a PR or a number of PRs and keep track of the work needed.

[baldwinSPC]

OK, I would suggest that we use Kubernetes Deployments where possible. I'm reading through the Docker Compose work in openwhisk-devtools and will convert it to a basic Kubernetes manifest. This will allow us to at least begin experimenting with OW + K8S.

[animeshsingh]

Thanks @baldwinSPC - once you are done please submit a PR. As for the community call, I have added in slack channel, and would send the email around to make sure the folks are informed and can attend. Also at your end please invite others

[animeshsingh]

@baldwinSPC you should probably have your folder named like 'compose-to-k8' just to distinguish it from what @DanLavine is doing. His work will be going under a folder named 'kubernetes'

[ghost]

hey @baldwinSPC I just submitted the first PR for this work here. What I'm thinking as a current deployment strategy is to essentially create a Kube Job that deploys OpenWhisk. This way we can try and reuse as much of the Ansible scripts or modify them for handling Kube specific setup.

• Seeding consul/couchdb, etc we can leverage Kube-DNS for internal routing for things of this nature.
• Seeding the invoker process with the Docker images we can use Kube's poststart hooks.
• Generated Ansible Configs will be mapped to Kube ConfigMaps and get applied to the required Kube Deployment Pods.
• Some things it might be easier to generate custom Docker images. For example, when setting up Nginx, it requires some custom code to exist at specific locations.

I was working on setting up a cluster with my colleague @kkbankol to see what it would take to have a decent deployment strategy and we were able to successfully deploy OpenWhisk on Kube. The only issues is that it was somewhat hacky and quit manual. We are currently taking what we learned and making more streamline for anyone to use. Also, solving some of the issues we didn't really care about along the way.

[markusthoemmes]

Out of curiosity: What are the plan to have the Invoker schedule its containers? Will that be through k8s APIs or by directly talking to the underlying docker host?

[animeshsingh]

In the current PR its directly talking to docker host - but phase 2 will be through K8 apis

[animeshsingh]

@markusthoemmes is it you or @csantanapr who are responsible for merging in this repo?

[csantanapr]

Hi @animeshsingh Any of our committers have the authority to merge PRd to any of the Apache OpenWhisk repos. Merging a PR is the easiest part since it's a mechanical step as to clicking a button.

The part that it's not as easy it's finding the time to do code reviews and design discussions, that's why it can take from a day to a few days to get something reviewed and approved for merging. To review a PR we usually try to ask for help to review the PRs to the SMEs on the topic, reviewers don't have to be committers and write access to the repo.

For invoker changes since it's one of the most critical components and sensitive to performance characteristics a few more folks would be interested to help out with design and reviews like @markusthoemmes @rabbah @jeremiaswerner and others.

In terms of deployment and orchestration @markusthoemmes is one the SMEs could also help with reviews

[mbehrendt]

but phase 2 will be through K8 apis

@animeshsingh why would we be doing that, given it's going to be intrinsically slower than directly talking to the docker host?

[animeshsingh]

@mbehrendt well - what`s the point of using orchestrator if we don't want to use it to orchestrate containers? talking to docker directly on host machine wont be any different to what the current distributed deployment model has..... Also generally you will be going on a shared kubernetes cluster, and and at that point we cant assume we have direct access to host machines

In a dedicated Kubes cluster - we can take some liberties - but then kubes might have a split brain issue where what it thinks of the available capacity is not the actual capacity etc

[markusthoemmes]

That's quite specifically why we haven't yet looked at Kubes in terms of our standard deployment mechanism. The container orchestration points are undeniable and make sense. For us though, fast and predictable container management is crucial and probably differs from what Kubernetes is doing scheduling wise (to be verified, this is merely guessing). Always keep in mind that OpenWhisk is all about shaving off as many milliseconds as we can.

I think it would be worthwhile to undermine this with numbers, i.e. what will the cold-start latency be for starting a new user-container of a certain kind and whether it's feasible to go down that route or if we need to find a different approach (for example talking directly with the underlying host).

[rabbah]

In part, having a consistent benchmarking methodology will be necessary to make meaningful comparisons.

[duglin]

I think we may find that there will be different usecases at play and therefore different container management strategies. For example, some may find that the asynchronous scheduling model of kubernetes works just fine for them, while others may need a direct management model where the invokers talk directly to the container management system. I think it would be nice if we allowed for a variety of options.

However, in both cases I think the control plane can be hosted in Kube and that should be our main focus right now. I think it would be good to start to explore the options around talking directly to the container manager but I suspect that solution will be non-trivial and will take some time to get right.

Therefore, my suggested course of actions would be: 1 - get the control plane ported to Kube, leveraging invokers that are outside of the kube env 2 - port the invoker to Kube so it can leverage Kube's scheduler - my assumption is this will be the easier path 3 - add support for having the invoker talk directly to the container manager - or whatever solution we come up with to meet our performance requirements.

And, as @markusthoemmes said, using numbers to validate our decisions/plans.

[animeshsingh]

For number 1, from an outside user`s perspective who wants to run OpenWhisk on K8, there is nothing for me. If I have to identify extra machines outside of my K8 cluster to deploy OpenWhisk, I might just do the clean distributed deployment which is already there.

I as a user, would use the solution if I can run this on my K8 cluster completely.

So from an end user`s perspective, two outputs make sense

1. Phase One: Everything on K8, and OpenWhisk treats some K8 slave machines as Invokers (Not ideal, but gets everything running on K8).

2. Phase Two: Everything on K8, and there is NO Invoker 'machine' - K8 cluster is the 'giant invoker' where everything is dispatched.

Of course phase 2 is going to be non trivial, performance will be issue - and those are something which need to be ironed out as we move along.

Also we have a community call on every 2nd Friday with Dan, Matt and others - lets take some of this discussion there. Its Google hangouts, so please letme know if you would like to join and the email id you would like to use

[duglin]

just FYI, my #1 (while it could be a deliverable for some folks) was more of a stepping stone to the others. It just allows us to prove the control plane works ok on K8s (not there should be any reason why it doesn't).

[animeshsingh]

Makes sense, and that`s already covered (though with still some manual steps) in the PR which has come in. So lets move beyond that discussion. Right now there are two high level deliverables from a community perspective, and we are pretty close to phase one.

[ghost]

With the last [PR}(https://github.com/apache/incubator-openwhisk-devtools/pull/31) merged, all of the initial ground work to deploy OpenWhisk on Kube has been completed. There are still a number of enhancements that can be worked on and I will submit another PR soon to document them on the README for kube.