Kafka Trigger only firing orignal function pod

[dap332]

Brief Summary: I have deployed a kubeless function as well as a Kafka deployment via instructions on Kubeless docs. I created a Kafka trigger to to fire my kubeless. function. Works as normal. I decided to scale my kubless depolyment to 3 pods instead of just one. Kafka events only trigger the original kubeless function pod rather than 3.

What you expected to happen: I would expect the trigger to fire all 3 pods of the kubeless function, or some command to scale the pods manually and attach the trigger to them

How to reproduce it : In a k8s cluster with a pvc and kubeless you need to:

• Deploy your kubeless function with the kubeless function deploy command.
• Deploy Kafka and a trigger to the function as shown in: https://kubeless.io/docs/pubsub-functions/
• Scale up the pods manually with a command such as: kubectl scale deployment <deployment name> --replicas=3

• Publish to your Kafka broker and watch as only one of the pods that hold your kubeless functions get run on publish.

Environment:

• Kubernetes version:

  $ kubectl version 
  Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubeless version:

  $ kubeless version
  Kubeless version: v0.6.0

[murali-reddy]

I am not able reproduce this issue on Minikube or on AWS cluster. As I understand @sebgoa is able to reproduce as well. From kubeless side it just creating a Kubernetes service for each function deployment. So it got to do with how kube-proxy is load balancing the requests. Also use of Kafka as event source is not necessarily causing this issue. You might as well observe similar behaviour if you do kubeless function call.

If you can please try a sample Kubernetes service outside of Kubeless and see how service is getting load balanced.

[dap332]

I have made a repository that automates the deployment (and cleanup) of my cluster, its dependencies (kubeless, kafka, resources, cassandra). And a small test client script. This may help reproduce the error.

Note this is still very raw, and the shell scripts may not work perfectly on other machines has I have not tested on other devices or k8s clusters.

Repo can be found here. https://github.com/dap332/k8s-kafka-cql-kubeless_Stack