search

vmware-archive / kafka-trigger

Kubernetes CRD controller for Kafka topic as event source for Kubeless functions
Apache License 2.0
28 stars 34 forks source link

Cache the value of the function port from k8s api #11

Open dannyqiu opened 5 years ago

dannyqiu commented 5 years ago

Issue Ref: https://github.com/kubeless/kubeless/issues/826

Description:

Address the issue of constantly hitting the k8s api to get the port associated with a function. We only need to retrieve this value once upon creating the consumer process.

TODOs:

dannyqiu commented 5 years ago

I agree that a simpler solution can be used. I like the idea of enforcing the port to be 8080, since this is already the default behavior if you don't use a custom deployment.

wayne-thomson-cxa commented 5 years ago

Hi there. I also seem to be experiencing the 5qps limitation reportd in kubeless/kubeless#826). Was this fix abandoned or replaced by something else?

andresmgot commented 5 years ago

we never changed this but I don't think that a limitation of 5 qps is a limitation because of calls to the kubernetes API. See the discussion here: https://github.com/kubeless/kubeless/issues/826 (but unfortunately no progress there)

wayne-thomson-cxa commented 5 years ago

we never changed this but I don't think that a limitation of 5 qps is a limitation because of calls to the kubernetes API. See the discussion here: kubeless/kubeless#826 (but unfortunately no progress there)

Thanks @andresmgot. I have read that. Do you have better throughput in clusters you are working with than 5 messages per second or do you experience the same behavior also? I thought from these two threads that wdamron and @dannyqiu identified the k8s API as the issue (hence this issue)?

Whether my topic has 1 or 10 partitions which my producer writes to the consumers seem be be 5/s per controller. As I increase the number of controllers I see a corresponding linear throughput. I just want to check that I am not doing something wrong.

andresmgot commented 5 years ago

You are probably don't. I think this issue was never fixed but I doubt the bottle neck is the K8s API for resolving the port. If you have time to investigate it, you can modify the function getFunctionPort to simply return 8080:

https://github.com/kubeless/kafka-trigger/blob/68a3798a36e725fb15c2dac4d4dc56274d075c3b/pkg/utils/event_sender.go#L40

And check if that helps.