search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.25k stars 3.58k forks source link

Pulsar function replicas auto scaling #18584

Open zzzming opened 1 year ago

zzzming commented 1 year ago

Search before asking

Motivation

Pulsar function and connector in Kubernetes runtime should be able to auto scale based on resource usage or the input topic backlog size or some other metrics. The auto scale should be horizontal scaling.

Solution

No response

Alternatives

Use the existing Kubernetes HPA. But the state of replicas won't be saved because function stop removes the function statefulset. The parallelism is save in bookkeeper not Kubernetes (upon statefulset removal) in the current implementation.

Anything else?

No response

Are you willing to submit a PR?

hpvd commented 1 year ago

maybe this pulsar function in mesh scaling is interesting https://github.com/streamnative/function-mesh/issues/457

michaeljmarshall commented 1 year ago

Thanks for the suggestion @hpvd. I think this issue is requesting an improvement to the kubernetes runtime that ships with pulsar.

hpvd commented 1 year ago

jep. You are right and sorry the issue was not even the right target to link (it's a preview how function scaling in function mesh should/will work in flexible way in the near future -> using keda)

-> Wanted to point to a solution how functions could be auto-scaled today (with additional tool: function-mesh)

With Kubernetes Horizontal Pod Autoscaler (HPA), Function Mesh supports automatically scaling the number of Pods (Pulsar instances) that are required to run Pulsar functions, sources, and sinks.

Function Mesh auto-scales the number of Pods based on the CPU usage, memory usage, and custom metrics.

source: https://functionmesh.io/docs/scaling

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.