Closed sansmoraxz closed 9 months ago
🤔 this autoscaling is directly doable with jina, e.g. https://docs.jina.ai/concepts/jcloud/configuration/#autoscaling-with-jinaai-serverless
🤔 this autoscaling is directly doable with jina, e.g. https://docs.jina.ai/concepts/jcloud/configuration/#autoscaling-with-jinaai-serverless
This seems to be specific to JCloud unless I am reading it wrong.
Also the format requires the image to be hosted in Jina which might introduce a whole lot of NAT related costs.
Hey @sansmoraxz,
Thanks for the proposal, we will consider this but be aware that we may not have a lot of resources to implement this at the moment.
Also, could you share some info as per why it is easier to implement autoscaling with message queues than for this direct messaging protocols?
Also, could you share some info as per why it is easier to implement autoscaling with message queues than for this direct messaging protocols?
It's easier to handle backpressure through queues, than direct messaging, and restore on pod crash.
Although I haven't tried KNative looks interesting, might be easier than Queues.
Perhaps similar to how there is Executors for jinaai+docker://
and docker://
there may be a serverless://
executor which can pull image from any registry. And to_kubernetes_yaml
would also put out the KNative CRD objects. WDYT?
Hello @sansmoraxz ,
This sounds like a good idea, we are using this in our cloud service.
We are currently discussing the best way to enable the community to use our solution or part of it in their Kubernetes cluster.
Anyway, with the to_kubernetes_yaml
you can add the necessary editions to adapt to KNative.
Thanks for the suggestion.
@jina-ai/product This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days
Describe your proposal/problem
This is mostly in terms of Kubernetes but may work with Docker too.
Imagine an
Flow
made up of a bunch ofExecutor
(them having stateless replicas). Let's say we want to implement autoscaling to save costs or scale out on increased demand. The currently supported messaging protocols are either not compatible with autoscaling (GRPC) or very difficult and expensive to implement (WS or HTTP).Message queues give us the freedom to monitor the state of the flow far more easily, and additionally gives more flexibility in terms of scaling (automatic 0 scaling when not in use and release for example the A100 node when not in use).