search

jina-ai / jina

☁️ Build multimodal AI applications with cloud-native stack
https://docs.jina.ai
Apache License 2.0
20.87k stars 2.22k forks source link

Message Queue for Flow pipeline #5982

Closed sansmoraxz closed 9 months ago

sansmoraxz commented 1 year ago

Describe your proposal/problem

This is mostly in terms of Kubernetes but may work with Docker too.

Imagine an Flow made up of a bunch of Executor (them having stateless replicas). Let's say we want to implement autoscaling to save costs or scale out on increased demand. The currently supported messaging protocols are either not compatible with autoscaling (GRPC) or very difficult and expensive to implement (WS or HTTP).

Message queues give us the freedom to monitor the state of the flow far more easily, and additionally gives more flexibility in terms of scaling (automatic 0 scaling when not in use and release for example the A100 node when not in use).

hanxiao commented 1 year ago

🤔 this autoscaling is directly doable with jina, e.g. https://docs.jina.ai/concepts/jcloud/configuration/#autoscaling-with-jinaai-serverless

sansmoraxz commented 1 year ago

🤔 this autoscaling is directly doable with jina, e.g. https://docs.jina.ai/concepts/jcloud/configuration/#autoscaling-with-jinaai-serverless

This seems to be specific to JCloud unless I am reading it wrong.

Also the format requires the image to be hosted in Jina which might introduce a whole lot of NAT related costs.

JoanFM commented 1 year ago

Hey @sansmoraxz,

Thanks for the proposal, we will consider this but be aware that we may not have a lot of resources to implement this at the moment.

JoanFM commented 1 year ago

Also, could you share some info as per why it is easier to implement autoscaling with message queues than for this direct messaging protocols?

sansmoraxz commented 1 year ago

Also, could you share some info as per why it is easier to implement autoscaling with message queues than for this direct messaging protocols?

It's easier to handle backpressure through queues, than direct messaging, and restore on pod crash.

Although I haven't tried KNative looks interesting, might be easier than Queues.

Perhaps similar to how there is Executors for jinaai+docker:// and docker:// there may be a serverless:// executor which can pull image from any registry. And to_kubernetes_yaml would also put out the KNative CRD objects. WDYT?

JoanFM commented 1 year ago

Hello @sansmoraxz ,

This sounds like a good idea, we are using this in our cloud service.

We are currently discussing the best way to enable the community to use our solution or part of it in their Kubernetes cluster.

Anyway, with the to_kubernetes_yaml you can add the necessary editions to adapt to KNative.

Thanks for the suggestion.

jina-bot commented 10 months ago

@jina-ai/product This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days