refact: Make topic explicit in message queue API

run-llama / llama_deploy

Deploy your agentic worfklows to production

https://docs.llamaindex.ai/en/stable/module_guides/llama_deploy/

MIT License

1.86k stars 193 forks source link

refact: Make topic explicit in message queue API #358

Closed masci closed 2 weeks ago

masci commented 2 weeks ago

Apologies for the mega-refactoring, breaking this down into smaller PR would take a week 😛

Fixes #349 for good.

Problem

Message publishers and subcribers assumed the "topic" to listen to was the same as the message type. This works in a single-tenant environment but becomes a problem when the same message queue holds data for multiple control planes and services, causing clashes.

Solution

Make the concept of "topic" explicit, so that when publishing/reading messages a topic must be provided
Make the topics "nested" under the control plane, like control_plane_one.my_service so that services with the same name attached to different control planes can coexist
Make the topic "namespace" configurable, so that users can specify where to nest topics used in Llama Deploy and avoid conflicts with their existing, unrelated topics

Follow up

Fix topic management for the other message queues we support, for now only Kafka is taking advantage of this feature

coveralls commented 2 weeks ago

coverage: 72.127% (+0.03%) from 72.1% when pulling 1997f1e8fd20c900ad969e5315ee401e96678202 on massi/topic-refact into 1fcfa2f67cda1c0158c0b6aa7b2d0ab1f78c93e1 on main.

masci commented 2 weeks ago

Hmm, this makes sense. Trying to understand the hierarchy of control now
* task definitions have an `agent_id` specifying what service to send to (this probably needs to be renamed in the future)
Correct, this didn't change.
* messages have a `type` used for judging how a service handles a message
Same as before
* the `topic` for a consumer is defined by the control plane config topic + msg type?
Correct, this is brand new
  * does this mean each service + message type combination has its own topic/queue now?
Exactly, which makes it possible to share the same Kafka cluster (but really any message queue broker) across multiple control planes

Slightly fuzzy if there's anything beyond this 😅 I feel like I need to draw this out now

That's really it :)

logan-markewich commented 2 weeks ago

@masci thanks for confirming!

masci commented 2 weeks ago

Follow up issues:

https://github.com/run-llama/llama_deploy/issues/359