run-llama / llama_deploy

Deploy your agentic worfklows to production
https://docs.llamaindex.ai/en/stable/module_guides/llama_deploy/
MIT License
1.86k stars 193 forks source link

feat: make the kafka topic configurable #353

Closed masci closed 2 weeks ago

masci commented 3 weeks ago

Part of #349 (not a fix until we propagate the topic settings up to the control plane and api server)

Problem

Currently we use the name of the "message type" string to create the Kafka topic that'll be used in the message queue. Problem is that different instances of control planes using the same Kafka instances will step into each other, with catastrific effects. Note that this problem is common to other message queue brokers like S3, so eventually this fix will be rolled out for all the supported queues.

Solution

Make the Kafka topic name configurable, so that different control planes will use different topics. This PR only makes the topic configurable, another PR will make use of this new feature and resolve the original issue.

Notes

While working at the fix, I noticed a few things we can improve and already implemented in this PR in a backward compatible manner:

coveralls commented 3 weeks ago

Coverage Status

coverage: 71.969% (+0.9%) from 71.038% when pulling ac5a92a81547131449964e13bd656b027ed9e202 on massi/349 into 34b9299748dd902f032e91827e5c778690c721b9 on main.