Part of #349 (not a fix until we propagate the topic settings up to the control plane and api server)
Problem
Currently we use the name of the "message type" string to create the Kafka topic that'll be used in the message queue. Problem is that different instances of control planes using the same Kafka instances will step into each other, with catastrific effects. Note that this problem is common to other message queue brokers like S3, so eventually this fix will be rolled out for all the supported queues.
Solution
Make the Kafka topic name configurable, so that different control planes will use different topics. This PR only makes the topic configurable, another PR will make use of this new feature and resolve the original issue.
Notes
While working at the fix, I noticed a few things we can improve and already implemented in this PR in a backward compatible manner:
I am not sure the base class for queues should be a pydantic model, that's very limiting when you need to treat the class as a regular class (get and set local instance variables for example). I think a plain abstract class derived from ABC will make the job.
There's a lot of duplication in the configuration settings: the class constructor, the config settings model, the static methods... I think we should have a single source of truth, being the config settings model. For Kafka I made the constructor take the config model instance directly.
I moved the original code behind the if __name__ == "__main__" that was used for testing into a proper e2e test. Tests for message queues will most certainly need docker to be avaiable, I'll add the required CI bits (this part is WIP, you'll see it's done when tests are all green).
coverage: 71.969% (+0.9%) from 71.038%
when pulling ac5a92a81547131449964e13bd656b027ed9e202 on massi/349
into 34b9299748dd902f032e91827e5c778690c721b9 on main.
Part of #349 (not a fix until we propagate the topic settings up to the control plane and api server)
Problem
Currently we use the name of the "message type" string to create the Kafka topic that'll be used in the message queue. Problem is that different instances of control planes using the same Kafka instances will step into each other, with catastrific effects. Note that this problem is common to other message queue brokers like S3, so eventually this fix will be rolled out for all the supported queues.
Solution
Make the Kafka topic name configurable, so that different control planes will use different topics. This PR only makes the topic configurable, another PR will make use of this new feature and resolve the original issue.
Notes
While working at the fix, I noticed a few things we can improve and already implemented in this PR in a backward compatible manner:
ABC
will make the job.if __name__ == "__main__"
that was used for testing into a proper e2e test. Tests for message queues will most certainly need docker to be avaiable, I'll add the required CI bits (this part is WIP, you'll see it's done when tests are all green).