PeterJCLaw
commented
3 months ago Currently, running Kafka for integration or end-to-end testing can be challenging, especially in a microservice environment where setting up and managing Kafka instances can be complex and resource-intensive.
Could you expand on what this is based on?
My experience is that setting up a local kafka container (using the basic apache/kafka image) for development is pretty easy. While I haven't tried to set it up for use in testing or CI, I am expecting that that would be no more complex than using any other container. As far as I know it operates in-memory by default (certainly I haven't ever configured any permanent storage when using it) so it appears that it would likely already provide a lot of what you're considering here.
While the apache/kafka image doesn't directly offer any support around testing I'm assuming that what you're proposing here would be utilities outside the container anyway, leaving the proposed container essentially on par with what already exists.
If you wanted to provide utilities which extend the default image to aid in testing scenarios that might be valuable, though I would recommend exploring what is already provided within the image first -- there are a number of scripts within /opt/kafka/bin/ in the docker image which aid in interacting with a running instance and could easily be used to configure an instance for testing against.
Summary:
I’d like to propose a feature enhancement for the mockafka library. The goal is to expand its functionality to act as a lightweight, in-memory Kafka container that can be used for testing purposes across different programming languages. This feature would enable developers to write and execute tests without needing a separate Kafka instance.
Background:
Currently, running Kafka for integration or end-to-end testing can be challenging, especially in a microservice environment where setting up and managing Kafka instances can be complex and resource-intensive. The mockafka library, while useful for faking Kafka interactions, primarily targets specific Kafka clients like aiokafka or confluent-kafka-python.
Proposed Enhancement:
In-Memory Kafka Container: Implement an in-memory Kafka instance that runs in a lightweight container. This container should:
Operate using Kafka's socket protocol, making it compatible with any Kafka client. Be capable of handling basic Kafka operations (e.g., producing and consuming messages) without requiring a full Kafka setup. Cross-Language Compatibility: Ensure that the container can be accessed from various programming languages, not just Python. This would involve:
Using a standard protocol that Kafka clients across different languages can interact with. Documenting the container’s usage and integration steps for various languages. Testing Integration: Provide mechanisms for:
Setting up and tearing down the container programmatically. Configuring the container to simulate different Kafka scenarios. Motivation:
The ability to use a lightweight, in-memory Kafka instance would greatly simplify testing in development environments, particularly for integration or end-to-end tests. It would eliminate the need for separate Kafka instances and streamline the testing setup process.
Challenges:
I am not familiar with Kafka’s socket programming or the protocol it uses, so I recognize that this may be a complex project. The implementation would need to handle Kafka’s protocol requirements and ensure compatibility across different clients.
Request for Assistance:
Given the complexity of this feature, any guidance or collaboration on the implementation would be greatly appreciated. If anyone has experience with Kafka's internal protocol or socket programming, your input would be invaluable.