search

confluentinc / kafka-streams-examples

Demo applications and code examples for Apache Kafka's Streams API.
Apache License 2.0
71 stars 1.15k forks source link

Extract KafkaEmbeded to a dedicated test lib #317

Closed danielpetisme closed 3 years ago

danielpetisme commented 4 years ago

Hello folks, I'm working for Michelin and we're using the Enterprise and OSS confluent platform.

We have one question, do you ever considered to extract the classes from https://github.com/confluentinc/kafka-streams-examples/tree/5.4.0-post/src/test/java/io/confluent/examples/streams/kafka to a dedicated kafka-integration-test lib.

Just to give you feedback, we copy/pasted the 2 classes coming from to our internal projects but we're not happy with that. I think proposing an "official" embedded Kafka would definitively help more developer to properly do some Kafka integration testing thus worth the extraction.

To give you context, these are the alternatives we identified: TopologyTestDriver we use it for Kafka Stream (and it's awesome) but:

  1. we faced edge-cases issues
  2. sometimes we need to use the basic producer/consumer APIs and we do not have nothing to properly test.

We had a look to TestContainers, the solution is cool too but way too slow to start (and trust me the longer the tests take to run, the less you run them). Plus, it requires Docker to be installed/configured and in a corporate context full of proxies it can be a pain.

Finally, we have been crawling github for options like https://github.com/jetoile/hadoop-unit/tree/master/hadoop-unit-kafka but most of them are a 1-person/hero effort with an uncertain reliability in the future.

What do you think?

mjsax commented 3 years ago

Personally, I think it would be a good idea. However, it should be done in the Apache Kafka project. There is already a ticket for this: https://issues.apache.org/jira/browse/KAFKA-5041 -- Feel free to follow up there.