Closed julianharty closed 6 years ago
A highly relevant article is https://blog.codecentric.de/en/2017/12/topic-configuration-kafka/
This feature has been added to the Groovy Consumer script and the code merged to master
. The basic functionality seems to work with a back-off algorithm that reduces the chance of overloading any Zookeeper node receiving the queries.
Adding similar logic to the producer may complicate the runTest(...)
method so I'm not going to add a similar feature there at the moment. At some point we may also want to revise and improve the consumer code, however we'll deal with that if and when. I'll close this issue now.
When testing large-scale & distributed Kafka clusters, there may be delays when topics are auto-created. This has been particularly prevalent when using replication between clusters. The existing Consumer code simply tries to read from a particular topic. This may inadvertently cloud the test results.
Rather than have our code blindly expect topics to be created, let's add code that checks whether the topic exists & - where appropriate and useful - wait until it does exist before producing or consuming data to the topic.