search

commercetest / pepper-box

Pepper-Box is kafka load generator plugin for jmeter. It produces kafka messages of type plain text (JSON, XML, CSV or any other custom format) as well as java serialized objects.
http://pepperbox.gslab.com/
Apache License 2.0
2 stars 3 forks source link

Add code to determine if/when a kafka topic is created #17

Closed julianharty closed 6 years ago

julianharty commented 6 years ago

When testing large-scale & distributed Kafka clusters, there may be delays when topics are auto-created. This has been particularly prevalent when using replication between clusters. The existing Consumer code simply tries to read from a particular topic. This may inadvertently cloud the test results.

Rather than have our code blindly expect topics to be created, let's add code that checks whether the topic exists & - where appropriate and useful - wait until it does exist before producing or consuming data to the topic.

julianharty commented 6 years ago

A highly relevant article is https://blog.codecentric.de/en/2017/12/topic-configuration-kafka/

julianharty commented 6 years ago

This feature has been added to the Groovy Consumer script and the code merged to master. The basic functionality seems to work with a back-off algorithm that reduces the chance of overloading any Zookeeper node receiving the queries.

Adding similar logic to the producer may complicate the runTest(...) method so I'm not going to add a similar feature there at the moment. At some point we may also want to revise and improve the consumer code, however we'll deal with that if and when. I'll close this issue now.