search

commercetest / pepper-box

Pepper-Box is kafka load generator plugin for jmeter. It produces kafka messages of type plain text (JSON, XML, CSV or any other custom format) as well as java serialized objects.
http://pepperbox.gslab.com/
Apache License 2.0
2 stars 3 forks source link

Add additional kafka properties to control the behaviour of writes #6

Closed julianharty closed 6 years ago

julianharty commented 6 years ago

For our testing we want to minimise the risk of duplicates and out-of-order messages, etc. The defaults used by the Kafka client libraries don't suit our needs. This enhancement is intended to add the various parameters that'd help our testing.

julianharty commented 6 years ago

max.in.flight.requests.per.connection=1 to prevent out-of-order messages (default is 5) acks=all (default is 1) retries >= 1 (default is 0) replication.factor=3 (assuming you have a cluster of at least 3 Kafka nodes, else set it to the number of nodes.)

See various documents including the free to download whitepaper from Confluent https://www.confluent.io/white-paper/optimizing-your-apache-kafka-deployment/

julianharty commented 6 years ago

Note: replication.factor is not currently practical to set within the current functionality of pepper-box. In our testing we'll configure the server to create sufficient replicas instead by using default.replication.factor see http://kafka.apache.org/documentation.html#prodconfig for more info.

julianharty commented 6 years ago

Now I've changed the default settings to reduce the risk of missing or out-of-order messages it's time to close this issue since I'm not aware of any missing settings at this stage.