Pepper-Box is kafka load generator plugin for jmeter. It produces kafka messages of type plain text (JSON, XML, CSV or any other custom format) as well as java serialized objects.
Through a variety of testing I've discovered the consumer and how it interacts with Kafka can result in it missing significant amounts of data (60% +).
I suspect one cause is the use of enable.auto.commit = true in the code. Also, I noticed it is more likely to read later messages than earlier ones when the consumer runs on replicated nodes and/or after the producer has completed.
The change seems to have been effective. Perhaps there's a slight chance it'd read early data that's been read previously (conjecture as I've not seen this behaviour). Anyway, I'm happy to close this issue now.
Through a variety of testing I've discovered the consumer and how it interacts with Kafka can result in it missing significant amounts of data (60% +). I suspect one cause is the use of
enable.auto.commit = true
in the code. Also, I noticed it is more likely to read later messages than earlier ones when the consumer runs on replicated nodes and/or after the producer has completed.