search

commercetest / pepper-box

Pepper-Box is kafka load generator plugin for jmeter. It produces kafka messages of type plain text (JSON, XML, CSV or any other custom format) as well as java serialized objects.
http://pepperbox.gslab.com/
Apache License 2.0
2 stars 3 forks source link

The consumer doesn't always read all the messages #13

Closed julianharty closed 6 years ago

julianharty commented 6 years ago

Through a variety of testing I've discovered the consumer and how it interacts with Kafka can result in it missing significant amounts of data (60% +). I suspect one cause is the use of enable.auto.commit = true in the code. Also, I noticed it is more likely to read later messages than earlier ones when the consumer runs on replicated nodes and/or after the producer has completed.

julianharty commented 6 years ago

The change seems to have been effective. Perhaps there's a slight chance it'd read early data that's been read previously (conjecture as I've not seen this behaviour). Anyway, I'm happy to close this issue now.