Heka's KafkaInput plugin writes the offset for every Kafka message. When it's a low volume topic, this doesn't cause any issues, but with high volume ones, disk IO becomes a limiting factor to how quickly Heka can process messages.
Solution
Instead of writing the offset on every message to disk, this PR mmaps it. A quick benchmark shows the advantages of using this method:
Problem
Heka's KafkaInput plugin writes the offset for every Kafka message. When it's a low volume topic, this doesn't cause any issues, but with high volume ones, disk IO becomes a limiting factor to how quickly Heka can process messages.
Solution
Instead of writing the offset on every message to disk, this PR
mmaps it. A quick benchmark shows the advantages of using this method:No new tests were written, and the ones that verify checkpoint correctness still pass.
r. @fbogsany @rafrombrc