Kafka consumer throughput was poor because, despite handling message
reading and ack sending on different goroutines, these were effectively
doing one message at a time because the message sending goroutine could
not send to the toConfirmIds channel until the ack was being sent.
This splits things so that we don't block in this way any more and
instead keep a slice of messages needing to be acked.
Kafka consumer throughput was poor because, despite handling message reading and ack sending on different goroutines, these were effectively doing one message at a time because the message sending goroutine could not send to the toConfirmIds channel until the ack was being sent.
This splits things so that we don't block in this way any more and instead keep a slice of messages needing to be acked.