phaistos-networks / TANK

A very high performance distributed log service
Apache License 2.0
938 stars 70 forks source link

TANK client: batch produced messages for efficiency #69

Open markpapadakis opened 4 years ago

markpapadakis commented 4 years ago

When you use the produce APIs to publish events/messages, a new batch is created immediately and transmitted to the respective TANK endpoint(s).

The Kafka Producer supports batching (controlled via "batch.size" and "linger.ms"), where it will buffer produced events (semantics depend on those tunables) as opposed to immediately transmit the request. This creates various optimization opportunities specific to compression of batches and also reduces number of packets required for relaying the request to the endpoint(s).

It should be trivial to support those semantics in the new TANK2 client.