The semantics and guarantees here are analogous to Kafka's.
One Topic is composed of a number of Partitions.
One Topic can have any number of Producers.
One Producer can produce to any Partition, as determined by the Message's partitioning key.
One Topic can have any number of Consumer Groups.
One Consumer Group can have any number of Consumers.
One Consumer reads messages from one or more Partitions in order.
A Consumer will return the first available Message from its Partitions in round-robin manner.
A Message is first read and later its offset can be committed.
Adding and removing Consumers causes a rebalance. Partitions that change consumers during a rebalance are rewinded to the latest committed Offset.
Benchmarks
A Topic can use any kind of partition. The benchmarks were run with the file-backed partition implementation.
Run with ./utils.sh bench-save
benchmarking queue/topic/write 10000 messages to 1 partition in series
time 29.56 ms (29.19 ms .. 29.83 ms)
0.999 R² (0.999 R² .. 1.000 R²)
mean 29.52 ms (29.30 ms .. 29.80 ms)
std dev 487.5 μs (395.5 μs .. 647.5 μs)
benchmarking queue/topic/write 10000 messages to 5 partitions in series
time 35.45 ms (35.05 ms .. 35.84 ms)
0.999 R² (0.999 R² .. 1.000 R²)
mean 34.84 ms (34.41 ms .. 35.16 ms)
std dev 733.5 μs (560.5 μs .. 1.017 ms)
benchmarking queue/topic/write 10000 messages to 10 partitions in parallel
time 72.30 ms (72.17 ms .. 72.45 ms)
1.000 R² (1.000 R² .. 1.000 R²)
mean 72.46 ms (72.35 ms .. 72.69 ms)
std dev 297.7 μs (191.8 μs .. 435.3 μs)
Tests
unit
queue
partition
reads what is written [✔]
reopens [✔]
no concurrent opens [✔]
blocks reads when reaches the end [✔]
reads concurrently [✔]
topic
1 consumer reads from 1 partition [✔]
1 consumer reads from 2 partitions [✔]
2 producers write to 1 partition [✔]
allows consumers without readers [✔]
reads ordered per partition [✔]
rebalance splits partitions between consumers [✔]
rebalance joins partitions in consumers [✔]
rebalance rewinds to last comitted offset [✔]
The semantics and guarantees here are analogous to Kafka's.
Benchmarks
A Topic can use any kind of partition. The benchmarks were run with the file-backed partition implementation. Run with
./utils.sh bench-saveTests