Durable queues

[AmineDiro]

In This PR:

Durable queues:

• This is a disk-backed queue for persisting the messages.
• The tasks are served in a FIFO manner.
• Durable queues append serialized messages to a fixed-sized file called LogSegment.
• The durable queues also append queue operations to an IndexSegment.
• The index segment serves as both a :
  • Bitcask index for storing the messages offsets
  • A WAL file: it is a Log where we append message status after each queue operation (ready, delivered, acked, and failed) and an offset to the message in one of the LogSegments. At init time, we just scan this file to rebuild the in-memory data structure.

Consumers

• Implement an acking mechanism for consumers.
• Changes the consumer join() to wait for the queue to be empty before canceling the consumers.

Performance :

• benchmarked the throughput

  2023-01-09 22:58:48,186,186 INFO: Created 2 queues in Cluster and one QueueManager.
  2023-01-09 22:58:49,043,43 INFO: Starting 2 consumers
  2023-01-09 22:58:49,043,43 INFO: Waiting for the 2 consumers to process all items in queue_pool...
  2023-01-09 22:58:49,612,612 INFO: All consumers are done ! 10000 items processed.
  Mean write ops [1tests] 79182.39 wop/s
  Mean read ops [1tests] 17411.01 rop/s