Queue implementation

[lazamar]

This combines the implementations of Partition and Topic. A Queue takes care of managing the Partition resources of a Topic, and the creation and teardown of the Topic itself.

A Queue periodically stores its state. A file structure of a queue with one partition and two topics looks like this:

queue
├── inventory.json
└── test-topic
    ├── 0.partition.index
    ├── 0.partition.records
    ├── 1.partition.index
    └── 1.partition.records

The Inventory contains information about all topics, partitions and consumer offsets. Here is an example inventory content:

{
    "test-topic": {
        "s_consumers": {
            "test-group": {
                "0": 10,
                "1": 10
            }
        },
        "s_partitions": [
            0,
            1
        ]
    }
}

This inventory says that there is one topic test-topic which has a single consumer group test-group, which has the latest committed offset as 10 for both of the topic's partitions. The topic has 2 partitions numbered 0 and 1.

Tests

unit
  queue
    no concurrent opens [✔]
    restores state as it was [✔]
    resumes consumption from last commit [✔]