If we do not have the topic locally, we should look up from gossip whether we know about other partitions that have tasks in a topic, and forward the dequeue there.
Attempt to dequeue locally, if that returns nothing look up in the index if the topic exists on another partition, return a random partition.
We also need to enforce a max redirects to ensure that we never keep bouncing around.
If we do not have the topic locally, we should look up from gossip whether we know about other partitions that have tasks in a topic, and forward the dequeue there.
Attempt to dequeue locally, if that returns nothing look up in the index if the topic exists on another partition, return a random partition.
We also need to enforce a max redirects to ensure that we never keep bouncing around.