Rework pop policy

[Zannick]

Recent changes around history storage improvements have allowed growing from 200M to over 1B total states processed, but the later states able to be processed are still in the same range, which makes me realize a pretty big drawback to our Steiner-graph-based heuristic for multi-point A*:

Because the underestimation "gap" is per-requirement, the gap grows with more requirements. And because we always process the state with the lowest total of elapsed+estimated, we actually end up prioritizing inefficient states that have accomplished less--over the states that are a few steps ahead even if the total estimate is higher. The extra-case handling isn't ever shown in the status update, but also the later states are more likely to be evicted--I also haven't seen many states with the progress counter above 8, despite the initial routes seeding the queue with progress items all the way to 18.

75 can help address some of this thanks to the extra handling, but we really ought to change the whole queue system as well, via one of these ideas, for instance:

1. Change pop to return the min of each segment of the queue. Returning multiple states on each pop would let threads have more work to do between sync points, potentially increasing throughput. Having eviction happen separately per segment would also help avoid relegating good later states to the disk.
2. Change pop to round-robin the segments of the queue.
3. Change the main loop to occur in a rayon scope instead of in a parallel iterator. This would let us have entirely thread-specific behavior, instead of each thread having to do a piece of normal work (i.e. pop the absolute min) before its own behavior (e.g. pop the min of a specific segment). The downside would be having to manage the thread exit behavior manually (but we already have a "done" signal for the background thread).

[Zannick]

Wow, the heap stats graphs look so much more well-distributed now! Remains to be seen if this will get to the better solution...