LockServiceImpl reaper cycles through open locks

[jeremyk-91]

Currently the lock reaper maintains a BlockingQueue of tokens to check for reaping. The lock reaper sleeps for the minimum of the time before the lock actually expires, and the allowed clock drift (after #2041). Quoting myself:

You can imagine the worst case to be if we have say N queued requests that are all long-blocking and then something that expires quickly; we would still block for a minimum of 5N seconds.

(N.B. 5 is the default "allowed clock drift")

One way of getting around this would be to use a DelayQueue rather than walking through the queue and replacing tokens if they haven't actually expired yet.

[gmaretic]

That sounds like exactly what we want

[nziebart]

The BlockingQueue is actually secretly a PriorityQueue (should probably be typed as such for clarity).