This PR attempts to reduce the consequences (ie scheduling lag) of attempting to schedule a task whose request lock is locked elsewhere for a long period of time (ie history persisters).
Changes:
Actually make offer scoring parallel while waiting on request locks. The offer scoring thread pool uses an unbounded queue, so it would never grow past the core pool size of 1 / was still single threaded.
Limit the amount of time that the offer loop will wait for a request lock - less relevant now that the thread pool can have more than 1 thread, but would still help if the thread pool was saturated with locked requests.
Took a very quick/lazy approach to the tryRunWithLock in the offer loop, so please send feedback - should it throw an error, log whether it was able to run, etc.
Considered attempting a larger scale refactor (poll from a single priority queue of pending tasks, remove request locks, etc), but that seemed very likely to introduce new and exciting bugs. The fact that there's separate offer-scheduler and SingularitySchedulerPoller threads that schedule tasks slightly differently (drainPendingQueue) is annoying, but doesn't seem to be the main source of problems.
ssalinas
commented
2 years ago
This PR attempts to reduce the consequences (ie scheduling lag) of attempting to schedule a task whose request lock is locked elsewhere for a long period of time (ie history persisters).
Changes:
Took a very quick/lazy approach to the tryRunWithLock in the offer loop, so please send feedback - should it throw an error, log whether it was able to run, etc.
Considered attempting a larger scale refactor (poll from a single priority queue of pending tasks, remove request locks, etc), but that seemed very likely to introduce new and exciting bugs. The fact that there's separate
offer-schedulerandSingularitySchedulerPollerthreads that schedule tasks slightly differently (drainPendingQueue) is annoying, but doesn't seem to be the main source of problems.cc - @ssalinas, @pschoenfelder