This PR adds a helper class to manage engine operator requests that may come in for various engines held by the same continuous batching scheduler
the main idea is that as EngineOperator requests come in, they will be queued in separate Queues key'd by their EngineOperator. The scheduler will then use ContinuousBatchingQueues.pop_batch() to select the next most important batch for running.
Right now, the heuristic for the next batch is 1) the queue with longest wait time if it is over 100ms 2) the queue that can fill the largest batch size. For future work - the scheduler can implement a select_fn with more involved heuristics that take into account other worker threads if needed.
test_plan:
Unit tests included for basic functionality of ContinuousBatchingQueue and ContinuousBatchingQueues
This PR adds a helper class to manage engine operator requests that may come in for various engines held by the same continuous batching scheduler
the main idea is that as
EngineOperatorrequests come in, they will be queued in separate Queues key'd by theirEngineOperator. The scheduler will then useContinuousBatchingQueues.pop_batch()to select the next most important batch for running.Right now, the heuristic for the next batch is 1) the queue with longest wait time if it is over 100ms 2) the queue that can fill the largest batch size. For future work - the scheduler can implement a
select_fnwith more involved heuristics that take into account other worker threads if needed.test_plan: Unit tests included for basic functionality of
ContinuousBatchingQueueandContinuousBatchingQueues