search

neuralmagic / deepsparse

Sparsity-aware deep learning inference runtime for CPUs
https://neuralmagic.com/deepsparse/
Other
2.98k stars 172 forks source link

[continuous batching] singleton pattern for scheduler #1391

Closed bfineran closed 10 months ago

bfineran commented 10 months ago

singleton pattern for continuous batching scheduler is favorable so that engine requests coming from different pipelines are scheduled together and do not interfere with each other.

this diff provides a simple implementation and extends the docstring with a small example