neuralmagic / deepsparse

Sparsity-aware deep learning inference runtime for CPUs
https://neuralmagic.com/deepsparse/
Other
3.03k stars 173 forks source link

[Continuous Batching][Text Generation] Add back continuous batching tests #1585

Closed dsikka closed 9 months ago

dsikka commented 9 months ago

Summary