AsyncEngine create cancel task in exception.

InternLM / lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

https://lmdeploy.readthedocs.io/en/latest/

Apache License 2.0

3.13k stars 280 forks source link

Closed grimoire closed 1 week ago

grimoire commented 1 week ago

related PR: #1789

optimize performance of pytorch engine threadsafe mode.