neuralmagic / nm-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://nm-vllm.readthedocs.io
Other
251 stars 10 forks source link

[ CI ] Fix Failing Magic Wand Test #311

Closed robertgshaw2-neuralmagic closed 4 months ago

robertgshaw2-neuralmagic commented 4 months ago

SUMMARY:

robertgshaw2-neuralmagic commented 4 months ago

thanks

The del is a custom __del__ for VLLMModelRunner ... I am hesitant to touch that code to avoid upstream sync issues.

But can try it there...