[Usage]: Out of Memory w/ multiple models

[yudataguy]

Your current environment

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB. GPU 

How would you like to use vllm

I'm running a eval framework that's evaluating multiple models. vllm doesn't seem to free the gpu memory after initialize the 2nd model (with the same variable name), how to free up gpu memory with each vLLMEngine call llm = LLM(new_model)

[yudataguy]

Tried methods from https://github.com/vllm-project/vllm/issues/1908 no success

[russellb]

the LLM engine internal to the LLM class should get destroyed when your LLM instance is garbage collected. You could try forcing that with del(llm).

[russellb]

more detailed input on this in #3281

[russellb]

going to close this since it's a duplicate of #3281