Open yaronr opened 2 months ago
Note this is probably duplicate of https://github.com/vllm-project/vllm/issues/5344
temporarily you might avoid the error by removing --enable-prefix-caching
(it can still get OOM with long enough context, but there seem to be ongoing effort to improve it e.g: https://github.com/vllm-project/vllm/issues/5907)
Your current environment
vllm 5.0.0-post-1
🐛 Describe the bug
Running the following query:
Resulted in vllm crashing, having the following in the logs: