Open alugowski opened 2 months ago
This error is in the profile_run
used to determine memory usage. If I patch the code to ignore the error, and a few lines below patch the model length to make raise_if_cache_size_invalid
happy, the model starts. Sure it won't reach the full 1M context, but it will work with 200k on an 80GB GPU.
This will be more pressing for users of small GPUs as the popular models increase their context lengths beyond 8k.
I'm happy to submit my monkey patches if there isn't already a plan to support large context models.
You can manually set --max-model-len
to reduce the context length.
Not sure whether it's a good idea to automatically limit the context length based on available memory. @simon-mo any thoughts?
You can manually set
--max-model-len
to reduce the context length.Not sure whether it's a good idea to automatically limit the context length based on available memory. @simon-mo any thoughts?
Agreed that a purely automatic setting may give folks the wrong impression that they can use the full context of the model even if their hardware won't allow it. One alternative is --max-model-len max
that would start the model no matter what and report the actual max context in the logs.
Right now someone must start vllm, see the crash, parse out the max context size from the log, and set that with --max-model-len
. But that's only if the profile_run()
doesn't OOM with the exception in the OP, in that case the user must guess at the max model len (the log message with the actual max is printed later, and depends on profile_run()
succeeding).
Your current environment
🐛 Describe the bug
Loading a model with a huge context length (100k+) causes an out of memory error when allocating space for the KV cache. While it is correct that technically the model does not fit, such models should still start with reduced context size.
An easily reproducible example is
gradientai/Llama-3-8B-Instruct-Gradient-1048k
which is 8B with a 1M context. It will fail to start on any single GPU:Results in: