Closed muc-martin closed 5 months ago
+1
From my understanding, your prompt token size cannot exceed : GPU block BlockSize. For example, running TheBloke/Phind-CodeLlama-34B-v2-GPTQ on a RTX4090 : Max_prompt_length = block_size(16) gpu_block(123)=1968 token. gpu_block can be increase with gpu-memory-utilization
getting same error with a yi awq
调大了gpu-memory-utilization的值,解决了这个问题
I loaded Mixtral TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ in VLLM successfully. Everything works for short prompts. However, longer prompts lead to the error:
This is the command I use to start the model:
The model starts with:
Running on a RTX3090