Open kuangdao opened 1 month ago
i set vllm version to 0.5.2 and flashinfer version to 0.8, same error, who can tell me why .
any aone care it ? @DarkLight1337
@Yard1 can you help with this?
I'm pretty sure this is not supported yet. We should be raising an error...
thanks, when this feature will be support? @Yard1
Met the same issue, is there any plan to support this feature?
same error, hope flashinfer to support fp8 kvcache + cuda graph
Your current environment
🐛 Describe the bug
export CUDA_VISIBLE_DEVICES=1 export VLLM_ATTENTION_BACKEND=FLASHINFER
python -m vllm.entrypoints.openai.api_server \ --port=8006 \ --host=0.0.0.0 \ --served-model-name=vllm_fp8_kvcache \ --model=deepseek-ai/DeepSeek-Coder-V2-Instruct \ --quantization fp8 \ --kv-cache-dtype=fp8 \ --tensor-parallel-size=1 \ --dtype=bfloat16 \ --max-model-len=8192
i run vllm use the script, i want to use fp8 attention which Qtransopse(Key) or softmax value in attention support fp8 compute by FLASHINFER, such as this issue https://github.com/vllm-project/vllm/issues/6246 . the error .