Open pseudotensor opened 1 month ago
Same issue with head of main: ddc369fba147046f5044aaddbb867b5333f7068c
after installing flashinfer
is it that rope scaling is not supported? https://github.com/vllm-project/vllm/issues/6175
I can't be sure what the issue is.
Yes, also have this problem.
The warning on the third line in your last output block tells what is the problem here:
WARNING 07-08 19:39:49 utils.py:562] Gemma 2 uses sliding window attention for every odd layer, which is currently not supported by vLLM. Disabling sliding window and capping the max length to the sliding window size (4096).
You should pass --disable-sliding-window
(even though vllm does this by default for gemma 2) and set --max-model-len
no larger than 4096, whereas you set it to 8192. Also make sure to set environment variable VLLM_ATTENTION_BACKEND=FLASHINFER
as mentioned in this comment https://github.com/vllm-project/vllm/pull/5908#issuecomment-2217919812 and this issue https://github.com/vllm-project/vllm/issues/6218. As you can see, the Gemma 2 model is not working perfectly for everybody yet, but will probably see improvements shortly.
Yes, so I guess sliding attention support should be added.
Yes, I've encountered the same problem as you. I used the PI extension to expand the length of gemma2 to 16K. Currently, the first issue is that vLLM does not support inference beyond 4096 in length, and the second issue is that it does not support custom rope length extension.
is there any plan to support this in the future?
flashinfer v0.1.2 was just released with sliding window support, I think that vllm's flashinfer.py can now be updated to use it and get the 8k context window of gemma2.
bump
Even if flashinfer==0.1.3
is installed, the vllm code does not support it.
https://github.com/vllm-project/vllm/blob/v0.5.4/vllm/config.py#L163-L171
pip install https://github.com/flashinfer-ai/flashinfer/releases/download/v0.1.3/flashinfer-0.1.3+cu121torch2.3-cp310-cp310-linux_x86_64.whl
@noamgat
Is there another way to use 8k context window?
I apologize if I made a mistake. I am using vLLM 0.5.4 and I set the environment variable VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
, and now my vLLM is working.
Your current environment
🐛 Describe the bug