-
Faced OOM on Arc with 6k input/512 out with VLLM serving, Mode: ChatGLM3-bB, Qwen1.5-32B on 4 ARC
-
### System Info
env: NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 V100 16G*8
docker images: nvcr.io/nvidia/tritonserver:24.02-trtllm-python-py3
### …
-
Got only 10 parallel request on 2 Arc with Qwen1.5 model (1024 input/512 out), could you please to improve the performance?
-
背景上下文:
https://qwen.readthedocs.io/zh-cn/latest/framework/function_call.html
Originally posted by @GabrielXie in https://github.com/xorbitsai/inference/pull/1598#issuecomment-2159605576
https://g…
-
Thanks for your contribution on accommondating qwen on self-extend.
Qwen1.5 has already been 32k context length. I'm wondering if i can use self-extend to make it to about 100K?
Have you ever test…
-
### Your current environment
VLLM is 0.5.0,A100 , CUDA 12.1
### 🐛 Describe the bug
1、
CUDA_VISIBLE_DEVICES=1 python -m vllm.entrypoints.openai.api_server \
--model /home/Qwen1.5-1.8B-Chat \
…
-
For tokenizers in `transformers`, in convention, `tokenizer.vocab_size` [as documented](https://github.com/huggingface/transformers/blob/092f1fdaa4224fdd88c616dc9678e6fcb37bfffd/src/tran…
-
![1721889849832](https://github.com/user-attachments/assets/7d9b57ed-74cf-4edd-b6aa-61adf2f1bece)
mergekit-moe支持qwen系列的模型吗,我想把两个基于lora微调并合并后的qwen模型与没有微调的qwen合并为moe模型,但是我尝试了qwen、qwen1.5和qwen2三个版本,都得到图…
-
A770,Ubuntu系统
```
for n in $(seq 16 16); do
echo "Model= $MODEL RATE= 0.7 N= $n..."
python3 benchmark_vllm_throughput.py \
--backend vllm \
--m…
-
model.tokenize result is different from tranformers tokenizer result
using the same mode Qwen1.5-7B
input_ids1 = model.tokenize(prompt.encode("utf-8"))
input_ids2 = tokenizer([prompt], padding…