Open cwlseu opened 1 month ago
It could be because of model compatibility. Currently, vLLM needs Medusa heads to be in a separate checkpoint dir and formatted in a certain way. Can you please try reformatting the Medusa heads using the model class from here? Here's a reference for how the checkpoint should look: https://huggingface.co/abhigoyal/vllm-medusa-vicuna-7b-v1.3/tree/main.
I'll soon release the heads I trained for Mistral and Llama3 models as well, and try to see if this implementation can be modified to support the format for already trained checkpoints on HF.
It could be because of model compatibility. Currently, vLLM needs Medusa heads to be in a separate checkpoint dir and formatted in a certain way. Can you please try reformatting the Medusa heads using the model class from here? Here's a reference for how the checkpoint should look: https://huggingface.co/abhigoyal/vllm-medusa-vicuna-7b-v1.3/tree/main.
I'll soon release the heads I trained for Mistral and Llama3 models as well, and try to see if this implementation can be modified to support the format for already trained checkpoints on HF.
is medusa supported topk? whta is default number
It could be because of model compatibility. Currently, vLLM needs Medusa heads to be in a separate checkpoint dir and formatted in a certain way. Can you please try reformatting the Medusa heads using the model class from here? Here's a reference for how the checkpoint should look: https://huggingface.co/abhigoyal/vllm-medusa-vicuna-7b-v1.3/tree/main.
I'll soon release the heads I trained for Mistral and Llama3 models as well, and try to see if this implementation can be modified to support the format for already trained checkpoints on HF.
hello, thanks for your work on medusa implementation. How do you train your medusa head? Is the training code same with the official version from FlashDecoding which is based on axolotl? In different training code, there are some small difference in medusa model.
Why is vLLM with Medusa 20% slower than TGI?
Why is vLLM with Medusa 20% slower than TGI?
Yep, I've observed similar results that align with your findings and those of others.
@cadedaniel I'm particularly curious about the root causes of this performance gap. Could you please provide any advice or insights on how to optimize vLLM with respect to Medusa speculative decoding? Any insights into factors contributing to this performance difference would be valuable.
Proposal to improve performance
Test new feature medusa speculative sampling with vllm v0.5.2. After using Medusa speculative sampling, the performance dropped significantly(mistral 7b tokens qps from 133 -> 93). Could it be that the model formats supported by vLLM are incompatible with those trained in FastDecoding? The same model in TensorRT-LLM achieves a 2x speedup.
model download: https://huggingface.co/FasterDecoding/medusa-1.0-zephyr-7b-beta
Report of performance regression
Llama 13 B 84 tokens/s -> 66 tokens/s
Without speculation
Processed prompts: 100%|███████████████████| 1/1 [00:01<00:00, 1.72s/it, est. speed input: 26.21 toks/s, output: 83.88 toks/s] Processed prompts: 100%|███████████████████| 1/1 [00:01<00:00, 1.70s/it, est. speed input: 26.40 toks/s, output: 84.48 toks/s] Processed prompts: 100%|███████████████████| 1/1 [00:01<00:00, 1.71s/it, est. speed input: 26.37 toks/s, output: 84.39 toks/s] 0.011855694982740615 text: 'The current president of the United States is Joe Biden. He was inaugurated on January 20, 2021, and is the 46th president of the United States. Prior to his presidency, Biden served as a United States Senator from Delaware for 36 years and as the 47th Vice President of the United States under President Barack Obama. As president, Biden has focused on addressing the COVID-19 pandemic, promoting racial justice and equity, and addressing climate change. He has also emphasized the importance of bipartisanship and working with Congress to pass legislation that benefits the American people.'
With speculation
Processed prompts: 100%|███████████████████| 1/1 [00:02<00:00, 2.37s/it, est. speed input: 19.02 toks/s, output: 60.85 toks/s] Processed prompts: 100%|███████████████████| 1/1 [00:02<00:00, 2.17s/it, est. speed input: 20.73 toks/s, output: 66.35 toks/s] Processed prompts: 100%|███████████████████| 1/1 [00:02<00:00, 2.17s/it, est. speed input: 20.72 toks/s, output: 66.30 toks/s] 0.01508857806523641 text: 'The current president of the United States is Joe Biden. He was inaugurated on January 20, 2021, and is the 46th president of the United States. Prior to his presidency, Biden served as a United States Senator from Delaware for 36 years and as the 47th Vice President of the United States under President Barack Obama. As president, Biden has focused on addressing the COVID-19 pandemic, promoting racial justice and equity, and addressing climate change. He has also emphasized the importance of bipartisanship and working with Congress to pass legislation that benefits the American people.'
Mistral 7B: 133 tokens/s -> 93 tokens/s
Without speculation
Processed prompts: 100%|██████████████████| 1/1 [00:01<00:00, 1.98s/it, est. speed input: 21.70 toks/s, output: 129.19 toks/s] Processed prompts: 100%|██████████████████| 1/1 [00:01<00:00, 1.92s/it, est. speed input: 22.34 toks/s, output: 133.02 toks/s] Processed prompts: 100%|██████████████████| 1/1 [00:01<00:00, 1.92s/it, est. speed input: 22.36 toks/s, output: 133.10 toks/s] 0.007516195066273212 text: "As the current president of the United States, Joe Biden has been leading the country through unprecedented times. Elected in November 2020, Biden has faced numerous challenges, including a global pandemic, economic uncertainty, and social unrest. Despite these obstacles, Biden has remained steadfast in his commitment to serving the American people and working to address the issues that matter most to them.\n\nBorn in Scranton, Pennsylvania in 1942, Biden has spent his entire career in public service. He served in the Senate for over three decades, representing Delaware, and was elected as vice president under Barack Obama in 2008. Biden's experience and expertise in government have been invaluable as he navigates the complexities of the presidency.\n\nOne of Biden's top priorities has been addressing the COVID-19 pandemic. He has worked closely with health experts and scientists to develop a comprehensive response plan, which includes a nationwide vaccination campaign, increased testing and contact tracing, and targeted mitigation measures. Biden has also called for a renewed focus on public health infrastructure and resources, recognizing the importance of investing in prevention and preparedness.\n\nEconomic recovery has also"
With speculation
Processed prompts: 0%| | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]WARNING 07-25 07:13:58 multi_step.py:57] Prompt logprob is not supported by multi step workers. (e.g., speculative decode uses multi step workers). Processed prompts: 100%|███████████████████| 1/1 [00:03<00:00, 3.27s/it, est. speed input: 13.16 toks/s, output: 78.32 toks/s] Processed prompts: 100%|███████████████████| 1/1 [00:02<00:00, 2.74s/it, est. speed input: 15.67 toks/s, output: 93.31 toks/s] Processed prompts: 100%|███████████████████| 1/1 [00:02<00:00, 2.74s/it, est. speed input: 15.67 toks/s, output: 93.30 toks/s] 0.010721690952777863 text: "As the current president of the United States, Joe Biden has been leading the country through unprecedented times. Elected in November 2020, Biden has faced numerous challenges, including a global pandemic, economic uncertainty, and social unrest. Despite these obstacles, Biden has remained steadfast in his commitment to serving the American people and working to address the issues that matter most to them.\n\nBorn in Scranton, Pennsylvania in 1942, Biden has spent his entire career in public service. He served in the Senate for over three decades, representing Delaware, and was elected as vice president under Barack Obama in 2008. Biden's experience and expertise in government have been invaluable as he navigates the complexities of the presidency.\n\nOne of Biden's top priorities has been addressing the COVID-19 pandemic. He has worked closely with health experts and scientists to develop a comprehensive response plan, which includes a nationwide vaccination campaign, increased testing and contact tracing, and targeted mitigation measures. Biden has also called for a renewed focus on public health infrastructure and resources, recognizing the importance of investing in prevention and preparedness.\n\nEconomic recovery has also"
Misc discussion on performance
No response
Your current environment (if you think it is necessary)