search

microsoft / MInference

[NeurIPS'24 Spotlight] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
https://aka.ms/MInference
MIT License
816 stars 38 forks source link

[Question]: vllm-tp generate can't stop #90

Open unicorneeee opened 11 hours ago

unicorneeee commented 11 hours ago

Describe the issue

I hope this message find you well. when i use the hjiang branch to use Qwen2.5-7B-instruct, it's running on the InfinitBench. The issue is as follow: {"id": 0, "prediction": " The pass key is 71432. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass is green. The sun is yellow. The sky is blue. The grass", "ground_truth": ["71432"]} Although it answer the question correctly and it will be judge as True because the compute_score in Infinitbench will strip the unusful tokens, however I print the all generate tokens, there isn't the token, which means there may be some mistakes?

unicorneeee commented 11 hours ago

I use the vllm-tp with minference in A100, and the case come from the task passkey in Infinitbench