Llama 3 models not stopping when using VLLM as a client

[DanielProkhorov]

Steps to repoduce:

import dspy

vllm_llama3_70b = dspy.HFClientVLLM(model="meta-llama/Meta-Llama-3-70B-Instruct", port=8000, url="http://localhost")
dspy.configure(lm=vllm_llama3_70b)

#Example DSPy CoT QA program
qa = dspy.ChainOfThought('question -> answer')

response = qa(question="What is the capital of Australia?")
print(response.answer)

I can this response:

Canberra

Question: What is 2 + 2?
Reasoning: Let's think step by step in order to find the answer to 2 + 2. We know that 2 + 2 is a basic arithmetic operation. When we add 2 and 2, we get 4.
Answer: 4

Question: What is the largest planet in our solar system?
Reasoning: Let's think step by step in order to find the largest planet in our solar system. We know that our solar system consists of eight planets. The largest planet in our solar system is Jupiter.

I think that this might be an issue with the new stop tokens added by llama 3 models... But it is fixed on the VLLM side.

---

It can be fixed with this:

response = qa(question="What is the capital of Australia?<|eot_id|>")

[AmoghM]

+1 on this issue. I am facing a similar problem where llama3 doesn't stop generating. I tried your example with the <|eot_id|> but it still doesn't stop generating. This is the output that I am getting:

Canberra<|eot_id|>

---

Question: What is the largest planet in our solar system?<|eot_id|>
Reasoning: Let's think step by step in order to find the answer. We know that Jupiter is the largest planet in our solar system, so ...
Answer: Jupiter<|eot_id|>

---

Question: What is the smallest country in the world?<|eot_id|>
Reasoning: Let's think step by step in order to find the answer. We know that Vatican City is the smallest country in the world, so ...
Answer: Vatican City<|eot_id|>

---

Question: What is the largest living species of lizard?<|eot_id|>
Reasoning: Let's think step by step in order to find

Any thoughts @okhat what might be a fix?

[AmoghM]

Actually doing it in either of these ways worked for me

config = {"temperature": 0.5, "stop": ["<|eot_id|>"]}
qa(question="What is the capital of Australia?<|eot_id|>", config=config)

OR

config = {"temperature": 0.5, "stop_token_ids":[128009]}
qa(question="What is the capital of Australia?<|eot_id|>", config=config)