[Question]: Using Mistral-7B-Instruct-v0.2 with ReAct

mathur-exe commented 8 months ago

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question

I am trying to replicate ReAct agent with Query Engine Tool but using Mistral-7B-Instruct-v0.2 from HuggingFace. Every thing suggested by the blog post has been followed to the dot, while the only difference is the LLM and context for ReAct agent

llm = HuggingFaceLLM(
    model_name="mistralai/Mistral-7B-Instruct-v0.2", 
    tokenizer_name="mistralai/Mistral-7B-Instruct-v0.2", 
    device_map="auto", 
    generate_kwargs={"temperature": 0.001, "do_sample":True}, 
    max_new_tokens=1024
)

embed_model = LangchainEmbedding(
    HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2")
)

Settings.llm = llm
Settings.embed_model = embed_model

context = """
    <s> [INST] You are a stock market sorcerer who is an expert on the companies Lyft and Uber. [/INST]
    [INST] You will answer questions about Uber and Lyft as in the persona of a sorcerer 
    and veteran stock market investor. [/INST] 

    [INST] I want you to reponse in a particular manner. For example: for this prompt, Compare and contrast the risks of Uber and Lyft in 2021, then give an analysis [/INST]
    [INST] This is how you should reply
             Thought: To compare and contrast the risks of Uber and Lyft in 2021, I will use the tools lyft_10k and uber_10k.
            Action: lyft_10k, uber_10k
            Action Input:
            {
            "input": "Please provide a detailed analysis of the risks faced by Uber and Lyft in 2021, including but not limited to financial risks, regulatory risks, and operational risks. Please provide specific examples and data to support your analysis."
            }
            Observation:
            {
            "output": "Lyft and Uber both faced significant risks in 2021, but the nature and severity of those risks differed between the two companies.    
    [/INST]
</s> 
"""

agent = ReActAgent.from_tools(
    query_engine_tools,
    llm=Settings.llm,
    verbose=True,
    context=context
)

But when I call the agent.chat,

response = agent.chat("what was Lyft's revenue growth in 2021?")
print(str(response))

this is the error I'm getting TemplateError: Conversation roles must alternate user/assistant/user/assistant/...

After searching a bit, I found this github issue but this does not solve either. After making the changes suggested in this issue,

response = agent.chat(
    [{
    "role": "assistant",
    "content": "You are a stock market sorcerer with expertise in companies Lyft and Uber",
    "role": "user",
    "content": "what was Lyft's revenue growth in 2021?"
    }],
)
print(str(response))

this is the new error ValidationError: 1 validation error for Task input str type expected (type=type_error.str)

dosubot[bot] commented 8 months ago

Hey @mathur-exe, good to see you back with another intriguing challenge! How's it been since our last coding adventure?

It seems like there are two issues you're encountering. Let's address them one by one.

TemplateError: Conversation roles must alternate user/assistant/user/assistant/...

This error is raised when the conversation history does not alternate between the user and the assistant. In your case, the context provided to the ReActAgent does not follow this pattern. The context should be a conversation history where each message has a role (either "user" or "assistant") and content (the text of the message). Here's an example of how you might structure your context:

context = [
   {"role": "assistant", "content": "You are a stock market sorcerer who is an expert on the companies Lyft and Uber."},
   {"role": "user", "content": "Compare and contrast the risks of Uber and Lyft in 2021, then give an analysis."},
   {"role": "assistant", "content": "To compare and contrast the risks of Uber and Lyft in 2021, I will use the tools lyft_10k and uber_10k."},
   {"role": "assistant", "content": "lyft_10k, uber_10k"},
   {"role": "assistant", "content": "Please provide a detailed analysis of the risks faced by Uber and Lyft in 2021, including but not limited to financial risks, regulatory risks, and operational risks. Please provide specific examples and data to support your analysis."},
   {"role": "assistant", "content": "Lyft and Uber both faced significant risks in 2021, but the nature and severity of those risks differed between the two companies."}
]

ValidationError: 1 validation error for Task input str type expected (type=type_error.str)

This error is raised when the input to the chat method is not a string. In your second example, you're passing a list of dictionaries to the chat method, which expects a string. You should pass a string to the chat method, like so:
```
response = agent.chat("what was Lyft's revenue growth in 2021?")
print(str(response))
```

I hope this helps! If you have any other questions or run into more issues, feel free to ask.

Sources

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

logan-markewich commented 8 months ago

Try supplying the messages to prompt callable. Huggingface refuses to accept some inputs that use system prompts. In this case, I would not format the react prompt with the special chars needed for mistral, and leave it for the callable hook to handle.

A basic example of what I mean using mistral instruct format

def messages_to_prompt(messages):
  prompt_str = "\n".join([str(x) for x in messages])
  return f"<s>[INST] {prompt_str} [/INST] </s>\n"

def completion_to_prompt(completion):
  return f"<s>[INST] {completion} [/INST] </s>\n"

llm = HuggingFaceLLM(
    model_name="mistralai/Mistral-7B-Instruct-v0.2", 
    tokenizer_name="mistralai/Mistral-7B-Instruct-v0.2", 
    device_map="auto", 
    generate_kwargs={"temperature": 0.001, "do_sample":True}, 
    max_new_tokens=1024,
    messages_to_prompt= messages_to_prompt,
    completion_to_prompt= completion_to_prompt,
)

mathur-exe commented 8 months ago

Thanks alot @logan-markewich the issue has been resolved.

run-llama / llama_index