Structured Output trouble with Command R

[beatobongco]

TLDR: There seems to be a problem extracting fields when using cohere's command-r

Given this example

class RephraseSearchQueryWithHistory(dspy.Signature):
    """You are an assistant tasked with taking a natural language
    query from a user and converting it into a query for a vectorstore.
    In this process, you strip out information that is not relevant for
    the retrieval task."""

    chat_history = dspy.InputField(
        desc="the chat history of the user and the assistant so far",
        format=passages2text,
    )
    query = dspy.InputField(desc="the most recent user query")

    rephrased_query = dspy.OutputField(desc="the rephrased query in English")

chat_history = [{"role": "assistant", "content": "Hi, I'm a helpful assistant, how can I help you"}, 
                {"role": "user", "content": "Hi, I am a single mother with my 2 children"},
                {"role": "assistant", "content": "Sure, single mother of two lovely children, what do you have in mind?"}]
query = """I have been told I can claim child-rearing finacial support worth 10 million yen, I am new here. 
           Can you help, how can I claim the financial support?"""

pred = dspy.Predict(RephraseSearchQueryWithHistory)
pred(chat_history=chat_history, query=query)

I get this output

Prediction(
    rephrased_query='Chat History: \n- Assistant: "Hi, I\'m here to help you. What\'s your query?"\n-'
)

Checking dspy.settings.lm.inspect_history(n=1) I get

You are an assistant tasked with taking a natural language
    query from a user and converting it into a query for a vectorstore.
    In this process, you strip out information that is not relevant for
    the retrieval task.

---

Follow the following format.

Chat History: the chat history of the user and the assistant so far
Query: the most recent user query
Rephrased Query: the rephrased query in English

---

Chat History:
[1] «{'role': 'assistant', 'content': "Hi, I'm a helpful assistant, how can I help you"}»
[2] «{'role': 'user', 'content': 'Hi, I am a single mother with my 2 children'}»
[3] «{'role': 'assistant', 'content': 'Sure, single mother of two lovely children, what do you have in mind?'}»
Query: I have been told I can claim child-rearing finacial support worth 10 million yen, I am new here. Can you help, how can I claim the financial support?
Rephrased Query:Chat History: 
- Assistant: "Hi, I'm here to help you. What's your query?"
-    (and 75 other completions)

You are an assistant tasked with taking a natural language
    query from a user and converting it into a query for a vectorstore.
    In this process, you strip out information that is not relevant for
    the retrieval task.

---

Follow the following format.

Chat History: the chat history of the user and the assistant so far
Query: the most recent user query
Rephrased Query: the rephrased query in English

---

Chat History:
[1] «{'role': 'assistant', 'content': "Hi, I'm a helpful assistant, how can I help you"}»
[2] «{'role': 'user', 'content': 'Hi, I am a single mother with my 2 children'}»
[3] «{'role': 'assistant', 'content': 'Sure, single mother of two lovely children, what do you have in mind?'}»
Query: I have been told I can claim child-rearing finacial support worth 10 million yen, I am new here. Can you help, how can I claim the financial support?
Rephrased Query:Chat History: 
- Assistant: "Hi, I'm here to help you. What's your query?"
-    (and 75 other completions)

I'd like to help with this issue -- just not tonight since I am braindead.

Any leads appreciated. cc @CShorten you might be interested since you worked on the command-r PR.

[chiggly007]

I think you need a stop sequence, as it sounds like you're getting an insane amount of competition. I had this issue with working with Databricks model (DBRX), and had to add a stop_sequence to the model dbrx = dsp.Databricks(model='databricks-dbrx-instruct',api_key='KEY',api_base="ENDPOINT",model_type='chat',stop=["\n"])

[CShorten]

Hey @beatobongco! I recommend using TypedPredictors with Command R!

class RephraseSearchQueryWithHistory(dspy.Signature):
    """You are an assistant tasked with taking a natural language
    query from a user and converting it into a query for a vectorstore.
    In this process, you strip out information that is not relevant for
    the retrieval task."""

    chat_history: str = dspy.InputField(
        desc="the chat history of the user and the assistant so far",
        format=passages2text,
    )
    query: str = dspy.InputField(desc="the most recent user query")

    rephrased_query: str = dspy.OutputField(desc="the rephrased query in English")

Then you implement with chat_history = dspy.TypedPredictor(RephraseSeachQueryWithHistory)

Code here if interested in digginging in further - https://github.com/stanfordnlp/dspy/blob/main/dspy/functional/functional.py

Here is a little more background from me also showing Assertions / Custom Guardrails approaches to this problem, but for this particular I think TypedPredictor is the way to go - https://github.com/weaviate/recipes/blob/main/integrations/dspy/4.Structured-Outputs-with-DSPy.ipynb

[CShorten]

Hey @beatobongco I changed the title of the issue to avoid confusion that the model itself is not working.

[beatobongco]

Thanks @CShorten will look into this :pray: