Open theta-lin opened 1 month ago
Hi @theta-lin , thanks for sharing this use case.
Both assertions and typed predictors are viable for this. Assertions have been tested more across some DSPy optimizers (BootstrapFewShot, not MIPROs) than typed predictors, which can make them easier to use for this task
@arnavsinghvi11 Thanks for the clarification, I guess I would lean more towards assertions for now, as Llama 3 8B appears not so good for producing structured output.
A follow up question is that, as shown above, TypedChainOfThought
does not appear to show any actual "chain of thought" in its reasoning.
Question: Who is John Doe?
Reasoning: Let's think step by step in order to
{
"value": "keyword"
}
Retriever Type:
{
"value": "keyword"
}
I wonder whether this is the intended behavior?
Bumping this thread because I have started using optimizers with TypedChainOfThought and resulted in worse model quality because it repeats output in it's reasoning as well. This seems to be random and comes at 50-50 chance when running the evaluations.
However, if I save the model to a file then the reasoning becomes output description 100% every time.
cc: @arnavsinghvi11
I wonder what would be the best way to implement a multiple choice question in DSPy? For example, consider this problem in a RAG pipeline where you what to choose the best retriever type out of either "vector" or "keyword" to query a database for relevant information, how would you ask the LM to do this?
One possible way might be with
ChainOfThought
+Suggestion
:The other way would be using
TypedChainOfThought
+Enum
:I think the first approach appears to work better for now, which could be illustrated with some prompting with Meta-Llama-3-8B-Instruct. Note that the exact conversation has being altered for privacy issues, but the conversation basically looks like this.
The first approach
Prompt:
Output:
The second approach
Prompt:
Response:
Therefore, I found that using assertions with normal
ChainOfThought
might be a better approach as there is an actual reasoning instead of simply repeat the output. I wonder whether there is an explanation regarding whyTypedChainOfThought
simply repeats the output for its reasoning?Also, I tried to also get the LM to do the query writing, so that the query passed to the retriever would not be the original question, but could be keywords if using keyword retriever, for example. Thus, I tried adding
to both signatures. However, the first approach simply failed with the message
I think this might be related with #957, #1001, #1057, #1125, and #1246, and it could be caused by Llama 3 8B's relatively poor function calling ability. Still, the second approach works better also in this case.
Therefore, I wonder what is the best practice for implementing multiple choice? Also, a side question is that am I providing too much prompt regarding the retrievers?