MIPRO with ChromadbRM - Throwing error

[pranjal-joshi]

While trying to setup a MIPRO optimizer with ChromadbRM, I am facing the following issue:

Error Logs

Failed to run or to evaluate example Example({'question': 'How to delete a study?', 'answer': 'My answer here'}) (input_keys={'question'}) with <function metric_function at 0x000001B4FB215F70> due to 'AzureOpenAI' object has no attribute 'kwargs'.
100%|██████████| 3/3 [00:00<00:00,  5.27it/s]

ValueError                                Traceback (most recent call last)
Cell In[14], [line 11](vscode-notebook-cell:?execution_count=14&line=11)
      [2](vscode-notebook-cell:?execution_count=14&line=2) teleprompter = MIPRO(
      [3](vscode-notebook-cell:?execution_count=14&line=3)     prompt_model=llm,
      [4](vscode-notebook-cell:?execution_count=14&line=4)     task_model=llm,
      [5](vscode-notebook-cell:?execution_count=14&line=5)     metric=metric_function,
      [6](vscode-notebook-cell:?execution_count=14&line=6)     num_candidates=3
      [7](vscode-notebook-cell:?execution_count=14&line=7) )
      [9](vscode-notebook-cell:?execution_count=14&line=9) kwargs = dict(num_threads=1, display_progress=True, display_table=0)
---> [11](vscode-notebook-cell:?execution_count=14&line=11) MIPRO_compiled_RAG = teleprompter.compile(
     [12](vscode-notebook-cell:?execution_count=14&line=12)     RAG(), 
     [13](vscode-notebook-cell:?execution_count=14&line=13)     trainset=dataset, 
     [14](vscode-notebook-cell:?execution_count=14&line=14)     num_trials=3, 
     [15](vscode-notebook-cell:?execution_count=14&line=15)     max_bootstrapped_demos=1, 
     [16](vscode-notebook-cell:?execution_count=14&line=16)     max_labeled_demos=0, 
     [17](vscode-notebook-cell:?execution_count=14&line=17)     eval_kwargs=kwargs, 
     [18](vscode-notebook-cell:?execution_count=14&line=18)     requires_permission_to_run=False
     [19](vscode-notebook-cell:?execution_count=14&line=19) )

File [c:\Users\Pranjal.Joshi\AppData\Local\anaconda3\envs\teams-bot\lib\site-packages\dspy\teleprompt\mipro_optimizer.py:368](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:368), in MIPRO.compile(self, student, trainset, num_trials, max_bootstrapped_demos, max_labeled_demos, eval_kwargs, seed, view_data, view_examples, requires_permission_to_run)
    [365](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:365)             demo_candidates[id(module_p)].append(candidate_p.demos)
    [367](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:367) # Generate N candidate prompts
--> [368](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:368) instruction_candidates, _ = self._generate_first_N_candidates(module, self.num_candidates, view_data, view_examples, demo_candidates, trainset)
    [370](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:370) # Reset demo_candidates to None for our optimization if the user asked for no fewshot examples
...
--> [227](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:227)         raise ValueError("No examples found for the given predictor")
    [228](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:228)     instruct = None
    [229](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:229)     for i in range(1, self.num_candidates):

ValueError: No examples found for the given predictor

Here is the section of code that I've written

from dspy.retrieve.chromadb_rm import ChromadbRM
from langchain_openai import AzureOpenAI

chromadb_rm = ChromadbRM(
    collection_name='conversations',
    persist_directory='chroma',
    embedding_function=emb_fn,
    k=5  
)

openai.api_type = "azure"
llm = AzureOpenAI(
            deployment_name = "gpt-35-turbo",
            openai_api_type = "azure",
            temperature = 0.7,
            api_key = os.getenv("azure_openai_key"),
            api_version = os.getenv("azure_openai_api_version"),
        )

dspy.settings.configure(rm=chromadb_rm, lm=llm)

# Created my trainset here with [dspy.Example] from some pandas dataframe columns
dataset = [dspy.Example(question=row['question'], answer=row['gold_answer']).with_inputs('question') for index, row in trainset.iterrows()]

class GenerateAnswer(dspy.Signature):
    """My dummy initial prompt that I am trying to tune"""

    context = dspy.InputField(desc="may contain relevant facts")
    question = dspy.InputField()
    answer = dspy.OutputField(desc="output answer of the question")

class RAG(dspy.Module):
    def __init__(self, num_passages=3):
        super().__init__()
        self.retrieve = dspy.Retrieve(k=num_passages)
        self.generate_answer = dspy.ChainOfThought(GenerateAnswer)

    def forward(self, question):
        context = self.retrieve(question).passages
        prediction = self.generate_answer(context=context, question=question)
        return dspy.Prediction(context=context, answer=prediction.answer)

# Configure and run MIPRO optimizer
teleprompter = MIPRO(
    prompt_model=llm,
    task_model=llm,
    metric=metric_function,
    num_candidates=3
)

kwargs = dict(num_threads=1, display_progress=True, display_table=0)

MIPRO_compiled_RAG = teleprompter.compile(
    RAG(), 
    trainset=dataset, 
    num_trials=3, 
    max_bootstrapped_demos=1, 
    max_labeled_demos=0, 
    eval_kwargs=kwargs, 
    requires_permission_to_run=False
)

Let me know what is missing in this.

Note: I've tested that ChromadbRM can get the similarity search results. So

[pranjal-joshi]

UPDATE

LLM must be a dspy wrapper, thus made following changes:

llm = dspy.AzureOpenAI(
            api_base = os.getenv("azure_openai_endpoint"),
            api_version = os.getenv("azure_openai_api_version"),
            model = "gpt-35-turbo",
            model_type = "text",
            temperature = 0.7,
            api_key = os.getenv("azure_openai_key"),
        )

Now the compile logs are not showing errors, However, still the exception has been thrown.

Logs

100%|██████████| 3/3 [00:02<00:00,  1.30it/s]
Bootstrapped 0 full traces after 3 examples in round 0.
100%|██████████| 3/3 [00:00<00:00,  4.42it/s]
Bootstrapped 0 full traces after 3 examples in round 0.

Exception

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[13], [line 11](vscode-notebook-cell:?execution_count=13&line=11)
      [2](vscode-notebook-cell:?execution_count=13&line=2) teleprompter = MIPRO(
      [3](vscode-notebook-cell:?execution_count=13&line=3)     prompt_model=llm,
      [4](vscode-notebook-cell:?execution_count=13&line=4)     task_model=llm,
      [5](vscode-notebook-cell:?execution_count=13&line=5)     metric=metric_function,
      [6](vscode-notebook-cell:?execution_count=13&line=6)     num_candidates=3
      [7](vscode-notebook-cell:?execution_count=13&line=7) )
      [9](vscode-notebook-cell:?execution_count=13&line=9) kwargs = dict(num_threads=1, display_progress=True, display_table=0)
---> [11](vscode-notebook-cell:?execution_count=13&line=11) MIPRO_compiled_RAG = teleprompter.compile(
     [12](vscode-notebook-cell:?execution_count=13&line=12)     RAG(), 
     [13](vscode-notebook-cell:?execution_count=13&line=13)     trainset=dataset, 
     [14](vscode-notebook-cell:?execution_count=13&line=14)     num_trials=3, 
     [15](vscode-notebook-cell:?execution_count=13&line=15)     max_bootstrapped_demos=1, 
     [16](vscode-notebook-cell:?execution_count=13&line=16)     max_labeled_demos=0, 
     [17](vscode-notebook-cell:?execution_count=13&line=17)     eval_kwargs=kwargs, 
     [18](vscode-notebook-cell:?execution_count=13&line=18)     requires_permission_to_run=False
     [19](vscode-notebook-cell:?execution_count=13&line=19) )

File [c:\Users\Pranjal.Joshi\AppData\Local\anaconda3\envs\teams-bot\lib\site-packages\dspy\teleprompt\mipro_optimizer.py:368](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:368), in MIPRO.compile(self, student, trainset, num_trials, max_bootstrapped_demos, max_labeled_demos, eval_kwargs, seed, view_data, view_examples, requires_permission_to_run)
    [365](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:365)             demo_candidates[id(module_p)].append(candidate_p.demos)
    [367](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:367) # Generate N candidate prompts
--> [368](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:368) instruction_candidates, _ = self._generate_first_N_candidates(module, self.num_candidates, view_data, view_examples, demo_candidates, trainset)
    [370](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:370) # Reset demo_candidates to None for our optimization if the user asked for no fewshot examples
...
--> [227](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:227)         raise ValueError("No examples found for the given predictor")
    [228](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:228)     instruct = None
    [229](file:///C:/Users/Pranjal.Joshi/AppData/Local/anaconda3/envs/teams-bot/lib/site-packages/dspy/teleprompt/mipro_optimizer.py:229)     for i in range(1, self.num_candidates):

ValueError: No examples found for the given predictor

[pranjal-joshi]

It seems like the error changes dynamically with code changes but failing exactly at the same line of the compile method call! Check the last few words of the error log

Failed to run or to evaluate example Example({'question': 'How to delete a study?', 'answer': 'Dummy answer', 'context': 'Dummy context'}) (input_keys={'question', 'context'}) with <function metric_function at 0x000001E03FA8A0D0> due to 'str' object has no attribute '_store'.

[XenonMolecule]

This other issue may be related: https://github.com/stanfordnlp/dspy/issues/865

You may consider adding the if statement suggested in that thread to fix your issue while the changes are tested and shipped out more widely!

[pranjal-joshi]

@XenonMolecule Your suggestion is ambigous to me. Could you please explain it further?

[XenonMolecule]

MIPRO bootstraps example traces of your program running to show to an LLM to write better instructions. However, in your case it is logging Bootstrapped 0 full traces after 3 examples in round 0. so the example bootstrapping is failing.

Right now MIPRO doesn't have a check for if that fails which causes the instruction proposal based on instructions to fail, so there is a suggested fix in the thread I linked above to add that if statement into mipro_optimizer.py in order to check that there are examples before proposing instructions based on those examples. We will ship out a fix on our end soon once we test this more broadly!

[okhat]

MIPRO updated. Hopefully fixes this.