Expected dict_keys(['proposed_instruction', 'proposed_prefix_for_output_field']) but got dict_keys(['proposed_instruction'])

[arthurgreef]

Run a simple COPRO compile https://dspy-docs.vercel.app/docs/deep-dive/optimizers/copro

Exception when execution compile compiler.compile(program, trainset=trainset, eval_kwargs=eval_kwargs)

Exception in chat_adpater.p

if fields.keys() != signature.output_fields.keys():
            raise ValueError(f"Expected {signature.output_fields.keys()} but got {fields.keys()}")

Exception is:

Expected dict_keys(['proposed_instruction', 'proposed_prefix_for_output_field']) but got dict_keys(['proposed_instruction'])

[arthurgreef]

Adding code sample to reproduce problem. This problem is also in 2.5.13

import os

from azure.identity import DefaultAzureCredential, get_bearer_token_provider

import dspy
from dspy import Module
from dspy.evaluate import Evaluate
from dspy.datasets.hotpotqa import HotPotQA
from dspy.teleprompt import BootstrapFewShotWithRandomSearch

azure_ad_token = get_bearer_token_provider(
    DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default"
)()

gpt = dspy.LM(
    model="azure/"+os.environ["AZURE_AI_MODEL_GPT_4o_mini"],
    api_version=os.environ["AZURE_OPENAI_API_VERSION"],
    api_base=os.environ["AZURE_OPENAI_API_BASE"],
    azure_ad_token=azure_ad_token,
    max_tokens=250,
)
dspy.settings.configure(lm=gpt)

from dspy.datasets import HotPotQA

dataset = HotPotQA(train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0)

trainset, devset = dataset.train, dataset.dev

class CoTSignature(dspy.Signature):
    """Answer the question and give the reasoning for the same."""

    question = dspy.InputField(desc="question about something")
    answer = dspy.OutputField(desc="often between 1 and 5 words")

class CoTPipeline(dspy.Module):
    def __init__(self):
        super().__init__()

        self.signature = CoTSignature
        self.predictor = dspy.ChainOfThought(self.signature)

    def forward(self, question):
        result = self.predictor(question=question)
        return result

from dspy.evaluate import Evaluate

def validate_context_and_answer(example, pred, trace=None):
    answer_EM = dspy.evaluate.answer_exact_match(example, pred)
    return answer_EM

cot_baseline = CoTPipeline()

devset_with_input = [dspy.Example({"question": r["question"], "answer": r["answer"]}).with_inputs("question") for r in devset]

evaluate = Evaluate(
    devset=devset,
    metric=validate_context_and_answer,
    num_threads=1,
    display_progress=True,
)

evaluate(cot_baseline, devset=devset_with_input)

from dspy.teleprompt import COPRO

teleprompter = COPRO(
    metric=validate_context_and_answer,
    verbose=True,
)

kwargs = dict(num_threads=64, display_progress=True, display_table=0) # Used in Evaluate class in the optimization process

trainset_with_input = [dspy.Example({"question": r["question"], "answer": r["answer"]}).with_inputs("question") for r in trainset]
compiled_prompt_opt = teleprompter.compile(cot_baseline, trainset=trainset_with_input, eval_kwargs=kwargs)

[okhat]

Hey @arthurgreef , I think the issue may be max_tokens=250. Just remove that, or make it much larger like 1500.

[arthurgreef]

That solved the problem. Thanks.