Can we have custom evaluator?

[hasalams]

I have a Q&A task, and I want to use a teleprompter to compile the Baleen program.

Instead of using the validation metric validate_cotext_and_answer_hops I want to use my own evaluator, which is a gpt model graded answer. I have eval function which returns true for accepted answers.

The following is the metric function I use

def validate_model_graded_score(example, pred):
    question = example.question
    ideal = example.answer
    return eval(question=question, expected=ideal, predicted=pred)

I get the following error when doing the compilation. AttributeError: 'Example' object has no attribute 'dspy_uuid when using this custom evaluator. Any idea what could be the issue and recommended steps for having a custom evaluator? Thanks.

[hasalams]

Ok, I had to add trace=None to the function

def validate_model_graded_score(example, pred, trace=None):
    question = example.question
    ideal = example.answer
    return eval(question=question, expected=ideal, predicted=pred)

and define teleprompter as follows.

teleprompter = BootstrapFewShot(metric=validate_model_graded_score)

[salbatarni]

I still get the same error even after adding trace=None

[okhat]

We had some updates on main that were not pushed to pip yet.

Now they are pushed. Please do pip install -U dspy-ai to get the latest DSPy package.

[salbatarni]

I installed the latest one but still getting the same error

[okhat]

okay can you paste the error?

[okhat]

I think what's happening is that your metric has an error. Can I see your metric?

[salbatarni]

Oh yes, you are right, I wasn't using the right attribute in the metric! Thanks, it's solved now :)