dspy.Signature has trouble parsing results

[Treesrule]

So I'm having a problem where the output of a Signature is failing to parse responses from gpt-4o properly The code doing the calling is here

def metric(markdown, base_summary, new_summary): questions = [] with dspy.context(lm=gpt4T): assesment = dspy.Predict(Assess)( question="Does the summary accurately state the base API url", website_markdown=markdown, base_summary=base_summary, new_summary=new_summary ) print(assesment) return 1 if "Yes" in assesment.new_is_better else 0 if "Yes" in assesment.base_is_better else 0.5 Asses is defined here `class Assess(dspy.Signature): """Asses the quality of two different summaries of website_markdown for each question"""

question = dspy.InputField()
website_markdown = dspy.InputField()
base_summary = dspy.InputField()
new_summary = dspy.InputField()
base_is_better = dspy.OutputField()
new_is_better = dspy.OutputField()`

I'm getting response like

Prediction( base_is_better='New Is Better: Yes\n\n---\n\nThe new summary is better because it provides a more concise and clear overview of the general attributes and requirements that apply to all endpoints of the Unbounce API. It effectively highlights key points such as authentication, response metadata, standard query parameters, URI parameters, response body schemas, example response structures, CRUD operations, and error handling. This makes it easier for readers to quickly grasp the essential information without getting bogged down in too much detail.', new_is_better='---\n\nQuestion: Does the summary accurately state the base API url\n\nWebsite Markdown: Unbounce APIGetting Started API Reference Communitytoggle menuDeveloper Portal Getting Started API Reference CommunityUnbounce API Global API Accounts + /{account\_id} - /sub\_accounts - /pages Sub Accounts + /domains + /page\_groups + /pages Domains + /pages Pages + /{page\_id} - /form\_fields - /leads /{lead\_id} - /lead\_deletion\_request /{lead\_deletion\_request\_id} Page Groups Leads Users Global API ========== Global API meta-information. get/ - Retrieve the global API meta-information. ### Response Body Schema ``` { "$schema" : "http://json-schema.org/draft-04/schema#", "description" : "Describes the main API entry points, supported formats and specification locations.", "type" : "object", "additionalProperties" : false, "required" : [ "documentation", "' )

I tried looking for other issues but couldn't find anything simmilar at a glance

[okhat]

Use

dspy.configure(experimental=True)

[Treesrule]

what does that do, also i realized that part of the problem could be that you guys arn't escaping markdown well

[Treesrule]

It worked!