stanfordnlp / dspy

DSPy: The framework for programming—not prompting—foundation models
https://dspy-docs.vercel.app/
MIT License
14.89k stars 1.14k forks source link

optimize_signature for typed predictors needs more documentation #902

Open alcinos opened 3 months ago

alcinos commented 3 months ago

The documentation doesn't specify what to do with the result of optimize_signature The code specifies that it should return a dspy.Program: https://github.com/stanfordnlp/dspy/blob/303c66991899a7fc5123ee5642f4ff4d7f76560f/dspy/teleprompt/signature_opt_typed.py#L149C1-L149C19

However this is not the case and it returns instead a list of (scores, signatures) It is unclear what one should do with the signatures. One possibility is to manually install the signature as follows:

result = optimize_signature(student_program=predictor, ....)

from copy import deepcopy
predictor2 = deepcopy(predictor)
best = result.scores.index(max(result.scores))
predictor2.signature = result.signatures[best]["self"]

This is not good UX (and possibly brittle, eg the key "self" used to be "base" in previous versions) but seems to work locally. It has one major caveat though: the signature won't survive saving the model as it is not returned by predictor2.dump_state(), which makes me feel it's not the intended way.

arnavsinghvi11 commented 3 months ago

Hi @alcinos , thanks for raising this! feel free to push a PR updating the typing inconsistencies and adding documentation. The OptimizerResult is akin to the return values in dspy.Evaluate, but this can be modified to be included with a flag.

Tagging @thomasahle for more context on TypedOptimizers and use cases for the returned signatures.