Open mikeedjones opened 2 months ago
@okhat @arnavsinghvi11 have you been able to reproduce this issue at all? Would be great to get this fixed!
Would be great to get a view on how best to fix this! Cheers :)
Potentially https://github.com/stanfordnlp/dspy/issues/1312 could be linked?
The extend generation logic in
dsp/primitives/predict.py
is producing the behaviour illustrated below.The example uses an artificially low max_tokens to illustrate the bug with real LM calls.
Explanation
When the model does not generate all the correct field prefixes in the correct order, logic in
dsp/primitives/predict.py
is used to continue the generation until the signature is completed.https://github.com/stanfordnlp/dspy/blob/55510eec1b83fa77f368e191a363c150df8c5b02/dsp/primitives/predict.py#L95
The "Last field"'s value is set to
""
- presumably to throw away a partial completion?https://github.com/stanfordnlp/dspy/blob/55510eec1b83fa77f368e191a363c150df8c5b02/dsp/primitives/predict.py#L98
And a further call to the LM is made with the updated prompt - with the "last field" as the trailing prefix - ie
Whilst the prompt is rendered correctly,
template.extract
compares the field value toNone
to decide where the new completion from the LM is meant to go in the signature:https://github.com/stanfordnlp/dspy/blob/55510eec1b83fa77f368e191a363c150df8c5b02/dsp/templates/template_v2.py#L156
Since the value of the last field is
""
the new completion is instead attached to a field which is stillNone
. This logic stands in contrast to how the prompt was built - where the field value is compared to""
to determine if it is "complete".https://github.com/stanfordnlp/dspy/blob/55510eec1b83fa77f368e191a363c150df8c5b02/dsp/templates/template_v2.py#L81
The bug can be fixed by either
None
to the partial fieldTemplateV2.extract
logic to check for""
when assigning outputsTemplateV2.query
logic to exclude fields with empty-string values as completed when determining which field to leave as the trailing prefix.Speculation
I think the bug makes Dspy fail more quietly when the LM does not generate the fields as specified in the
"Follow the following format"
section of the prompt - as reported by https://github.com/stanfordnlp/dspy/pull/1169#issuecomment-2179472268 after a now reverted change to the extend generation logic (#920). The current logic means that at least one extra OutputField is added to the results for every recursion ingenerate
- independent of what the LM produces. I think this means complex signatures will be reliably completed (up to the user-provided recursion depth) even if the LM is not actually generating the required field prefixes.I think a "fix" to this bug might cause some examples to start to "fail" more loudly when the extend generation logic reaches its maximum recursion depth.
A more atomised fix to this bug would ensure that the last incomplete field is skipped correctly when the generation is extended, with appropriate documentation.
Cheers