3 bugs in one testcase: Hang, broken Retry on v2.4.9 and some other bonus bug on main

[tarasglek]

Inspired by "using ai for your metric" and nested dspy pipelines, I found a few bugs that are blocking me

1. 2.4.9 and main both hang. due to lock in https://github.com/stanfordnlp/dspy/blob/78d95df6367c6e30d71f8066e1c464b7c893e154/dspy/primitives/assertions.py#L189 When I try to use a nested DSPy module with num_detector = CoT_NumberDetector().activate_assertions()

2. Once i get rid of 2.4.9 .activate_assertions() on NumberDetector, the Suggestions get ignored during retries(eg we do 3 identical llm calls, and hit the cache). Cause is https://github.com/stanfordnlp/dspy/blob/78d95df6367c6e30d71f8066e1c464b7c893e154/dspy/predict/retry.py#L59 but not sure what the proper fix is. Changing if dspy.settings.backtrack_to == self: to if dspy.settings.backtrack_to is not None makes Suggestion()s get incorporated into retries.

3. With num_detector = CoT_NumberDetector() below, on main I get

   
   Prediction(
   rationale='determine if the phrase has numbers. The phrase contains the number "12" spelled out as "twelve", so it does not have any digits.',
   has_numbers='No'
   )
   Traceback (most recent call last):
   File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 220, in wrapper
   result = func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
   File "/Users/taras/Documents/DeepStructure/dspy/suggestion_bug.py", line 51, in forward
   dspy.Suggest(nested == "No", "Use words instead of numbers")
   File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 74, in __init__
   self.__call__()
   File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 112, in __call__
   raise DSPySuggestionError(
   dspy.primitives.assertions.DSPySuggestionError: Use words instead of numbers

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/taras/Documents/DeepStructure/dspy/suggestion_bug.py", line 56, in ans = qa_unoptimized(question="When was Lincoln born") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/program.py", line 26, in call return self.forward(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 294, in forward return wrapped_forward(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 260, in wrapper output_fields = error_state[0].new_signature.output_fields ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'ChainOfThought' object has no attribute 'new_signature'. Did you mean: 'signature'?

## Minimal testcase
Sorry this isn't more minimal. Not sure why github wont let me file it as attachment.

This is a contrived example where I do a second llm call to produce Suggestion()s to result of first llm call. This is the most faithful example I could come with from docs for using one DSPy pipeline to test another

```python

import re
import os
# os.environ["DSP_CACHEBOOL"] = "False" # for dspy
os.environ["LOG_LEVEL"]="DEBUG" # for dspy

import dspy
from dspy.primitives.assertions import assert_transform_module, backtrack_handler
from dspy.predict import Retry
import json

os.environ["LOG_LEVEL"]="DEBUG" # for dspy
llm = dspy.OpenAI(model="gpt-3.5-turbo")
dspy.settings.configure(lm=llm)

class NumDetector(dspy.Signature):
    """has_numbers=bool(re.search(r'\d+', phrase))"""
    phrase = dspy.InputField()
    rval: str = dspy.OutputField(desc="True or False")

class CoT_NumberDetector(dspy.Module):
    def __init__(self):
        super().__init__()
        self.prog = dspy.ChainOfThought(NumDetector)

    def forward(self, *args, **kwargs):
        ret = self.prog(*args,**kwargs)
        dspy.Suggest(ret.rval == "True" or ret.rval == "False", "Only output True Or False as your answer")
        return ret

# num_detector = CoT_NumberDetector().activate_assertions() # .activate_assertions() causes hang due to following lock:
"""
def backtrack_handler(func, bypass_suggest=True, max_backtracks=2):
    def wrapper(*args, **kwargs):
        error_msg, result = None, None
        with dspy.settings.lock:
"""
num_detector = CoT_NumberDetector() # no hang

class CoT_Answer(dspy.Module):
    def __init__(self):
        super().__init__()
        self.prog = dspy.ChainOfThought("question -> answer")

    def forward(self,  *args, **kwargs):
        ret = self.prog(*args, **kwargs)
        with dspy.context(lm=llm):
            # this gonna hang
            nested = num_detector(phrase=ret.answer)
        print(ret.answer,nested)
        dspy.Suggest(nested.rval == "False", "Use words instead of numbers")
        return ret

qa_unoptimized = CoT_Answer().activate_assertions()

ans = qa_unoptimized(question="When was Lincoln born")
print(ans)
# print(llm.history)

On 2.4.9 with is not None change I'm able to get the following output:

Prediction(
    past_answer='Question: When was Lincoln born\n\nPast Answer: February 12, 1809',
    answer='February twelfth, eighteen hundred and nine'
)

[okhat]

Thank you @tarasglek ! This is indeed composing quite a few things, and I want to understand if the issue is just in the assertions in particular. So we can narrow this down, I suspect everything else is OK?

[tarasglek]

How can I help you narrow these down. There are definitely 2 clear bugs.

1) Retry getting confused and not changing llm params on retries (with code as is above) (number 2 in above list) 2) Deadlock in assertions (with # num_detector = CoT_NumberDetector().activate_assertions() # .activate_assertions() uncommented) (number 1 in above list) 3) This bug i am not able to debug further as a dspy newbie since it's hard to tell what types DSPy expects at different points in code without mypy

I've refactored code to use multiple processes to get around this and am able to use v2.4.9 with assertions and to use DSPy signatures/suggestions as written so long as they run in separate proceses

[arnavsinghvi11]

Hi @tarasglek , there was a recent PR pushed to main for assertions. Can you check if this error is resolved with the latest changes?

[tarasglek]

if i run above code without change with current dspy git:

python suggestion_bug.py 

It fails:

Lincoln was born on February 12, 1809. Prediction(
    rationale='produce the rval. We check if there are any numbers present in the phrase using a regular expression search.',
    rval='True'
)
Traceback (most recent call last):
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 220, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/suggestion_bug.py", line 50, in forward
    dspy.Suggest(nested.rval == "False", "Use words instead of numbers")
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 74, in __init__
    self.__call__()
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 112, in __call__
    raise DSPySuggestionError(
dspy.primitives.assertions.DSPySuggestionError: Use words instead of numbers

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/taras/Documents/DeepStructure/dspy/suggestion_bug.py", line 55, in <module>
    ans = qa_unoptimized(question="When was Lincoln born")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/program.py", line 26, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 294, in forward
    return wrapped_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 260, in wrapper
    output_fields = error_state[0].new_signature.output_fields
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ChainOfThought' object has no attribute 'new_signature'. Did you mean: 'signature'?

[Drzhivago264]

Hello, I am not quite sure what is going on and whether my issue relates to this or not. However, dspy.HFClientVLLM is currently hanging indefinitely if the vllm server is down. I cannot even Ctrl-C out of the process, there is no error code to debug. This is quite problematic for me because I run it on webserver and if the vllm is down then the frontend is also down.

I am currently on dspy-ai 2.4.10

client = dspy.HFClientVLLM(model="casperhansen/llama-3-8b-instruct-awq", port=80, url=split_url.scheme+"://"+split_url.hostname)
dspy.configure(lm=client)
predict = dspy.Predict('document -> sentiment')
response = predict(document=message)  ###<-- it hangs here###

[xnmp]

I faced the same issue, and I fixed it with by modifying this line: dspy.Suggest(nested.rval == "False", "Use words instead of numbers")

With an extra keyword argument:: dspy.Suggest(nested.rval == "False", "Use words instead of numbers", target_module=self.prog.signature)

Basically, dspy.Suggest reruns the target_module when it hits an AssertionError. If the target module isn't specified, it'll rerun the last module that was run. But in this case, you've run another module just to do the assertion check, and this isn't the module you want to be rerunning (instead you want to rerun self.prog).

Alternatively, and this is a bit of a hack, you can use a context manager to freeze the dspy.settings.trace while doing the assertion check, like this following:

from contextlib import contextmanager
import dsp
from copy import copy
@contextmanager
def preserve_trace():
    original_value = copy(dsp.settings.trace)
    try:
        yield
    finally:
        dsp.settings.trace = original_value

And then add it to the CoT_NumberDetector module:

class CoT_NumberDetector(dspy.Module):
    def __init__(self):
        super().__init__()
        self.prog = dspy.ChainOfThought(NumDetector)

    def forward(self, *args, **kwargs):
        ret = self.prog(*args,**kwargs)
        with preserve_trace():
            dspy.Suggest(ret.rval == "True" or ret.rval == "False", "Only output True Or False as your answer")
        return ret