Dspy Assertions make execution sequential

[dvd42]

I am trying to parellelize the calls to the OpeanAI API using the ThreadPoolExecutor, within the function that each thread runs I have the following:

model = CoT(signature)
answer = model(
    source=segment.source,
    translation=segment.mt,
    language=segment.language,
    style_guide_rules=sorted(list(style_guide_rules)),
    glossary_terms=sorted(list(glossary_terms)),
    translation_pairs=similar_translation_pairs,
)

return answer

This runs fine in parallel with no issues. However, adding model = CoT(signature).activate_assertions() makes the execution sequential.

Is there any workaround? Am I doing something wrong? Thanks in advance!

PS:

Here is the CoT class in case its useful:

class CoT(dspy.Module):

    def __init__(self, signature: dspy.SignatureMeta):
        super().__init__()
        self.generate_answer = dspy.ChainOfThought(signature)

    def forward(self, **kwargs):
        pred = self.generate_answer(**kwargs)

        json_answer = re.findall(r"(\{[^}]*\})", pred.answer)
        if json_answer:
            pred.answer = json_answer[0]
        print(pred.answer)
        dspy.Suggest(
            bool(json_answer),
            "answer must be a string that can be passed directly to json.loads",
        )
        return pred

[dvd42]

Any updates? I am still facing this problem, which effectively prevents making many requests in parallel with the retry mechanism. Since the library does not support async requests yet, threads are the main way to achieve this concurrency. Any help would be much appreciated.

[dvd42]

It seems that there is a threading lock in the backrack_handler wrapper . Is it possible to build our own backtrack_handler and use that? How should we go about this? Thanks in advance!

[dvd42]

I should say, I am more than happy to make a PR for this, if in fact, threaded execution is not possible with retries, since this is very important for us at the moment.

Will also be happy to work on Async support which would simplify a lot of this. Let me know if and how I can help :)

[arnavsinghvi11]

Hi @dvd42 , would love to see a PR for this! Currently, DSPy assertions require the lock to ensure the Retries are mapped to the correct requests (and were breaking otherwise) and have not been tested with async. activate_assertions uses the default backtrack_handler, but you can configure a custom backtrack_handler logic compatible with assertions.py and pass it to the assert_transform_module:

Example:

from dspy.primitives.assertions import assert_transform_module#, backtrack_handler

# define custom_backtrack_handler

program_with_assertions = assert_transform_module(ProgramWithAssertions(), custom_backtrack_handler)

[dvd42]

I see. But doesn't the settings singleton handle threads correctly? https://github.com/stanfordnlp/dspy/blob/55510eec1b83fa77f368e191a363c150df8c5b02/dsp/utils/settings.py#L51

Couldn't this be leveraged to get the relevant config for the current thread in the backtrack_handler?

Let me know if I am on the right track here and I'll create a PR :)

[RahulBalan]

@arnavsinghvi11 Am also facing this problem. Can you show an example of custom_backtrack_handler. What all things should be considered in a custom_backtrack_handler

[chiragshah285]

following looking to do the same

[theta-lin]

I am also looking forward to see multi-threaded assertions.

I see. But doesn't the settings singleton handle threads correctly?

https://github.com/stanfordnlp/dspy/blob/55510eec1b83fa77f368e191a363c150df8c5b02/dsp/utils/settings.py#L51

Couldn't this be leveraged to get the relevant config for the current thread in the backtrack_handler?

I think this is a good observation as I also think the only thing making assertions not supporting threads is the sharing of global config (which is not really "config", but rather storing the module call traces and backtracking information globally). However, I also note that each thread does get its copy of the config:

https://github.com/stanfordnlp/dspy/blob/af5186cf07ab0b95d5a12690d5f7f90f202bc86e/dsp/utils/settings.py#L51-L65

This implementation is not without its problems though, as pointed out in this part of the evaluator code also leveraging threads: https://github.com/stanfordnlp/dspy/blob/af5186cf07ab0b95d5a12690d5f7f90f202bc86e/dspy/evaluate/evaluate.py#L153-L157 You can see that as the thread-local config only copies from the main thread, it won't work if threads spawn more threads. Therefore, I can see the problem of maybe using multi-threaded module with assertions inside multi-threaded evaluator.

However, it appears that settings_v2.py is created to address this issue? It would be better for the developers to offer more insight here.

What is the most problematic at hand is that I guess even if we can create our own custom_backtrack_handler, then how are we supposed to build our own thread-safe version of dsp.settings.trace? I mean the access of dsp.settings.trace might be thread-safe, but having multiple threads might mess up the traces.

[enchyisle]

Facing the same issue here. Hope that multi-thread support would be available soon so that we won't need to do work-arounds.

[dvd42]

This seems like a solution.

def funcion_that_your_threads_execute()
  model = your_llm()
  dspy.configure(
      lm=model,
      trace=[],
      assert_failures=0,
      suggest_failures=0,
      backtrack_to=None,
      predictor_feedbacks={},
      backtrack_to_args={},
  )

If I replace the lock with nullcontext (dspy.settings.lock = nullcontext()). This seems to work for me, resulting in consistent outputs with multithreading.

@arnavsinghvi11 @okhat could you confirm if I am missing something?

If this is correct, I could make a PR updating/creating an example wherever you point me to with this use case.

[chiggly007]

@dvd42 this is great thanks! Do you have a working example of the full use of this? I think that would be helpful for everyone. thanks!