ReAct module not parsing output properly

[lukakoning]

Hi, I have started to use DSPy, unfortunately I am running into issues with the ReAct-module.

I have this code:

class Mod(dspy.Module):
    def __init__(self):
        super().__init__()

        class Signature(dspy.Signature):
            """Provide information about available tables."""
            question = dspy.InputField()
            answer = dspy.OutputField()

        self.prog = dspy.ReAct(
            Signature,
            tools=[ListTables()]
        )

    def forward(self, question):
        answer = self.prog(question=question)

        dspy.Suggest(
            answer['answer'] != 'answer',
            "Do not call Finish[...] with the literal text 'answer' within it. "
            "Instead, call Finish[...] with the actual content of the answer."
        )

        # print(answer)
        return answer

mod = Mod().activate_assertions()
prediction = mod.forward("can you list the names of the tables that are available?")
prediction

Unfortunately, that leads to this error:

Traceback (most recent call last): File "<input>", line 30, in <module> File "C:\Users\dhrko\PycharmProjects\localllm-dspy\.venv\Lib\site-packages\dspy\primitives\assertions.py", line 318, in forward return wrapped_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dhrko\PycharmProjects\localllm-dspy\.venv\Lib\site-packages\dspy\primitives\assertions.py", line 242, in wrapper result = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "<input>", line 16, in forward File "C:\Users\dhrko\PycharmProjects\localllm-dspy\.venv\Lib\site-packages\dspy\primitives\program.py", line 26, in __call__ return self.forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dhrko\PycharmProjects\localllm-dspy\.venv\Lib\site-packages\dspy\predict\react.py", line 114, in forward if action_val := self.act(output, hop): ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dhrko\PycharmProjects\localllm-dspy\.venv\Lib\site-packages\dspy\predict\react.py", line 105, in act raise e File "C:\Users\dhrko\PycharmProjects\localllm-dspy\.venv\Lib\site-packages\dspy\predict\react.py", line 88, in act action_name, action_val = action.strip().split("\n")[0].split("[", 1) ^^^^^^^^^^^^^^^^^^^^^^^ ValueError: not enough values to unpack (expected 2, got 1)

When I inspect the history of the model, things do seem to look good:

...

Thought 2: I should probably finish the task.

Action 2: Finish[Kerncijfers wijken en buurten 2023, Gezondheid per wijk en buurt; 2012/2016/2020/2022 (indeling 2022), Gebieden in Nederland 2013, Wmo-arrangementen; stand, instroom, uitstroom, regio, 2015-2020, and Voortijdig schoolverlaters; geslacht, onderwijssoort en herkomst.]

Apparently something is wrong in the text parsing done by the ReAct module?

[arnavsinghvi11]

Hi @lukakoning , dspy.ReAct is indeed experimental and may require some patches to the text parsing.

Could you try printing what the action variable has before it is parsed? ideally, it should be the contents following "Action 2:"

Also, does this error come up when you remove the Assertions-related code? ReAct and Assertions haven't been fully tested together but curious if its on the latter's end for us to fix the parsing

[okhat]

Thanks for opening this! We released DSPy 2.5 yesterday. I think the new dspy.LM and the underlying dspy.ChatAdapter will probably resolve this problem.

Here's the (very short) migration guide, it should typically take you 2-3 minutes to change the LM definition and you should be good to go: https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb

Please let us know if this resolves your issue. I will close for now but please feel free to re-open if the problem persists.