[Bug]: ReAct Agent ending prematurely and returning the Thoughts to the user as a response

[dbeltranruiz]

Bug Description

Hi,

I noticed that when the LLM responds to the ReAct Agent by using the strict format for the response including the backticks

Though: bla bla
Action: blala

The agent return the thought to the user instead of continuing with the identified action. I found that the problem is in the _infer_stream_chunk_is_final function within the step.py in the code snipped below (this is for the version 0.10.53 but I believe the newest version has the same issue)

if len(latest_content) > len("Thought") and not latest_content.startswith(
                "Thought"
            ) 

The problem is that the backticks (```\n) are the first characters so it never find the Though at the beginning of the response from the LLM. Modifying the code with an additional guard as shown below solves the issue.

if len(latest_content) > len("Thought") and not latest_content.startswith(
                "Thought"
            ) and not "Though" in latest_content:

Let me know if you need more information. And Thanks for the great framework!

Version

0.10.53

Steps to Reproduce

Use gpt3-5 as the llm and the ReactAgent. With gpt4 it also happens but less frequently.

Relevant Logs/Tracbacks

No response

[dosubot[bot]]

To fix the issue in the _infer_stream_chunk_is_final function where the agent returns the thought to the user instead of continuing with the identified action due to the backticks at the beginning of the response, you need to modify the function to handle the backticks properly. Specifically, you should strip any leading backticks from the latest_content before checking its content.

Here is the updated _infer_stream_chunk_is_final function:

def _infer_stream_chunk_is_final(self, chunk: ChatResponse) -> bool:
    """Infers if a chunk from a live stream is the start of the final
    reasoning step. (i.e., and should eventually become
    ResponseReasoningStep — not part of this function's logic tho.).

    Args:
        chunk (ChatResponse): the current chunk stream to check

    Returns:
        bool: Boolean on whether the chunk is the start of the final response
    """
    latest_content = chunk.message.content
    if latest_content:
        # Strip leading backticks
        latest_content = latest_content.lstrip('`')
        if len(latest_content) > len("Thought") and not latest_content.startswith(
            "Thought"
        ):  # doesn't follow thought-action format
            return True
        elif "Answer: " in latest_content:
            return True
    return False

This modification ensures that any leading backticks are removed before the content is checked, allowing the function to correctly determine whether the chunk is the start of the final response ^[1][2].

To continue talking to Dosu, mention @dosu.

[dbeltranruiz]

I can't fix the issue as it is from the llama index side and I don't want to fix it locally as with any upgrade it will be overwritten.

[dosubot[bot]]

This issue seems to need your expertise, @logan-markewich, could you take a look?

To continue talking to Dosu, mention @dosu.