[Question]: astream_chat is not working whereas achat is working

[mirallm]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

@dosu can you please help me. when i call achat i get back the response whereas astream_chat doesnt give back a response.

          async def setup_chatengine(self):

                  self.chat_engine = self.index_handler.vector_index.as_chat_engine(
                      chat_mode="condense_plus_context",
                      memory="",
                      llm=self.llm,
                      context_prompt=(
                          "You are a chatbot, able to have normal interactions, as well as talk"
                          " about an essay discussing Paul Grahams life."
                          "Here are the relevant documents for the context:\n"
                          "{context_str}"
                          "\nInstruction: Use the previous chat history, or the context above, to interact and help the user."
                      ),
                      verbose=False,
                  )

              async def get_response(self, text: str):
                  try:

                      response = await self.chat_engine.stream_chat(text)

                      print(response.sources)
                      is_reset = 0

[dosubot[bot]]

I found some relevant information that might help you with the issue where astream_chat is not returning a response while achat is working.

1. Similar Threads: There is an open issue discussing a similar problem where stream_chat() does not trigger tool calling, while chat() does. The difference in behavior is due to the different ChatResponseMode used in each function. chat() uses ChatResponseMode.WAIT, which allows for the extraction of a thought-action-input triplet and the triggering of tool calling, whereas stream_chat() uses ChatResponseMode.STREAM, which returns responses as they are generated and does not allow for the extraction of a thought-action-input triplet or the triggering of tool calling ^[1].

2. Configuration and Parameters: Ensure that the astream_chat function is configured correctly. It requires the stream parameter to be set to True for streaming chat responses. Additionally, the model used must be supported for chat, and the API client should be properly initialized with the necessary configurations ^[2].

3. Logging and Debugging: The astream_chat method is decorated with @trace_method("chat"), which suggests that it is being traced for logging purposes. Check the logs for any information that might indicate why the response is not being returned. The logs should include details about the condensed question and context nodes, which are part of the response generation process ^[3].

4. Testing: The astream_chat function is tested to ensure it returns an AsyncIterator and that the full response is not empty. You might want to refer to similar test cases to debug the problem and ensure that the method returns the expected async generator and that the response is properly accumulated from the streamed chunks ^[4].

These points should help you diagnose and resolve the issue with astream_chat.

To continue talking to Dosu, mention @dosu.

[logan-markewich]

@mirallm it could also the llm you are using.

Besides that, this is the proper async streaming syntax fyi

resp = await chat_engine.astream_chat("hello")
async for r in resp.async_response_gen():
  print(r, end="", flush=True)