cannot stream with langchain QA

[linxz-coder]

When I was using langchain Retrieval QA, I noticed that the "streaming = true" feature wasn't working properly, and I realized that the stream effect was only available in the terminal and not in the user interface.

To help out, I went ahead and wrote my own code as an example for everyone to refer to:

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain import PromptTemplate
import chainlit as cl
import time

embeddings = OpenAIEmbeddings(openai_api_key="your-api-key")
db = Chroma(embedding_function = embeddings, persist_directory="your-path")
retriever = db.as_retriever(search_type="similarity", search_kwargs={"k":1})
llm = OpenAI(openai_api_key="your-api-key", temperature=0)

qa = RetrievalQA.from_chain_type(
    llm=llm, 
    chain_type="stuff", 
    retriever=retriever
)

@cl.on_message  # this function will be called every time a user inputs a message in the UI

async def main(message: str):

    msg = cl.Message(
        content="",
    )

    for stream_resp in qa.run(message):
        token = stream_resp
        if token is not None:
            await msg.stream_token(token)
            time.sleep(0.05)

    # send back the final answer
    await msg.send()

[willydouhard]

This is not how streaming works with Langchain. Here is a full example on how to stream intermediate steps and the final answer (introduced in the latest release).

[linxz-coder]

This is not how streaming works with Langchain. Here is a full example on how to stream intermediate steps and the final answer (introduced in the latest release).

Thank you for answering that, I'll definitely check it out.