dosubot[bot]
commented
3 weeks ago Hey, @zongyicoding! I'm here to assist you with any bugs, questions, or contributions. Let me know how I can help you with the Langchain issue.
To correctly wait for the LLM output to be completed when using langchain's chain.stream and then record the complete output in the Trulens dashboard, you can follow this approach:
- Set up the asynchronous task to call the chain with the provided inputs and a callback handler.
- Process the callback iterations to print each token as it is received.
- Ensure the task is completed and retrieve the recording for the Trulens dashboard.
Here's an updated version of your code to achieve this:
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse, FileResponse
from langchain_openai import ChatOpenAI
from langchain.schema import Document
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores.chroma import Chroma
from langchain_core.runnables import RunnablePassthrough
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from trulens_eval import TruChain, Feedback
from trulens_eval.feedback.provider import OpenAI
from trulens_eval import Tru
from langchain.callbacks import AsyncIteratorCallbackHandler
import asyncio
tru = Tru()
tru.run_dashboard(port=9999, force=True)
embedding_function = OpenAIEmbeddings(
model="text-embedding-ada-002",
check_embedding_ctx_length=False
)
docs = [
Document(
page_content="the dog loves to eat pizza", metadata={"source": "animal.txt"}
),
Document(
page_content="the cat loves to eat lasagna", metadata={"source": "animal.txt"}
),
]
db = Chroma.from_documents(docs, embedding_function)
retriever = db.as_retriever()
template = """Answer the question based only on the following context:
{context}
Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)
model = ChatOpenAI(temperature=0, streaming=True, model='gpt-3.5')
provider = OpenAI()
retrieval_chain = (
{"context": retriever, "question": RunnablePassthrough()}
| prompt
| model
| StrOutputParser()
)
f_answer_relevance = (
Feedback(provider.context_relevance_with_cot_reasons, name="Answer Relevance")
.on_input_output()
)
tru_recorder = TruChain(
retrieval_chain,
app_id='conversation_stream',
feedbacks=[f_answer_relevance],
feedback_mode="deferred"
)
app = FastAPI()
async def generate_chat_responses(message):
callback = AsyncIteratorCallbackHandler()
with tru_recorder as recording:
task = asyncio.create_task(
retrieval_chain.acall(
inputs=dict(question=message),
callbacks=[callback]
)
)
response = ""
async for token in callback.aiter():
content = token.replace("\n", "<br>")
print(f'content: {content}')
response += content
yield f"data: {content}\n\n"
await task
record = recording.get()
print(f'Record: {record}')
@app.get("/")
async def root():
return FileResponse("static/index.html")
@app.get("/chat_stream/{message}")
async def chat_stream(message: str):
return StreamingResponse(generate_chat_responses(message=message), media_type="text/event-stream")
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="127.0.0.1", port=8000)Key Changes:
- Callback Handler: Added
AsyncIteratorCallbackHandlerto handle streaming tokens. - Asynchronous Task: Created an asynchronous task to call the chain and process the callback iterations.
- Recording: Ensured the task is completed and retrieved the recording for the Trulens dashboard.
This approach ensures that the streaming outputs are correctly captured and recorded in the Trulens dashboard, providing the correct score after the LLM generation is completed [1][2].
To continue talking to Dosu, mention @dosu.
zongyicoding
Bug Description I am currently using langchain's chain.stream to batch output results to the frontend web page. After the output ends, I cannot correctly capture the output and scores of LLM (there is no data at all on the Dashboard). How can I correctly wait for the LLM output to be completed and then record the complete output in the Trulens dashboard and give the correct score?
To Reproduce
Expected behavior A clear and concise description of what you expected to happen. The following two behaviors are both acceptable.
Complete the waiting for the generation results of the streaming and record the complete information in the Dashboard.
After the LLM generation is completed, provide the response to Trulens for analysis and save it to the dashboard.
Environment:
Additional context Add any other context about the problem here.