dosubot[bot]
commented
1 month ago Issue 1: Defining Feedback Functions for chain.astream
When using chain.astream, the structure of the data and the way you access it can differ from chain.invoke. Here’s how you can correctly define your feedback functions:
-
Understanding the Data Structure:
- For
chain.invoke, you usedSelect.Record.app.middle[0].invoke.retsto get the context. - For
chain.astream, you need to adapt the selectors to match the streaming data structure.
- For
-
Defining Feedback Functions:
- Use
on_input()to specify the main app input. - Use
on_output()to specify the main app output. - Use
on()to specify other parts of the app record.
- Use
Here’s an example of how you can define your feedback functions for chain.astream:
from trulens_eval import Feedback, Select
import numpy as np
# Define feedback functions
def feedback_groundedness_function(context, response):
groundedness = openai_provider.groundedness_measure_with_cot_reasons(context, response)
return groundedness
f_groundedness = (
Feedback(feedback_groundedness_function, name="Groundedness")
.on(Select.Record.app.middle[0].rets) # Adjust selector for astream
.on_output()
.aggregate(np.mean)
)
f_context_relevance = (
Feedback(openai_provider.context_relevance_with_cot_reasons, name="Context Relevance")
.on_input()
.on(Select.Record.app.middle[0].rets) # Adjust selector for astream
.aggregate(np.mean)
)
f_feedback_cot = Feedback(feedback_cot_function, name="Answer Relevance COT").on_input().on_output()Issue 2: Slow astream Output
The astream method can be slower due to several potential bottlenecks:
- Network Latency: Streaming data can be affected by network latency.
- Token and Cost Tracking: Managing and processing chunks can introduce overhead.
- Garbage Collection: Memory leaks or inefficient garbage collection can slow down the process.
- Callback Handling: Inefficient management of callbacks can introduce delays.
- Model Configuration: Ensure the model and its configuration are optimized for streaming.
- Latency in LLM API Calls: Delays in API calls can impact performance.
To mitigate these issues, consider the following:
- Optimize network conditions.
- Ensure efficient garbage collection and memory management.
- Optimize callback handling and model configuration.
- Monitor and address any latency in LLM API calls.
Example Usage with TruChain
Here’s an example of how to use TruChain with the defined feedback functions and chain.astream:
from trulens_eval import TruChain, TruSession
from app.utils.langchain_funcs import get_chain
from app.datastructures import Conversation, Message
from langchain.memory import ConversationBufferMemory
from langchain_core.messages.ai import AIMessageChunk
# Initialize TruSession
tru = TruSession()
tru.reset_database()
tru.start_dashboard()
# Define conversation
conversation = Conversation(
title="Ny Samtale",
id="conversation-1234",
filename="sample_file.txt",
indexName="livsvitenskap",
promptType="juridisk ekspert",
messages=[Message(role="user", content="Hei, hva er de juridiske implikasjonene for saken?")]
)
# Get chain
chain = await get_chain(conversation=conversation, prompt_type="RESPONSE")
# Initialize memory
memory = ConversationBufferMemory(memory_key="history", return_messages=False)
for message in conversation.messages:
if message.role == "user":
memory.chat_memory.add_user_message(message.content)
elif message.role == "assistant":
memory.chat_memory.add_ai_message(message.content)
history = memory.load_memory_variables({})["history"]
# Initialize TruChain with feedback functions
tru_recorder = TruChain(
app_name="testapp",
app=chain,
provider=openai_provider,
feedbacks=[f_feedback_cot, f_groundedness, f_context_relevance]
)
# Stream responses
result = ""
async with tru_recorder as recording:
async for chunk in chain.astream({"conversation": conversation, "history": history}):
if "response" in chunk:
content = chunk["response"]
if isinstance(content, AIMessageChunk):
result += content.content
else:
raise Exception("Something went wrong in yield chunk in chain response, try opening a new chat")This setup should help you correctly define and use feedback functions with chain.astream and address the performance issues.
To continue talking to Dosu, mention @dosu.
tomdgr
sfc-gh-jreini
sfc-gh-chu
sfc-gh-dhuang
sfc-gh-pdharmana
Bug Description Issue number 1:
Is there a way to figure out how to correctly pass the on_input(), on() and on_ouput() to the feedback-functions in general? for .invoke it was Select.Record.app.middle[0].invoke.rets to get the context, on_input() got the answer, and on_output() got the response. However this is now not the same for chain.astream
This is maybe different for different langcain-chains, so here is mine added:
which results in the following output from
chain = await get_chain(conversationt=conversation,prompt_type="RESPONSE"):So the main question is: how do I figure out how to properly define my feedback-functions based on a langchain? The examples provided in examples/notebooks/ was to simple and I also could not find any in the documentation
Issue number 2:
To Reproduce
Expected behavior Groundedness, Context Relevance and Answer Relevance all with COT in dashboard, comparably fast to chain.invoke, when calling chain.astream instead.
Relevant Logs/Tracebacks Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. If the issue is related to the TruLens dashboard, please also include a screenshot.
Environment:
3.11.8 python trulens 1.0.1 trulens-apps-langchain 1.0.1 trulens-core main ON PR #1487 trulens-dashboard 1.0.1 trulens_eval 1.0.1 trulens-feedback 1.0.1 trulens-providers-openai 1.0.1 langchain 0.2.16 langchain-cli 0.0.31 langchain-community 0.2.17 langchain-core 0.2.40 langchain-openai 0.1.25 langchain-text-splitters 0.2.4
Additional context The datastructures used can be found in my previous issue #1487