dosubot[bot]
commented
1 week ago Hey, @harsh-garg98! š I'm here to help you out with any bugs, questions, or contributions you have in mind. Let's squash those bugs together!
The AttributeError occurs because the methods context_relevance_with_cot_reasons and groundedness_measure_with_cot_reasons are not defined in the AzureOpenAI class. The notebook you are following defines a custom class Custom_AzureOpenAI that extends AzureOpenAI and includes a method context_relevance_with_cot_reasons_extreme, but it does not define the methods you are trying to use.
To resolve this issue, you need to either define the missing methods in the AzureOpenAI class or use the methods provided in the Custom_AzureOpenAI class. Here is an example of how you can define the missing methods in a custom class:
from trulens_eval.feedback.provider import AzureOpenAI
from typing import Tuple, Dict
from trulens_eval.feedback import prompts
class Custom_AzureOpenAI(AzureOpenAI):
def context_relevance_with_cot_reasons(self, question: str, context: str) -> Tuple[float, Dict]:
system_prompt = prompts.CONTEXT_RELEVANCE_SYSTEM
user_prompt = str.format(prompts.CONTEXT_RELEVANCE_USER, question=question, context=context)
user_prompt = user_prompt.replace("RELEVANCE:", prompts.COT_REASONS_TEMPLATE)
return self.generate_score_and_reasons(system_prompt, user_prompt)
def groundedness_measure_with_cot_reasons(self, context: str) -> Tuple[float, Dict]:
system_prompt = prompts.GROUNDEDNESS_SYSTEM
user_prompt = str.format(prompts.GROUNDEDNESS_USER, context=context)
user_prompt = user_prompt.replace("GROUNDEDNESS:", prompts.COT_REASONS_TEMPLATE)
return self.generate_score_and_reasons(system_prompt, user_prompt)
# Initialize the custom class
provider = Custom_AzureOpenAI(deployment_name=os.getenv("DEPLOYMENT_NAME"))
# Define feedback functions
f_context_relevance = (
Feedback(provider.context_relevance_with_cot_reasons, name="Context Relevance")
.on_input()
.on(context)
.aggregate(np.mean)
)
f_groundedness = (
Feedback(provider.groundedness_measure_with_cot_reasons, name="Groundedness")
.on(context.collect())
.on_output()
)This should resolve the AttributeError and allow you to proceed with the notebook [1].
To continue talking to Dosu, mention @dosu.
harsh-garg98
sfc-gh-jreini
Bug Description I am following the Azure OpenAI LangChain Quickstart google colab notebook (https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/expositional/models/azure_openai_langchain.ipynb) to set up trulens using Azure OpenAI. I am the 'Initialize Feedback Function(s)' stage and I am getting an error:
According to the documentation the AzureOpenAI class derives from the OpenAI class which derives from the LLMProvider base class. The LLMProvider base class documentation page shows that it has the 'context_relevance_with_cot_reasons' and the 'groundedness_measure_with_cot_reasons' methods.
To Reproduce I am following the notebook and have not changed any bit of code.
Expected behavior The expected behaviour is to get the context relevance and groundedness scores.
Relevant Logs/Tracebacks Code:
Error:
Environment: