dosubot[bot]
commented
2 months ago Yes, you can customize CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT when evaluating answer correctness.
The CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT are defined as follows:
CORRECTNESS_INSTRUCTIONS = """\
Given a ground truth and an answer statements, analyze each statement and classify them in one of the following categories:
- TP (true positive): statements that are present in answer that are also directly supported by the one or more statements in ground truth,
- FP (false positive): statements present in the answer but not directly supported by any statement in ground truth,
- FN (false negative): statements found in the ground truth but not present in answer.
Each statement can only belong to one of the categories. Provide a reason for each classification.
"""
CORRECTNESS_PROMPT = Prompt(
name="answer_correctness",
instruction=CORRECTNESS_INSTRUCTIONS,
output_format_instruction=_output_instructions,
examples=[
{
"question": """What powers the sun and what is its primary function?""",
"answer": [
"The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
"The primary function of the sun is to provide light to the solar system.",
],
"ground_truth": [
"The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
"This fusion process in the sun's core releases a tremendous amount of energy.",
"The energy from the sun provides heat and light, which are essential for life on Earth.",
"The sun's light plays a critical role in Earth's climate system.",
"Sunlight helps to drive the weather and ocean currents.",
],
"classification": AnswerCorrectnessClassification.parse_obj(
{
"TP": [
{
"statement": "The primary function of the sun is to provide light to the solar system.",
"reason": "This statement is somewhat supported by the ground truth mentioning the sun providing light and its roles, though it focuses more broadly on the sun's energy.",
}
],
"FP": [
{
"statement": "The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
"reason": "This statement is incorrect and contradicts the ground truth which states that the sun is powered by nuclear fusion.",
}
],
"FN": [
{
"statement": "The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
"reason": "This accurate description of the sun’s power source is not included in the answer.",
},
{
"statement": "This fusion process in the sun's core releases a tremendous amount of energy.",
"reason": "This process and its significance are not mentioned in the answer.",
},
{
"statement": "The energy from the sun provides heat and light, which are essential for life on Earth.",
"reason": "The answer only mentions light, omitting the essential aspects of heat and its necessity for life, which the ground truth covers.",
},
{
"statement": "The sun's light plays a critical role in Earth's climate system.",
"reason": "This broader impact of the sun’s light on Earth's climate system is not addressed in the answer.",
},
{
"statement": "Sunlight helps to drive the weather and ocean currents.",
"reason": "The effect of sunlight on weather patterns and ocean currents is omitted in the answer.",
},
],
}
).dict(),
},
{
"question": """What is the boiling point of water?""",
"answer": [
"The boiling point of water is 100 degrees Celsius at sea level"
],
"ground_truth": [
"The boiling point of water is 100 degrees Celsius (212 degrees Fahrenheit) at sea level.",
"The boiling point of water can change with altitude.",
],
"classification": AnswerCorrectnessClassification.parse_obj(
{
"TP": [
{
"statement": "The boiling point of water is 100 degrees Celsius at sea level",
"reason": "This statement is directly supported by the ground truth which specifies the boiling point of water as 100 degrees Celsius at sea level.",
}
],
"FP": [],
"FN": [
{
"statement": "The boiling point of water can change with altitude.",
"reason": "This additional information about how the boiling point of water can vary with altitude is not mentioned in the answer.",
}
],
}
).dict(),
},
],
input_keys=["question", "answer", "ground_truth"],
output_key="classification",
output_type="json",
)You can modify the CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT to suit your specific requirements [1].
To continue talking to Dosu, mention @dosu.
landhu
[ ] I checked the documentation and related resources and couldn't find an answer to my question.
Your Question can customize CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT when evaluate answer_correctnes
Code Examples I found some statements inaccuracy. can custom CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT?