dosubot[bot]
commented
2 months ago To improve the answer correctness metric score on a context-based dataset, you can focus on enhancing both the factuality and semantic similarity aspects of the model's responses. Here are some specific steps you can take:
-
Improve Factuality:
- Ensure that the model's responses are factually accurate by refining the training data and incorporating more reliable sources.
- Use the
AnswerCorrectnessClassificationto identify and reduce false positives (FP) and false negatives (FN) in the model's responses. Factual correctness quantifies the factual overlap between the generated answer and the ground truth answer using concepts like True Positive (TP), False Positive (FP), and False Negative (FN).
-
Enhance Semantic Similarity:
- Improve the semantic similarity by fine-tuning the
AnswerSimilaritycomponent. This can involve using better embeddings or more sophisticated similarity measures. Semantic similarity measures how closely the generated answer aligns with the ground truth in meaning.
- Improve the semantic similarity by fine-tuning the
-
Optimize Weights:
- Adjust the weights for factuality and semantic similarity to better reflect their importance in your specific use case. The default weights are
[0.75, 0.25], but you can experiment with different values. The final answer correctness score is a weighted average of semantic similarity and factual similarity.
- Adjust the weights for factuality and semantic similarity to better reflect their importance in your specific use case. The default weights are
-
Refine Prompts:
- Improve the prompts used for generating and evaluating responses. Ensure that the
CORRECTNESS_PROMPTandLONG_FORM_ANSWER_PROMPTare clear and comprehensive.
- Improve the prompts used for generating and evaluating responses. Ensure that the
-
Segment Sentences Effectively:
- Ensure that the sentence segmentation is accurate by using a reliable
sentence_segmenter. This helps in breaking down the text into meaningful units for evaluation.
- Ensure that the sentence segmentation is accurate by using a reliable
-
Context Precision:
- Implement the
ContextPrecisionmetric to evaluate whether the context provided was useful in arriving at the given answer. This involves verifying the context's relevance and utility in generating the correct answer.
- Implement the
Here is a code snippet that shows how you might adjust the weights and refine the prompts, along with implementing context precision:
@dataclass
class AnswerCorrectness(MetricWithLLM, MetricWithEmbeddings):
name: str = "answer_correctness"
evaluation_mode: EvaluationMode = EvaluationMode.qga
correctness_prompt: Prompt = field(default_factory=lambda: CORRECTNESS_PROMPT)
long_form_answer_prompt: Prompt = field(default_factory=lambda: LONG_FORM_ANSWER_PROMPT)
weights: list[float] = field(default_factory=lambda: [0.8, 0.2]) # Adjusted weights
answer_similarity: t.Optional[AnswerSimilarity] = None
sentence_segmenter: t.Optional[HasSegmentMethod] = None
max_retries: int = 1
def __post_init__(self: t.Self):
if len(self.weights) != 2:
raise ValueError("Expects a list of two weights. First for factuality, second for semantic similarity")
if all([w == 0 for w in self.weights]):
raise ValueError("At least one weight must be non-zero")
if not all([w >= 0 for w in self.weights]):
raise ValueError("Weights must be non-negative")
if self.sentence_segmenter is None:
language = self.long_form_answer_prompt.language
self.sentence_segmenter = get_segmenter(language=language, clean=False)
@dataclass
class ContextPrecision(MetricWithLLM):
name: str = "context_precision"
evaluation_mode: EvaluationMode = EvaluationMode.qcg
context_precision_prompt: Prompt = field(default_factory=lambda: CONTEXT_PRECISION)
max_retries: int = 1
_reproducibility: int = 1
@property
def reproducibility(self):
return self._reproducibility
@reproducibility.setter
def reproducibility(self, value):
if value < 1:
logger.warning("reproducibility cannot be less than 1, setting to 1")
value = 1
elif value % 2 == 0:
logger.warning("reproducibility level cannot be set to even number, setting to odd")
value += 1
self._reproducibility = value
def _get_row_attributes(self, row: t.Dict) -> t.Tuple[str, t.List[str], t.Any]:
return row["question"], row["contexts"], row["ground_truth"]
def _context_precision_prompt(self, row: t.Dict) -> t.List[PromptValue]:
question, contexts, answer = self._get_row_attributes(row)
return [
self.context_precision_prompt.format(
question=question, context=c, answer=answer
)
for c in contexts
]
def _calculate_average_precision(
self, verifications: t.List[ContextPrecisionVerification]
) -> float:
score = np.nan
verdict_list = [1 if ver.verdict else 0 for ver in verifications]
denominator = sum(verdict_list) + 1e-10
numerator = sum(
[
(sum(verdict_list[: i + 1]) / (i + 1)) * verdict_list[i]
for i in range(len(verdict_list))
]
)
score = numerator / denominator
if np.isnan(score):
logger.warning(
"Invalid response format. Expected a list of dictionaries with keys 'verdict'"
)
return score
async def _ascore(
self: t.Self,
row: t.Dict,
callbacks: Callbacks,
) -> float:
assert self.llm is not None, "LLM is not set"
human_prompts = self._context_precision_prompt(row)
responses = []
for hp in human_prompts:
results = await self.llm.generate(
hp,
callbacks=callbacks,
n=self.reproducibility,
)
results = [
await _output_parser.aparse(item.text, hp, self.llm, self.max_retries)
for item in results.generations[0]
]
responses.append(
[result.dict() for result in results if result is not None]
)
answers = []
for response in responses:
agg_answer = ensembler.from_discrete([response], "verdict")
if agg_answer:
agg_answer = ContextPrecisionVerification.parse_obj(agg_answer[0])
answers.append(agg_answer)
answers = ContextPrecisionVerifications(__root__=answers)
score = self._calculate_average_precision(answers.__root__)
return score
def adapt(self, language: str, cache_dir: str | None = None) -> None:
assert self.llm is not None, "LLM is not set"
logging.info(f"Adapting Context Precision to {language}")
self.context_precision_prompt = self.context_precision_prompt.adapt(
language, self.llm, cache_dir
)
def save(self, cache_dir: str | None = None) -> None:
self.context_precision_prompt.save(cache_dir)By focusing on these areas, you can systematically improve the answer correctness metric score for your context-based dataset [1][2][3].
To continue talking to Dosu, mention @dosu.
Anjalisoni99-byte
[ ] I checked the documentation and related resources and couldn't find an answer to my question.
Your Question what is unclear to you? What would you like to know?
Code Examples This community speaks code. Share your code snippets to help us understand your question better.
Additional context Anything else you want to share with us?