Implement Quantitative Evaluation Script

[grayJiaaoLi]

User story

1. As a data engineer,
2. I want / need to implement and automate the calculation of key performance metrics
3. So that we can iteratively evaluate the performance of our LLM in answering CNCF-related factual questions.

Acceptance criteria

• Implement scripts to automatically run the evaluation

  • Use the prepared test data set

• All the selected metrics should be calculated

• This metrics script can be easily called from another script

  • the LLM fine-tuning should use this script to evaluate the model

• Ensure the script is reusable for the future iterated model

• Log the results

• Document the observed limitations and problems

Definition of done (DoD)

• Bill of Materials in the planning document has been updated
• All feature branches have been merged and closed
• New feature code has been documented
• Potential new licenses have been checked
• All GitHub Actions are passing
• The requirement.txt is updated

DoD general criteria

• Feature has been fully implemented
• Feature has been merged into the mainline
• All acceptance criteria were met
• Product owner approved features
• All tests are passing
• Developers agreed to release

[dominic0df]

already invested SP05, still left SP05