Conduct Comparison of Evaluation Between Models

[grayJiaaoLi]

User story

1. As a Machine Learning Engineer
2. I want/need to compare the evaluation results between different models.
3. So that we can see if our fine-tuned model improves performance on CNCF-related questions.

Acceptance criteria

• Implement the following evaluations:

  • Fine-tuned LLM: our initially fine-tuned model
  • Base Model: Gemma
  • Equivalent Competitor: LLama 2

• Use the same test data set

  • Organise the As from different models for the same Q
  • Q: "How does the ScaleWorkload function facilitate the scaling of a workload to specified replicas?"
    • Fine-tuned: ...
    • Gemma: ...
    • LLama: ...

• The selected Quantitative Metrics should be also compared for different models

• Documenting our impressions about the fine-tuned LLM's answers

• Document the results and upload the comparison results on Github

  • Label our initially fine-tuned model clearly
  • Leave space for updating evaluation on our further improved model
  • The evaluation of the base model and equivalent competitor can be reusable

Definition of done (DoD)

• Bill of Materials in the planning document has been updated
• All feature branches have been merged and closed
• New feature code has been documented
• Potential new licenses have been checked
• All GitHub Actions are passing
• The requirement.txt is updated

DoD general criteria

• Feature has been fully implemented
• Feature has been merged into the mainline
• All acceptance criteria were met
• Product owner approved features
• All tests are passing
• Developers agreed to release