Prepare a small test set for evaluation

[grayJiaaoLi]

User story

1. As a data engineer
2. I need to prepare a small test dataset which contains CNCF-related information
3. So that we can evaluate the performance of initially trained model on it

Acceptance criteria

• Collect reliable information from official CNCF documentation

• The size of the test data can be small, 15 / 30 Qs with corresponding As

  • Answers serve as the "ground truth" v.s. LLM’s responses

• The test data should at least cover the following topics within the CNCF:

  • Basic functionality of projects: Kubernetes, Prometheus, ...
  • Configuration and deployment instruction
  • Dependencies and relationships between the projects:
    • "What container orchestration tools integrate with Prometheus?"

• Store the test data on HuggingFace

  • Ensure scalability of data structure
  • We'll get expert help from Kubernetes

Definition of done (DoD)

• Bill of Materials in the planning document has been updated
• All feature branches have been merged and closed
• New feature code has been documented
• Potential new licenses have been checked
• All GitHub Actions are passing
• The requirement.txt is updated

DoD general criteria

• Feature has been fully implemented
• Feature has been merged into the mainline
• All acceptance criteria were met
• Product owner approved features
• All tests are passing
• Developers agreed to release

[christianwielenberg]

Questions are stored in Wiki

[dominic0df]

already invested Sp03, remaining Sp02, missing store it to HuggingFace