Utilise HPC for Data Generation Scripts - Githubissues

amosproj / amos2024ss08-cloud-native-llm

MIT License

6 stars 1 forks source link

Utilise HPC for Data Generation Scripts #71

Closed grayJiaaoLi closed 2 weeks ago

grayJiaaoLi commented 1 month ago

User story

As a data engineer
I want/need to run data generation scripts on HPC
So that we can have the Q&A pairs training data ready for model training and fine-tuning

Acceptance criteria

Run the scripts for the Q&A pairs data generation on HPC
Ensure the output data is ready for further LLM training and fine-tuning
Store the Q&A pairs on HuggingFace

Definition of done (DoD)

Bill of Materials in the planning document has been updated
All the complex logics have been tested
All feature branches have been merged and closed
New feature code has been documented
Potential new licenses have been checked
All GitHub Actions are passing
The requirement.txt is updated

DoD general criteria

Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release

dominic0df commented 3 weeks ago

added DoD