Run Q&A Generator with the Latest Collected Data

[grayJiaaoLi]

User story

1. As a data engineer
2. I want/need to generate Q&A pairs from the CNCF web pages and Stackoverflow row data
3. So that our train set can be enriched

Acceptance criteria

• The data from web pages are generated into Q&A pairs

• The data from Stackoverflow are generated into Q&A pairs

  • Align the format of Q&As from Stackoverflow with ours

• Store the generated pairs on the HuggingFace Q&A dataset repo

Definition of done (DoD)

• Bill of Materials in the planning document has been updated
• All feature branches have been merged and closed
• New feature code has been documented
• Potential new licenses have been checked
• All GitHub Actions are passing
• The requirement.txt is updated

DoD general criteria

• Feature has been fully implemented
• Feature has been merged into the mainline
• All acceptance criteria were met
• Product owner approved features
• All tests are passing
• Developers agreed to release

[dominic0df]

We encountered some issues with HPC and get significantly less resources. Computation takes ages. Christian was in contact with the Chair and HPC support but the cluster does not provide him with any output. We will have to see

[dominic0df]

already spent SP02, Sp03 remaining

[christianwielenberg]

Uploaded Q&A to Huggingface