Identify Webpages for the Kubernetes LLM dataset

[dominic0df]

User story

1. As Data Engineer
2. I want to identify websites for my dataset
3. so that the data could be automatically extracted to a dataset

Acceptance criteria

• [ ] The data of CNCF websites is identified
• [ ] Therefore Key URLs are able to be manuelly configured (f.e. in a yml-File)
• [ ] Outgoing of key URLs, relevant webpages of a website are automatically identified
  • [ ] documentation sites (.pdf, .md)-Files are identified
  • [ ] landscape (.yml, .yaml)-Files are identified
• [ ] The result of the search (URLs) is stored, so that in a previous step data could be automatically retrieved from the webpages

DoD general criteria

• Feature has been fully implemented
• Feature has been merged into the mainline
• All acceptance criteria were met
• Product owner approved features
• All tests are passing
• Developers agreed to release

[dominic0df]

Potential follow up issue:

• Identify other Websites outgoing of a website that could be part of our dataset
• Identify sources of white papers, blog posts, conference presentations

[dominic0df]

Possible approaches

• crawl repos directly
• crawl webpages

[dominic0df]

Already done work in the last sprint: SP 02

[julioc-p]

You can download the output file here: https://huggingface.co/datasets/Kubermatic/cncf-raw-data-for-llm-training/resolve/main/landscape_augmented_repos_websites.yml?download=true