Ingest Documents from Astronomer's Cosmos Website - Githubissues

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

https://ask.astronomer.io/

Apache License 2.0

201 stars 47 forks source link

Ingest Documents from Astronomer's Cosmos Website #277

Closed davidgxue closed 9 months ago

davidgxue commented 10 months ago

Please describe the feature you'd like to see Start data ingestion from Astronomer's Cosmos website to provide knowledge regarding Cosmos related questions. Website: https://astronomer.github.io/astronomer-cosmos/contributing.html

Describe the solution you'd like

Add 1 DAG to ingest from Cosmos documentation website
Add task(s) in the bulk ingestion DAG to ingest from Cosmos doc website

Acceptance Criteria

[ ] Add 1 DAG to ingest from Cosmos documentation website
[ ] Add task(s) in the bulk ingestion DAG to ingest from Cosmos doc website
[ ] Create additional questions to test quality of ingestion
[ ] Ensure response quality of answers to existing test questions in evaluation dataset do no degrade