Data Ingestion Improvement/Cleanup/Bug Fix - Part 2

[davidgxue]

Code ready for review, evaluations/testing is still in progress

Description

• This is part 2 of the data ingestion improvements. The goal is to analyze and investigate any potential issues with the data ingestion process, remove noisy data such as badly formatted content or gibberish text, find bugs, and enforce standards on the process to have predictable cost estimation/reduction.

Technical Changes

• Renamed the original split.py file to chunking_utils.py (multiple files changed due to import naming)
• During the above change, found 2 DAGs somehow were NOT changed. Provider docs and SDK docs's incremental ingest DAGs somehow never had a chunking/splitting task (even though the bulk ingest does). I added chunking to these 2 DAGs
• Airflow Docs extraction process has been improved(see code for details)
• Provider Docs extraction process has been improved (see code for details)
• Astro SDK Docs extraction process has been improved (see code for details) -> Astro SDK docs are not currently being ingested, but fixing code since they exist.
• Provider docs used to ingest all past versions leading to many duplicate docs in the DB. It is now changes to only ingest the stable version of the docs.

Evaluations

• Answer quality generally improved with some add on of more concise answering and linking hyperlinks when the model knows the answer is not exhaustive.
• Please attached for a quick list of results

• data_ingest_comparison_part_2.csv data_ingest_results_part_2.csv

• Bulk ingest also runs with no issues

Related Issues

closes #221 closes #258 closes #295 (Reranker has been addressed in GCP environment variables, embedding model change completed in a different PR) closes #285 (This PR prevents empty docs from being ingested)

[cloudflare-workers-and-pages[bot]]

Deploying with    Cloudflare Pages

Latest commit:	3142709
Status:	✅  Deploy successful!
Preview URL:	https://dc48aa3a.ask-astro.pages.dev
Branch Preview URL:	https://improve-data-ingestion-pt2.ask-astro.pages.dev

View logs

[davidgxue]

Updated the PR description with test results!!