Fix the docs formatting issues in ingestion

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

https://ask.astronomer.io/

Apache License 2.0

192 stars 47 forks source link

Fix the docs formatting issues in ingestion #184

Closed sunank200 closed 9 months ago

sunank200 commented 10 months ago

Describe the bug

Currently, the ingestion of data from various sources like astro docs and other sources has formatting issues like one-line headers from blogs. Example look at second source from the top: Screenshot 2023-12-01 at 4 27 48 PM

[ ] Debug this issue. Ingestion of GitHub docs are causing this mostly and in case of HTML pages - ingestion of titles might be causing these
[ ] Fix the issue.
[ ] Validate the formatting is okay
[ ] Test from QA

Lee-W commented 9 months ago

Not yet been able to reproduce it on dev, but this still happen on prod