search

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer
https://ask.astronomer.io/
Apache License 2.0
192 stars 47 forks source link

Test and evaluate the quality of response after ingestion of astro forum docs #192

Closed sunank200 closed 9 months ago

sunank200 commented 10 months ago

After ingestion of astro forum we need to run the same test plan again and evaluate the results.

Depends on: #120

vatsrahul1001 commented 9 months ago

After the ingestion of Astro Forum docs (https://github.com/astronomer/ask-astro/pull/212), I observed that in certain responses, sources are cited from forum docs that lack any corresponding response. Additionally, it seems that the document may not be closely related, and the inclusion of the forum source appears to be a result of a word match in the question asked.

question | Status | crc_answer | Comments -- | -- | -- | -- I want to create a task dynamically does airflow support it? | Pass | https://ankit-test-group.slack.com/archives/C061F4LR4KT/p1702990644128149 | I see forum doc as first source which does has dynamic task in title, however, there was no discussion around dynamic mapping in there.Forum link: https://forum.astronomer.io/t/dynamic-tasks-based-on-runtime-config/1770 How can I extract airflow dag run information? Such as start_date, run_id? | Pass | https://ankit-test-group.slack.com/archives/C061F4LR4KT/p1702990649814709 | Forum doc with only question and not responsesForum link:https://forum.astronomer.io/t/can-airflow-tracks-each-dag-run-based-on-parameter-passed-into-it/1861 I'm using EcsRunTaskOperator in Airflow 2.5.1, how can I wait until the ECS task is finished? | Pass | https://ankit-test-group.slack.com/archives/C061F4LR4KT/p1702990653835769 | Forum link : https://forum.astronomer.io/t/run-next-task-group-if-the-same-task-group-from-previous-dag-run-finished/2546 showing not related to EcsRunTaskOperator, however , releated to task finished What is the best way to skip tasks based on a condition given in the parameters of the dag? | Pass | https://ankit-test-group.slack.com/archives/C061F4LR4KT/p1702990655367769 | Forum link: https://forum.astronomer.io/t/can-airflow-tracks-each-dag-run-based-on-parameter-passed-into-it/1861 looks like not related list all extras of astronomer-providers? | Fail | https://ankit-test-group.slack.com/archives/C061F4LR4KT/p1702993436719699 | Not related Forum Link: https://forum.astronomer.io/t/invoice-option-for-astronomer-certification-bundles/2414 Overall I see no degradation in responses and sources, however, we should look into the above mentioned issue, tracking it [here](https://github.com/astronomer/ask-astro/issues/213#issuecomment-1862975671)