Automate the test plan to run via script based on test plan

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

https://ask.astronomer.io/

Apache License 2.0

192 stars 47 forks source link

Automate the test plan to run via script based on test plan #143

Closed sunank200 closed 8 months ago

sunank200 commented 10 months ago

Currently, the test plan is being run manually and evaluated.

Automate running test plan via script as manually running it is error-prone.
Run the script to answer questions from APIs.
Run the script to retrieve the answers directly from Weaviate.
Populate the results to the cloud bucket for evaluation.
Evaluation is manual currently

Depends on: #142

sunank200 commented 10 months ago

@mpgreg can you link this issue to the PR you have created? @vatsrahul1001 can add more details as required, please?

mpgreg commented 10 months ago

Some feedback.

Should consider NOT providing 'answer' to vectordb search. The point of the search is to see which links are retrieved. This will save cost too. Use near_text instead of with_ask.
Remove duplicates from references.

sunank200 commented 10 months ago

@vatsrahul1001 @mpgreg can you create the PR for this, please?

vatsrahul1001 commented 10 months ago

@sunank200 as discussed this can be done after 28th milestone, also @mpgreg can you raise the PR of your script whenever you have bandwidth afterwards we can close this

phanikumv commented 10 months ago

Waiting for @mpgreg to create the PR with the script that he developed

vatsrahul1001 commented 10 months ago

@mpgreg we need to plan this out, let us know if there is any update on this?

sunank200 commented 10 months ago

@mpgreg would you be able to share the code URL for this if you already have in your fork?

phanikumv commented 9 months ago

Waiting for an ETA on this @mpgreg

sunank200 commented 9 months ago

@mpgreg This is important for testing phase 2.5 changes. Any update on this, else we might need to change the phase 2.5 release deadlines.

phanikumv commented 9 months ago

@mpgreg can you link the PR to this issue please?