search

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer
https://ask.astronomer.io/
Apache License 2.0
192 stars 47 forks source link

Document sources are wrong in the answers #80

Closed sunank200 closed 8 months ago

sunank200 commented 11 months ago

The answers from the Slackbot return the sources and these sources are wrong most of the time and are very irrelevant.

More details here: https://astronomer.slack.com/archives/C05QJA9LTR9/p1698954492896499

For example: this message on slack has query hello and it is returning various sources

shillion commented 10 months ago

Good example here The sources referenced seem much less relevant than (say) the Airflow docs.

vatsrahul1001 commented 10 months ago

@shillion correct sources/Related docs are not relevant sometimes

phanikumv commented 10 months ago

Might disable this feature if not fixed in time for the Nov 15 release, being discussed here

sunank200 commented 10 months ago

We ingested airflow docs which has improved the reference sources.

sunank200 commented 10 months ago

We also tried Hybrid search with Cohere re-ranking. This did not work well and answers as well as sources degraded. We will be continuing on this after Nov 28th release

Lee-W commented 10 months ago

@sunank200 The last time you mentioned waiting for our next step. Have we received a response? If not, should we move this to the backlog?

phanikumv commented 9 months ago

David will be working on this, as discussed in the sprint planning call

phanikumv commented 9 months ago

@davidgxue are we making progress on this ticket?

cc @shillion @kaxil

davidgxue commented 8 months ago

Closing this issue as it is duplicate/covered by another issue https://github.com/astronomer/ask-astro/issues/133