search

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer
https://ask.astronomer.io/
Apache License 2.0
201 stars 47 forks source link

Slack bot returning source documents in different order from backend/website/LangSmith #250

Closed davidgxue closed 10 months ago

davidgxue commented 11 months ago

Describe the bug The Ask Astro Slack bot is returning related documents in a different order from that of backend or on the ask astro website or the one on LangSmith. This is occuring on the dev server.

To Reproduce Steps to reproduce the behavior:

  1. Redeploy cloud build ask-astro-dev-00289-s58 and serve 100% traffic there for the dev server

  2. Go to the dev server's front end website, in my case https://hybrid-search-reword-and-rer.ask-astro.pages.dev/requests/097dca16-ab93-11ee-82fc-42004e494300, then send a question "What is the Astro SDK?"

  3. See results and related documents returned image

  4. Go to Ask Astro's slack bot channel and ask the same question "What is the Astro SDK?" image

  5. See the the related documents returned are ordered differently. The first document on the website is https://docs.astronomer.io/learn/astro-python-sdk-etl, but the first document returned from the slack bot is https://astro-sdk-python.readthedocs.io/en/stable/aep/README.html.

  6. Checking on LangSmith, the ordering of documents on the website/backend is correct, but the ordering from slack is different. See link here https://smith.langchain.com/public/66cee2ee-507c-480b-bd8f-3098ea93833a/r.

Expected behavior The slack bot's related documents returned should be the exact same ordering as the ones from backend/website and on LangSmith.

pankajkoti commented 11 months ago

@sunank200 I have other assignments as per Sprint planning. Assigned it to you