search

upstash / degree-guru

AI chatbot for expert answers on university degrees
https://degreeguru.vercel.app/
119 stars 32 forks source link

Hallucinations and unresponsiveness #5

Closed esanmar closed 4 weeks ago

esanmar commented 3 months ago

After doing several crawls of the same website, the answers are not exact at all. I have tried changing models but I have not achieved good results.

But, no results. Thanks!

CahidArda commented 3 months ago

Hi there,

Thank you for trying out our tool!

Firstly, I'm unable to check the repository you shared. It seems to be either deleted or set to private. Could you please make it public?

There are a few things we can check to find the issue:

  1. Check the crawler logs to ensure the website is being crawled correctly. The crawler should print out every URL it visited. Verify that the crawler visits the expected URLs.
  2. Check the vector store through the Upstash Console and see if the vectors are stored as expected. You can use the Data Browser under your vector store. In the metadata of the vectors, there is a "url" field which denotes the source of the chunk. There should be chunks from the visited URLs.
  3. If you have done multiple runs with the crawler without clearing the vector store first, you may have "corrupted" the vector store in a sense. Clear the vector store each time you run the crawler if you haven't already.
  4. Check if you need to update your agent template and initial message. We configured these for Stanford and you may need to change it according to your website.
  5. Which models have you experimented with? If you've tried different embedding models, ensure that you've updated the embedding model settings in both the crawler and the chatbot.
esanmar commented 4 weeks ago

Solved. Thanks!

CahidArda commented 4 weeks ago

Glad to hear that!