search

vanna-ai / vanna

🤖 Chat with your SQL database 📊. Accurate Text-to-SQL Generation via LLMs using RAG 🔄.
https://vanna.ai/docs/
MIT License
11.04k stars 862 forks source link

Summarization feature not working well - describing the SQL instead of the results #282

Closed ashishsingal1 closed 6 months ago

ashishsingal1 commented 7 months ago

Here's a quick Loom video - https://www.loom.com/share/a69de8e17576450daa0d066b9bce83e1?sid=c35e5e2d-d738-4fd8-952c-b06b42c875ce

Is your feature request related to a problem? Please describe. After asking a question, the summarization tells me what the SQL did instead of summarizing the actual results.

Qn - What is average revenue for every brand, sort them from highest to lowest, format average revenue as thousands with 2 decimal points Summary - The average revenue for each brand is sorted from highest to lowest, formatted in thousands with 2 decimal points.

Describe the solution you'd like I'd like the summary to actually be a description of the results. For example, Brand X was the highest revenue, with X sales, followed by Brand Y with Y sales.

zainhoda commented 7 months ago

Thanks -- we've noticed this as well. gpt-3.5-turbo doesn't do a great job at summarization with the current prompt but gpt-4 and gpt-4-turbo-preview do what we'd expect. If someone wants to take a stab at improving the prompt, that would be super helpful!

ashishsingal1 commented 7 months ago

Note that I ran this again and I got better results - so it may also just be randomness with the LLM.

zainhoda commented 7 months ago

I'd be curious to hear how often it happens. If it happens more than a little then we may need to look into adjusting the prompt, but if it's rare, we can leave it as is for now

zainhoda commented 6 months ago

Looks like it may have been a one-time thing. We've seen generally good summary results since.