search

vanna-ai / vanna

🤖 Chat with your SQL database 📊. Accurate Text-to-SQL Generation via LLMs using RAG 🔄.
https://vanna.ai/docs/
MIT License
11.04k stars 862 forks source link

how to train data once there is some errors #299

Closed njalan closed 6 months ago

njalan commented 6 months ago

Can we have a chance to set answer is wrong for training data once there are some errors in the answer.

zainhoda commented 6 months ago

Yes, we should add manual and automated (LLM-based) error correction

andreped commented 6 months ago

Yes, we should add manual and automated (LLM-based) error correction

I think simply passing the error message that is produced when attempting to extract the records to the LLM for a second iteration is a simple solution that should work (together with the rest of the context, current user question and previous sql query as well, of course).

This can be done several times, if of interest, to iteratively converge to the right SQL query. The number of retries could be a parameter that is set through the config.

@zainhoda I can make a PR for this, if the solution sounds sufficient :]

zainhoda commented 6 months ago

Thanks for the offer @andreped! I actually have both of these (manual and automated correction) mostly completed in a local branch and it works quite well. I'm just adding a bunch of options to the built-in UI and I'll include it in the next release

andreped commented 6 months ago

Thanks for the offer @andreped! I actually have both of these (manual and automated correction) mostly completed in a local branch and it works quite well. I'm just adding a bunch of options to the built-in UI and I'll include it in the next release

Beat me to it then 😅 That's great! Looking forward to testing it with the upcoming release!

njalan commented 6 months ago

@zainhoda @andreped Thanks your help. May I know when will be the next release?

andreped commented 6 months ago

@zainhoda Are you able to make this release today? We wanted this feature included before upgrading our Vanna version :]

andreped commented 5 months ago

@zainhoda Where do we allow generate_sql to retry if it fails by passing the error message?

I fail to see where in the code this logic happens. I saw the PR mentioned above and I can see that there is some retry logic for the VannaFlaskApp, but how does this affect me when I am just using Vanna as a library?

I am using vanna==0.3.4 and Azure OpenAI and Chroma clients.

zainhoda commented 5 months ago

@andreped you’re right — right now it’s in the flask app only. Do you have a proposal for where and how this would fit in the base package?