search

pragunbhutani / dbt-llm-tools

RAG based LLM chatbot for dbt projects
Other
60 stars 14 forks source link

Add a training module #4

Open jacobSingh opened 7 months ago

jacobSingh commented 7 months ago

Simple thumbs up / thumbs down on responses.

Just thinking out loud here, but where the RAG doesn't pick the right model, it would be nice to get feedback on this so you can improve the docs / examples.

Next steps:

  1. Just store it somewhere so DE teams can improve docs manually where stuff might be missing
  2. Provide users the option to chose from other models which might be missing from the RAG on a down vote (manual intervention here leads to better informed decision making for the bot later)
  3. Automatically / semi-automatically start adding multi-shots to your prompts when you see patterns of poor responses?

Another idea (which is related but not the same): Auto-docs for DBT from reading data in the tables. Should improve accuracy massively. Even if it doesn't go into DBT, but just additional data in the RAG

pragunbhutani commented 7 months ago

Yeah this is in line with a larger concept where the chatbot seeks clarification interactively where it isn't about how to respond and remembers that information for the future - trying to emulate how an actual person would learn an organisation's knowledge over time...