search

askthecode / documentation

MIT License
86 stars 14 forks source link

answers on helm repo are not useful #28

Closed emily-zall closed 7 months ago

emily-zall commented 7 months ago

hey guys, I love the idea but it didn't work very well for me on a helm repo: https://chat.openai.com/share/e6bfe886-83f0-409c-a253-fcfc10ced10e

dsomok commented 7 months ago

Hey @emily-zall,

Thanks for reaching out and for your interest in AskTheCode!

Let me start from the explanation of why it produced such a poor response. To extract the data needed to reply to your request, the GPT could use 2 separate flows (the following flows are the most suitable for your case, but the AskTheCode supports much more. All features can be found in the documentation):

  1. File Contents Flow. In this flow, the GPT queries the repository structure, then tries to elaborate based on full file names, which can be relevant to the query, and then queries the contents of those files.
  2. Search Code Flow. In this flow, it will try to extract search keywords out of your prompt and do the search across the whole repository to find suitable files.

What happened in your case is that for your first prompt it used the File Contents Flow, which resulted in poor results since the repository is quite large and none of the file names directly contain a reference to the watchAnyNamespace.

Then you've specified that you are looking for watchAnyNamespace: true, which now resulted in the usage of the Search Code Flow. However, the whole string watchAnyNamespace: true was used as a search term. This search term indeed yields only 2 results, which are not sufficient to answer your query: image

The initial prompt can be improved by splitting your prompt into 2 steps:

  1. Understand what the watchAnyNamespace parameter is.
  2. Then it can deduce what setting it to true means.

I've composed a sample conversation that shows how the prompt can be altered: https://chat.openai.com/share/b7f25fb1-bc15-4a8d-8cc9-29f8dcea20cb

As always with LLMs, prompting plays a very important role, and even small rephrasing can significantly improve the response.

However, I can definitely see the place for improvement here on the AskTheCode side. I will investigate improving the custom instructions on my end so that it will better distinguish between the flows and choose the better flow. Also, search term selection can be improved as well.

What do you think, does it make sense?

Thank you once again for your contribution and feedback.

emily-zall commented 7 months ago

very helpful, thanks!