Enhance code agent's search ability with ACR's context search API

[ryanhoangt]

What problem or use case are you trying to solve?

The current search skills available to the agent is:

- search_dir(search_term, dir_path='./'): # Searches for a term in all files in the specified directory.
- search_file(search_term, file_path=None): # Searches for a term in the specified file or the currently open file.

This is versatile but quite coarse-grained, and sometimes the agent doesn't use the skill at all but grep or find instead. (e.g. 1, 2)   Describe the UX of the solution you'd like

We could try to implement some more fine-grained search tools for the agent, which may help the agent to quickly locate the necessary code and possibly increase the possibility that the agent ends up use tools. ACR has defined some:

- search_class(class_name: str): # Search for a class in the codebase
- search_method_in_file(method_name: str, file_path: str): # Search for a method in a given file
- search_method_in_class(method_name: str, class_name: str): # Search for a method in a given class
- search_method(method_name: str): # Search for a method in the entire codebase
- search_code(code_str: str): # Search for a code snippet in the entire codebase
- search_code_in_file(code_str: str, file_path: str): # Search for a code snippet in a given file file

However, because these skills are for coding-oriented tasks, it may be more preferable to only use them in agents specific for coding, instead of generalist agents.

Do you have thoughts on the technical implementation?

The implementation from ACR is well-written and documented so we can reference from there. However, we will need to make some modifications to:

• Enable line number for these tools

Describe alternatives you've considered

None.

Additional context

• With this API as a baseline, we can try to improve further by implementing new tools if needed. (e.g. tree command to get the repository structure)
• Some works on swe-bench-lite that achieve good scores like CodeR or RepoUnderstander also utilize this API, so implement this can be a prerequisite to support these agents if we would like to in the future.

---

P/s: I'm not 100% sure that this is needed, any thoughts are welcome!

[neubig]

Hey @ryanhoangt , since we have a reference implementation this looks interesting to try out!

[rezzie-rich]

This may or may not have some licensing restrictions. Since ACR is under GPL-3.0

[ryanhoangt]

This may or may not have some licensing restrictions. Since ACR is under GPL-3.0

Oh yeah I didn't notice this, maybe it's not desirable to use the original code as is? We can still discuss if we can come up with some ideas inspired by ACR that are useful for OpenDevin

[rezzie-rich]

@ryanhoangt, Instead of ACR, how about implementing 'mentat auto context'. It allows mentat-bot to look up specific code snippets very effectively. This can also be helpful if to incorporate 'rawdog' for testing code before implementing/submitting. #2821

[rezzie-rich]

Also, correct me if I'm wrong. Doesn't it serve the same purpose of pr #2518?

[ryanhoangt]

Instead of ACR, how about implementing 'mentat auto context'.

This issue aims to provide the agent with tools more tailored for coding tasks, improving upon the available search tools OpenDevin already has (using it or not depends on the agent). Mentat's auto context, if I understand correctly, is a way to automatically append the retrieved code snippets for the last user message into the prompt before sending to the LLM. This may be a useful feature we want to incorporate and try out for normal agent usage (I can have a closer look into the implementation), but I'm afraid it may not always help for the swe-bench evaluation of the agent:

• The retrieved code may not be useful for the issue, which negatively affect how the agent behaves accordingly (similarly to how the score decreases #2248)
• Their MentatBot use another approach, not this one, is that correct? (something called Library Context)

Also, correct me if I'm wrong. Doesn't it serve the same purpose of pr https://github.com/OpenDevin/OpenDevin/pull/2518?

They are both to enhance code searching ability I think, just in different approaches.

[rezzie-rich]

Instead of ACR, how about implementing 'mentat auto context'.

This issue aims to provide the agent with tools more tailored for coding tasks, improving upon the available search tools OpenDevin already has (using it or not depends on the agent). Mentat's auto context, if I understand correctly, is a way to automatically append the retrieved code snippets for the last user message into the prompt before sending to the LLM. This may be a useful feature we want to incorporate and try out for normal agent usage (I can have a closer look into the implementation), but I'm afraid it may not always help for the swe-bench evaluation of the agent:

• The retrieved code may not be useful for the issue, which negatively affect how the agent behaves accordingly (similarly to how the score decreases Add Aider-inspired RepoMap #2248)
• Their MentatBot use another approach, not this one, is that correct? (something called Library Context)

Also, correct me if I'm wrong. Doesn't it serve the same purpose of pr #2518?

They are both to enhance code searching ability I think, just in different approaches.

both auto-context and library context in Mentat are designed to draw relevant context specific to the project and issues at hand. However, they do so in slightly different ways:

Auto-context automatically selects relevant code snippets up to a preset token limit (e.g., 5000 tokens) to provide a broad, dynamically generated context. This feature uses embeddings to identify and include code snippets that are most pertinent to the user's current task, which might span across the entire codebase. It's especially useful for tasks that require a general understanding of various parts of the project but don't have predefined specific targets.

Library context, used by tools like Mentat Bot, is more targeted. It specifically draws on detailed documentation and project issues to guide the coding tasks, such as generating pull requests or resolving issues. This approach is structured to pull in context that is directly relevant to the task defined by project documentation or GitHub issues, making it ideal for precise and specific coding tasks.

While both methods aim to integrate relevant information, auto-context is more about breadth and adaptability, whereas library context focuses on the depth and specificity of the task derived from documentation and issue details.

in other words, they are both relatively the same with slight modifications where auto-context is more similar to 'repomap' and library context to ACR's approach primary difference being vector db. maybe a hybrid of repomap, auto-context, and ACR's context search API with switch statements can tackle both use cases more effectively.

[JeffKatzy]

Ok, didn't realize the licensing issues.

@neubig I can take another shot at this, and was exploring using the Jedi pip package, which has an MIT license here. Is that approach worth exploring?

[neubig]

This looks interesting! Although I haven't taken a very close look.

One thing is that now that we already have treesitter integrated into OpenDevin it actually wouldn't be too hard to implement many of the functionalities by just using the ASTs that TreeSitter provides us (which would reduce our dependencies, maintenance burden, etc.).

However, if other tools are better for the job it's still fine to introduce new tools of course!

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for over 30 days with no activity.