Open 0xdevalias opened 3 months ago
This seems to be aligned to how some other agents have chosen to go, eg.
aider.chat/2023/10/22/repomap.html
Building a better repository map with tree sitter
Where they saw it as an improvement on their older method:
aider.chat/2023/05/25/ctags.html
Improving GPT-4's codebase understanding with ctags
I understand it not being a current priority; but to discount the concept entirely (particularly without reasoning beyond seemingly personal opinion) seems counterintuitive to getting the best agent/outcome here.
Further to this,
aider
just set a new SOTA and topped the SWE-bench lite leaderboard with 26.3%. While all of that performance gain can't be attributed to just their smart code search/repo map'; I would happily bet that it helped it achieve it:
- https://aider.chat/2024/05/22/swe-bench-lite.html
Aider scored 26.3% on the SWE Bench Lite benchmark, achieving a state-of-the-art result. The current top leaderboard entry is 20.3% from Amazon Q Developer Agent. The best result reported elsewhere seems to be 25% from OpenDevin.
- https://github.com/swe-bench/experiments/pull/7
- https://www.swebench.com/
It will be interesting to see if they end up exploring stack graphs directly, and if that improves their performance further again:
Originally posted by @0xdevalias in https://github.com/princeton-nlp/SWE-agent/issues/38#issuecomment-2138547019
Stack Graphs (an evolution of Scope Graphs) sound like they could be really interesting/useful with regards to code navigation, symbol mapping, etc. Perhaps we could use them for module identification, or variable/function identifier naming stabilisation or similar?
cargo install --features cli tree-sitter-stack-graphs-javascript
tree-sitter-stack-graphs-javascript index SOURCE_DIR
tree-sitter-stack-graphs-javascript status SOURCE_DIR
tree-sitter-stack-graphs-javascript query definition SOURCE_PATH:LINE:COLUMN
Potentially Related
See Also