Open 0xdevalias opened 6 months ago
This seems to be aligned to how some other agents have chosen to go, eg.
aider.chat/2023/10/22/repomap.html
Building a better repository map with tree sitter
Where they saw it as an improvement on their older method:
aider.chat/2023/05/25/ctags.html
Improving GPT-4's codebase understanding with ctags
I understand it not being a current priority; but to discount the concept entirely (particularly without reasoning beyond seemingly personal opinion) seems counterintuitive to getting the best agent/outcome here.
Further to this,
aider
just set a new SOTA and topped the SWE-bench lite leaderboard with 26.3%. While all of that performance gain can't be attributed to just their smart code search/repo map'; I would happily bet that it helped it achieve it:
- https://aider.chat/2024/05/22/swe-bench-lite.html
Aider scored 26.3% on the SWE Bench Lite benchmark, achieving a state-of-the-art result. The current top leaderboard entry is 20.3% from Amazon Q Developer Agent. The best result reported elsewhere seems to be 25% from OpenDevin.
- https://github.com/swe-bench/experiments/pull/7
- https://www.swebench.com/
It will be interesting to see if they end up exploring stack graphs directly, and if that improves their performance further again:
Originally posted by @0xdevalias in https://github.com/princeton-nlp/SWE-agent/issues/38#issuecomment-2138547019
It would be interesting to see how aider
's existing repo map compares/contrasts with stack graphs/similar; and whether that would improve the performance on the SWE-bench lite even further.
I'm not sure if this would add anything substantially different/better than the existing
tree-sitter
implementation, but I spent some time recently looking into stack graphs + related tooling, and I'm wondering if there may be benefits to this project to look at usingtree-sitter-graph
and/or thestack-graphs
project (eg.tree-sitter-stack-graphs-javascript
, etc); rather than just plaintree-sitter
.A few notes/links/references I recently collated RE: stack graphs + related libs: