Closed 0xdevalias closed 1 month ago
Would you know how Stack Graphs compare to Lossless Semantic Trees ?
Would you know how Stack Graphs compare to Lossless Semantic Trees ?
@foragerr Not deeply/immediately.. but from skimming that page, some things that popped up in my mind while reading it:
It talks about LSTs being 'format-preserving' compared to ASTs (eg. not losing whitespace/etc). The stack-graphs
lib above is implemented on top of tree-sitter
, which creates Concrete Syntax Trees (CSTs) rather than ASTs, which also capture the full data in a 'lossless' way like these LSTs seem to:
Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited.
That said, I don't believe the stack-graphs themselves are aiming to be CSTs, I think they are more just the higher level 'map' between the files/dependencies/symbols/etc.
Where it talks about 'type-attributed', it seems to basically be saying that instead of just a symbol name, there is a way to know where that symbol comes from/etc; that is something stack graphs inherently handle as well.
OpenRewrite's LST lifecycle suggests that a new LST will be created for each run, as nothing is stored between recipe runs. For a small repo, it may be quick enough to do this, but for a larger repo this may introduce a time penalty. One aspect that the stack-graphs
implementation specifically had a goal of was to avoid needing to re-parse the entire repo whenever any file changed; and is designed in a way that only the relevant parts need to be re-parsed/etc.
Hopefully that's helpful. I haven't dug super deep into stack-graphs
myself yet in a 'practically using them' sense, but so far they have seemed like they will be a super interesting/useful tool; and the fact that these underlying libs are backed by GitHub and used for their code search/etc feels like they should be pretty 'battle tested' at scale/across a large number of languages/etc.
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
Not sure stale is relevant to this?
This seems to be aligned to how some other agents have chosen to go, eg.
aider.chat/2023/10/22/repomap.html
Building a better repository map with tree sitter
Where they saw it as an improvement on their older method:
aider.chat/2023/05/25/ctags.html
Improving GPT-4's codebase understanding with ctags
I understand it not being a current priority; but to discount the concept entirely (particularly without reasoning beyond seemingly personal opinion) seems counterintuitive to getting the best agent/outcome here.
Further to this,
aider
just set a new SOTA and topped the SWE-bench lite leaderboard with 26.3%. While all of that performance gain can't be attributed to just their smart code search/repo map'; I would happily bet that it helped it achieve it:
- https://aider.chat/2024/05/22/swe-bench-lite.html
Aider scored 26.3% on the SWE Bench Lite benchmark, achieving a state-of-the-art result. The current top leaderboard entry is 20.3% from Amazon Q Developer Agent. The best result reported elsewhere seems to be 25% from OpenDevin.
- https://github.com/swe-bench/experiments/pull/7
- https://www.swebench.com/
It will be interesting to see if they end up exploring stack graphs directly, and if that improves their performance further again:
Originally posted by @0xdevalias in https://github.com/princeton-nlp/SWE-agent/issues/38#issuecomment-2138547019
See also:
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
Personally I still think this would be relevant to explore; unless the thought it just to continue to leverage code context via '3rd party' integrations (eg. aider)
See also:
Personally I still think this would be relevant to explore; unless the thought it just to continue to leverage code context via '3rd party' integrations (eg. aider)
I agree, and I believe we're working on different approaches to improve this both with and without additional tools.
That said, it's not fully clear to me how actionable is this issue because it seems very generic, wouldn't it be better as one issue per proposed approach?
it's not fully clear to me how actionable is this issue because it seems very generic, wouldn't it be better as one issue per proposed approach?
@enyst I guess the perspective I opened it for is to highlight a single tool/library (stack graphs) that gives that sort of code context/navigation/etc aspect of things:
I don't know the OpenDevin project enough to know exactly where it would fit in/how to do so/etc. I was hoping to generate some discussion that might then lead to someone with more of that 'other side' of the knowledge figuring how it might better fit in; particularly given that I keep seeing places where it seems like the same sorts of tools are being re-created/etc.
In my original post I included a bunch of other relevant context/links that explain it a bit more/how it works/is used by GitHub's code navigation/etc.
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
This issue was closed because it has been stalled for over 30 days with no activity.
Not really a feature request per se, but just wanted to share some info/references/libs that I've been collating recently RE: stack graphs, and how they can be used for better 'code navigation awareness'/etc; may be useful/interesting for this project: