Open 0xdevalias opened 3 months ago
Hi, thanks for the tips and the reading list, I already had read all of them but LSP and LSIF are still pending.
We are definitely looking into the option of using tree-sitter-graphs. I'm currently weighing two options:
Integrating 'tree-sitter-graph'. This has several pros like performance enhancement, not reinventing the wheel, and meeting industry standards. The problem is that I'm not sure about the complexity it will introduce to the code, especially handling a CLI integration with the tool.
Creating my own implementation of a stack-graph in Python. Keeping everything in-house will simplify the architecture, make maintenance easier, and is a fun challenge 😃 . The downside is obvious: it will be complex and time-consuming to replicate stack-graph logic, and Python doesn't perform as well as Rust.
I think I'm going to explore the integration option, try to make it run, and if it's not too difficult, that's the route I'll go.
I was checking LSP and seems to be a really power full tool. But it does a lot more of what I really need in this stage, because we are focused in give LLM a tool to navigate and understand code base repositories in a easy way. For that I think the best is to implement 'tree-sitter-graph' and later see in what could help LSP.
We are definitely looking into the option of using
tree-sitter-graphs
I think I'm going to explore the integration option, try to make it run, and if it's not too difficult, that's the route I'll go.
@berrazuriz1 I would make sure to look at both tree-sitter-graph
(a more generic underlying lib), but also the more specific stack-graphs
project (which is built on top of tree-sitter-graph
):
eg.
The stack graph rules are written in tree-sitter-graph. Checkout the examples, which contain self-contained TSG rules for specific language features.
The problem is that I'm not sure about the complexity it will introduce to the code, especially handling a CLI integration with the tool.
@berrazuriz1 Yeah, that's fair enough. There are theoretically ways to call/embed rust within python as well I believe, but that may end up being even more complex than shelling out to the CLI
e.g. Some random related links:
PyO3 Rust bindings for Python, including tools for creating native Python extension modules. Running and interacting with Python code from a Rust binary is also supported.
Using Rust from Python
Calling Python in Rust code
I was checking LSP and seems to be a really power full tool. But it does a lot more of what I really need in this stage
@berrazuriz1 Yeah, I think LSP is likely not relevant at this stage. I really only included the links there as part of the "why we aren't using" context.
I think running Rust on Python is a good solution. I'm going to give it a try.
Not really a feature request per se, but came across this project after it was mentioned in another issue (Ref), and figured I would create an issue to share the same info here in case it's useful.
I notice that you're already using
tree-sitter
, so it may not be a lot of extra effort to make use oftree-sitter-graph
and/or thestack-graphs
project (eg.tree-sitter-stack-graphs-javascript
, etc)From watching the demo video on linkedin + briefly skimming this repo, it looks like there might be some useful crossovers, and it may allow you to add support for a whole bunch more languages without needing to re-invent the wheel to do so.
A few notes/links/references I recently collated RE: stack graphs + related libs: