search

bazelbuild / vscode-bazel

Bazel support for Visual Studio Code
https://marketplace.visualstudio.com/items?itemName=BazelBuild.vscode-bazel
Apache License 2.0
233 stars 77 forks source link

Implement language server for Starlark #1

Open enriched opened 5 years ago

enriched commented 5 years ago

First off, I'm really excited to see the Bazel team creating a repository for vscode support and I would love to contribute to it!

Secondly, Is this Repository going to be developing a Language Server Protocol implementation? Seems like it could be a great opportunity to develop a resource that could be reused in other editors as well.

laurentlb commented 5 years ago

Inside Google, we have an implementation of LSP in progress (with completion, jump to definition, and diagnostics). The bad news is that it relies on an internal framework. I expect it to be open-sourced early 2019, but I have little control on that.

I created this repository to publish a debugger (to execute bzl files step by step and observe variables), I hope it will be added soon. And I'd like this repository to replace https://github.com/devoncarew/bazel-code (from @devoncarew) to have everything in one place.

enriched commented 5 years ago

Thanks for the info! I'll look forward to that release in 2019 and it also sounds good to have everything in one place. Does this plan on bringing in things from https://github.com/dprogm/vscode-bazel-tools (from @dprogm) as well? That extension seems more focused on running bazel tasks.

devoncarew commented 5 years ago

And I'd like this repository to replace https://github.com/devoncarew/bazel-code (from @devoncarew)

👍

laurentlb commented 5 years ago

If @dprogm is willing to contribute, that'd be great too!

(I can give write access if someone wants to help maintaining the repository)

dprogm commented 5 years ago

I also like the idea of having all bazel related vscode plugins in one place. So I would definitely join and contribute.

OmarTawfik commented 5 years ago

@laurentlb is that progress published somewhere? Or if was announced in a roadmap? It would be really exciting to see a starlark LSP for vscode :)

mcuadros commented 5 years ago

Any update about a starlark LSP?

jin commented 4 years ago

/sub

fisheye-123 commented 4 years ago

any update on lsp support?

laurentlb commented 4 years ago

No, sorry. There's some code at Google that I hoped would get open-sourced. It's unfortunately not happening.

If someone wants to work on a LSP implementation, I'll be happy to provide some guidance.

guw commented 4 years ago

@laurentlb How significant is that internal framework? Is it possible to open source without it and then pile down on re-implementing the missing pieces? Or is forking a Python LS a better option (like https://marketplace.visualstudio.com/items?itemName=phgn.vscode-starlark seems have done?

kigster commented 3 years ago

Adding my own +1 for the LSP implementation.

josiahsrc commented 3 years ago

Hey everyone! I have a team working on creating an LSP implementation for Bazel. Our project is a fork of this repo, we are working off of the develop branch, source code here. @laurentlb I've reached out to you on this issue before (apologies for all the pings), but we got little traction.

We're creating an implementation of a Bazel language server using Java. Why we're using java is outlined here. As it stands, our language server currently provides local path completion for deps and minimal syntax highlighting. We are currently working to expand our feature set to intelligently integrate with bazel. Our goal is to provide

That's the MVP of what we hope to provide. I'm posting here and hoping to get some guidance on how to overcome some issues we're running into. The biggest question we have right now is:

We believe that we need to evaluate starlark source in order to interpret which Bazel rules are valid as well as which attributes on a particular bazel rule are valid. We are thinking that our language server should analyze the WORKSPACE file for any given project and evaluate the starlark code to determine which rules are valid as well as their API. This will allow us to provide autocompletion/semantic highlighting for all BUILD files in any project. The issue here is that this code is locked down in the bazel repository.

We are trying to avoid mimicking/mocking the behavior Bazel, we would much rather hook into what already exists. We would really appreciate any feedback or ideas on how to deal with this. Thanks for any help.

Edit: Here is a better visualization of the problem we're trying to solve:

Bazel issues

cpsauer commented 2 years ago

Joining the chorus of folks who would love this, subscribing, and maybe helping connect a few useful dots.

@josiahsrc, any updates on how things are going? Not sure if you ever heard back from @laurentlb. Looks like ownership was passed off to @philwo and @coeuvre (who'll be back shortly after GitHub mysteriously pulled his profile) in #248.

Maybe one of them can help you get updates/contact on the effort inside google? Esp given this is the first, and seemingly most-wanted request? Seems like people here are pretty excited. Wonder if Google would be willing to release parts of theirs for use, open source or not. There might also be some goodies in the IntelliJ plugin, which has great BUILD autocomplete, IIRC.

And @laurentlb, I see elsewhere on the interwebs that you're working on building a new IDE for Google? Anything we vscode-bazelers should know about the efforts there?

[Normally I'd roll up my sleeves on this one, but I should stay focused on solving the autocomplete-in-source issue (#179 w/ bazel-compile-commands-extractor) esp since I also have a startup I need to focus on.]

josiahsrc commented 2 years ago

Hi @cpsauer 👋

We took the bazel language server as far as we had band-width for. Ultimately, we had to drop it due to 1) our capstone course ending (which was the reason we originally started building it) and 2) there not being enough support behind it. After capstone, I took the liberty of making some last minute changes to get the language server into a usable state. The source code for that is on this branch which has the features listed in this presentation. I can't guarantee that this LS implementation has held up over a year, but I'm happy to distribute a VSIX file for anyone interested in trying it out.

Having said that and having built a prototype for a Bazel LS, I have a much more clear idea of how to go about building a stable LS for Bazel. I don't have the resources at the moment to build out a stable LS alone, but I would be happy to contribute/consult on a group project for implementing an LS for Bazel provided there enough of a need for it and enough support from inside google.

cpsauer commented 2 years ago

Wow, that was fast! Thanks for the update! Hadn't realize it was a capstone project, but it makes me smile that you sought out such a real issue and went after it.

The presentation videos are super great. I mean, I'd love a version of this plugin that could do all that :)

I guess the question is: Based on your learnings, do you think it'd need rebuilding to be useful to people longer term, going forward?

Hopefully we'll also hear back on the google efforts and figure out a good way to move forward

josiahsrc commented 2 years ago

I uploaded a VSIX compression of the LS here. Again, there's no guarantee on stability, but it may be somewhat helpful. As a note, the server might get stuck or become out of sync with what's in memory. In such cases you can use these commands to restart or sync the server.

image

For stability, yes. Our LS used the java implementation of the Starlark parser which was taken directly from the main Bazel repo. This implementation has since drifted from how we use it in the LS. Long term, it would be wisest to use the golang implementation, which promises a more stable API.

cpsauer commented 2 years ago

Heh, spectacular! Awesome to get label autocompletion--and to get fast feedback when targets have been typed wrong. Love the logo, too :) Thank you, and go BYU!

Reuploading to GitHub, rather than hammering Josiah's google drive: vscode-byubazel-1.1.5.vsix.zip

Install this with:

wget https://github.com/bazelbuild/vscode-bazel/files/8046189/vscode-byubazel-1.1.5.vsix.zip
unzip vscode-byubazel-1.1.5.vsix.zip
code --install-extension vscode-byubazel-1.1.5.vsix.zip
rm vscode-byubazel-1.1.5.vsix*
# And also install the normal Bazel extension to get, e.g., Buildifier info. Seems to be broken in the modded one
code --install-extension BazelBuild.vscode-bazel

Notes for others, after playing with it for a bit:

This is a great stopgap, but it'd still be really great to have an official, ongoing effort, perhaps out of Google. Hopefully we'll at least hear from Google!

Key features I'm seeing in Josiah's fork but not this one, that I've found most useful so far:

Key features that would be great to add:

You might also see an error like the following in the Bazel Language Server output and be alarmed, but it seems to recover fine--and still be mighty useful

[Fatal Error] log4j2.xml:21:6: The processing instruction target matching "[xX][mM][lL]" is not allowed.
...
    at server.BazelLanguageServer.<clinit>(BazelLanguageServer.java:17)
cameron-martin commented 7 months ago

It appears facebook have a starlark language server for buck2 (https://github.com/facebookexperimental/starlark-rust). Maybe this could be of use?

aherrmann commented 7 months ago

It appears facebook have a starlark language server for buck2 (https://github.com/facebookexperimental/starlark-rust). Maybe this could be of use?

Linking to https://github.com/facebookexperimental/starlark-rust/pull/51 for potentially relevant context.

cameron-martin commented 7 months ago

There appears to be an implementation of the language server for bazel in starlark-rust. See https://github.com/facebookexperimental/starlark-rust/blob/main/starlark_bin/bin/bazel.rs

cameron-martin commented 7 months ago

I created a simple vscode extension to register it as a language server for starlark files and it seems to work reasonably for go to definition in load statements & label autocomplete. There's a few obvious bugs, but it seems like a great start. Since it says it is only temporary and marked for deletion, maybe we could extract it into a bazel-owned repo and add experimental support for it in this extension.

daivinhtran commented 7 months ago

FYI https://github.com/facebookexperimental/starlark-rust/blob/main/vscode/README.md is the unpublished VSCode extension frontend that is already well integrated with the starlark binary.

cameron-martin commented 7 months ago

I saw that after creating my own extension, but it doesn't really do much except launch a language server: https://github.com/facebookexperimental/starlark-rust/blob/main/vscode/client/src/extension.ts

I think there's still use for integrating the LSP into a bazel-specific extension.

withered-magic commented 6 months ago

Came across this thread and I wanted to share an implementation of a Starlark LSP I've been working on to learn Rust! I decided to pick this up after realizing that the LSP included with starlark-rust didn't yet support things like type inference, auto-complete for fields/methods, etc.

It's still in a pretty rough WIP state but can handle things like rudimentary type inference/error diagnostics + auto-complete for method on builtin types within a single file. I do plan on spending a good amount of my free time working on this; my plan is to implement the core Starlark spec first, and then start supporting Bazel-specific features (e.g. auto-complete/type inference for Bazel builtins, auto-complete/navigation for labels, etc.)

cameron-martin commented 6 months ago

The Bazel LSP in the starlark-rust repo never intended to be supported long-term, and as a result are slow in accepting PRs for it.

Therefore I've forked this work and extended it here: https://github.com/cameron-martin/bazel-lsp

daivinhtran commented 5 months ago

@cameron-martin I think https://github.com/bazelbuild/vscode-bazel/pull/317 doesn't fully address this issue. It opens a client to work with an existing Starlark LSP which lacks support for Bazel files.

Per https://github.com/facebookexperimental/starlark-rust/issues/104#issuecomment-1900153760, should we keep this issue open until we have a path forward for a more official LSP for Bazel?

cameron-martin commented 5 months ago

Good point, I guess it makes sense to keep this issue open until there's a mature LSP available.

cpsauer commented 5 months ago

I will say, the BYU one (above) has served us very well! Maybe there's a way to merge it in?

withered-magic commented 5 months ago

On the topic of having a Bazel-first LSP, something that I've been getting to work in my implementation, and something that could potentially be managed through this extension, is using the ApiExporter, which dumps a protobuf file of all Bazel builtin functions/types with this schema.

Combined with some sort of mechanism for supplying type information (e.g. for function parameters), it turns out this works pretty well for enabling features like type-inference + autocomplete + documentation hovers for not just rules but also common objects like ctx, and lets us get much closer to a "typical" IDE experience for writing Bazel rules. For example, here's an example of the starpls LSP providing autocompletions on ctx while writing a simple rule implementation function, with type annotations specified via PEP 484-style type comments:

ShareX_lqOsFCb4Bx

One thing I've been thinking about is how to distribute these proto files (each of which are bout around 500KB in size) for different versions of Bazel. Maybe they could be bundled with the vscode-bazel .vsix and exposed to an LSP implementation via some standard environment variable? It would also be pretty cool for the extension to let users supply their own protos via some configuration option and get autocomplete for other projects that use Starlark, e.g. Tilt's Tiltfiles.

cameron-martin commented 5 months ago

I implemented basic support for using these builtins for autocomplete in bazel-lsp here. It doesn't supply type information yet, but it could. One issue is these protobufs don't contain all the builtins, such as the python, java, and cc ones. @withered-magic have you found out how to obtain these from bazel?

vogelsgesang commented 4 months ago

One thing I've been thinking about is how to distribute these proto files (each of which are bout around 500KB in size) for different versions of Bazel. Maybe they could be bundled with the vscode-bazel .vsix and exposed to an LSP implementation via some standard environment variable?

To my understanding, those files differ between Bazel versions. I think there are too many different Bazel versions (including all the rolling releases...), so packaging this protobuf for all Bazel versions wouldn't be feasible.

I think, the best solution would be to teach the extension to download the proto files for the current Bazel version on-demand. Are those files uploaded as part of the Bazel releases (both stable & rolling releases)?

vogelsgesang commented 4 months ago

One issue is these protobufs don't contain all the builtins, such as the python, java, and cc ones. @withered-magic have you found out how to obtain these from bazel?

I just stumbled across bazel info build-language. Maybe this is what you are looking for? It should return protobuf data according to this proto definition. From a quick, superficial inspection this seems to also include cc_library.

Also see https://github.com/bazelbuild/bazel/issues/15817

withered-magic commented 3 months ago

re: builtins, I ended up also obtaining builtin rules through bazel info build-language, which has mostly worked well, except for the fact that the resulting proto doesn't contain any documentation strings for any of the builtin rules or their attributes. In the main Bazel repo there's infrastructure for extracting this documentation from the source files of the rules themselves but I haven't gotten to investigate it too closely. Luckily the proto does contain other important things like type information, making it easy to do type checking, etc for rule invocations.

Unfortunately, I wasn't able to figure out how to source WORKSPACE and MODULE.bazel builtins from the ApiExporter, so I ended up having to manually copy their docs from the build encyclopedia. This definitely isn't ideal though and hopefully there's a way to fix the ApiExporter to include these as well.

There's also some issues with some of the builtins in builtin.proto that I haven't been able to figure out either (e.g. certain providers like PyInfo are exported with the wrong type, etc.) so I might end up manually copying the docs for those as well until I can investigate further. I also had to add additional behavior in starpls to override some of the return types specified in builtin.proto to support autocomplete/validation for rule and repository rule attributes.

To my understanding, those files differ between Bazel versions. I think there are too many different Bazel versions (including all the rolling releases...), so packaging this protobuf for all Bazel versions wouldn't be feasible. I think, the best solution would be to teach the extension to download the proto files for the current Bazel version on-demand. Are those files uploaded as part of the Bazel releases (both stable & rolling releases)?

Unfortunately I'm not aware of any place these protos are uploaded at all. To my knowledge we'd have to build them as part of some separate process for each Bazel version (probably stopping at some minimum version). If we end up finding a fix for the issues with the protos that I mentioned above, I think we'd also have to backport the fix to the older versions - otherwise the experience would be heavily regressed for any Bazel versions other than latest.

vogelsgesang commented 3 months ago

Unfortunately I'm not aware of any place these protos are uploaded at all. To my knowledge we'd have to build them as part of some separate process for each Bazel version (probably stopping at some minimum version)

Alternatively, we might consider adding bazel help builtins-as-proto and / or bazel help buildlang-as-proto to the main Bazel executable. This would allow the LSP to simply call the Bazel binary and ask it for the builtins and the supported build language.

I am having something similar to the bazel help flags-as-proto command in mind here. This command currently exports the command line flags in the bazel_flags.proto format. This is currently implemented in HelpCommand.java.

(I am currently relying on bazel help flags-as-proto for a language server for bazelrc files. I hope I will be able to open-source / upstream that language server in the future)

withered-magic commented 3 months ago

Alternatively, we might consider adding bazel help builtins-as-proto and / or bazel help buildlang-as-proto to the main Bazel executable. This would allow the LSP to simply call the Bazel binary and ask it for the builtins and the supported build language.

I think this would definitely be the ideal approach and would avoid the need for otherwise bundling protos in the extension! Although I think it might not solve the problem of supporting older Bazel versions that don't support the new command, so we'd still need to have the protos for older versions as a fallback (either that or we just communicate that only Bazel versions with the bazel help builtins-as-proto command are supported).

fmeum commented 3 months ago

@withered-magic Happy to help with the Bazel side of this by adding a proto export feature and making sure MODULE.bazel and built-in info is extracted correctly. I'll read up on your approach and open a Bazel issue to collect what needs to be done.

withered-magic commented 3 months ago

Thank you so much! Let me know if you'd like me to do an audit on the current state of the exported proto, I had an in-progress list of the items that I noticed were either missing or exported incorrectly, so I can definitely get that for you to supplement the Bazel issue with.

fmeum commented 3 months ago

That would be great if you have it handy. I can expand on the Bazel-specific details.

withered-magic commented 3 months ago

For sure, I'll go ahead and put it together and update back in this thread!

withered-magic commented 3 months ago

@fmeum Here's a link to a list of issues that I've found with the proto as it's currently exported! https://github.com/withered-magic/starpls/blob/builtin-audit/docs/builtin-audit.md

Disclaimer that this might not be fully exhaustive, but it does contain all the issues about the proto that I was able to notice!

vogelsgesang commented 3 months ago

Although I think it might not solve the problem of supporting older Bazel versions that don't support the new command, so we'd still need to have the protos for older versions as a fallback (either that or we just communicate that only Bazel versions with the bazel help builtins-as-proto command are supported).

I wouldn't be too concerned about older versions of Bazel for now. Let's first build the right long-term solution (i.e. fix bazel help builtins-as-proto or whatever the right solution might turn out to be) and only after we figured out the long-term solution, consider what we should do with the older Bazel versions... If we are lucky, the older versions will already be out-of-maintenance by the time we have the language servers working smoothly on the latest Bazel version 😄

However, if we ever want to act on #263, we might still want to host those builtins dumps somewhere: I think the best way to also get the language servers running on the web would be by compiling them to Web Assembly. This would mean the language server itself would be able to run, but trying to compile Bazel itself to Web Assembly seems like a lost cause to me. Therefore, the language servers would need to get the ´builtins´ from somewhere else - likely downloading them on-demand would be the best solution... But anyhow - this is still pretty far out in the future. Let's focus on ironing out the language server on native machines first, before making it web-compatible 🙂

vogelsgesang commented 1 month ago

In case you are still interested in advancing the journey towards a first-class, LSP-based IDE experience for Bazel, please consider upvoting this comment as well as this proposal.

I am not sure why there was little visible progress over the last 2 months, but maybe it would help make progress, if the Bazel project becomes more aware of the community demand for auto-completion inside VSCode / other LSP-based editors.

Also, in case you are interested in contributing yourself towards the stardoc vision proposed as an alternative design (which I agree is better), you might want to comment in that thread stating that you would be willing to contribute code towards that vision 🙂

warpfork commented 1 month ago

Hi to everyone here and who (like me) might be newly finding this issue via the search engines!

Just want to mention that there's at least one person out where who's interested in a Starlark LSP in general, not just a Bazel one. (Me :))

If any of the tooling the community comes up with can support arbitrary starlark -- meaning, both without bake-in of bazel's many well-known symbols, and, with an ability to supply some info about other values in the global scope in a file that are introduced out of band by the interpreter -- that'd be amazing.

Cheers all!

cameron-martin commented 1 month ago

@warpfork this is pretty much what the language server in starlark-rust is.

vogelsgesang commented 1 month ago

Slightly off-topic: Language server for bazelrc files

The other day, I finally found the time to open-source a language server for .bazelrc config files which I wrote a couple months ago. In the screen recording, you can find a short demo.

https://github.com/bazelbuild/vscode-bazel/assets/6820896/b677bc26-8475-446c-bc94-08aadbd20cb9

Features:

You can find the source code under https://github.com/salesforce-misc/bazelrc-lsp and prebuilt binaries under https://github.com/salesforce-misc/bazelrc-lsp/releases

(Please let's move all follow-up discussion on bazelrc configuration files into separate Github issues. This issue should stay focused on the Starlark language server. I already feel bad enough for hijacking this thread)