lsp: Language Server Protocol Support

cueckoo commented 3 years ago

Originally opened by @rudolph9 in https://github.com/cuelang/cue/issues/142

Here is an example of a YAML file LSP https://github.com/redhat-developer/yaml-language-server

I've never created one so I can't say for certian how much work it would be but it would be great for integrations in to vim, emacs, VSCode, etc.

https://langserver.org/

cueckoo commented 3 years ago

Original reply by @rudolph9 in https://github.com/cuelang/cue/issues/142#issuecomment-557334636

Somewhat related, looks like someone made a vim plugin https://github.com/jjo/vim-cue

cueckoo commented 3 years ago

Original reply by @jlongtine in https://github.com/cuelang/cue/issues/142#issuecomment-557601291

Some other links as well:

I was chatting with @mpvl about this a bit at KubeCon this week, and I think this could be highly valuable for CUE.

cueckoo commented 3 years ago

Original reply by @mpvl in https://github.com/cuelang/cue/issues/142#issuecomment-560167844

To get a better idea of what this should look like: what kind of functionality and analysis would people want to see supported by a LSP implementation?

cueckoo commented 3 years ago

Original reply by @mxey in https://github.com/cuelang/cue/issues/142#issuecomment-560284507

To get a better idea of what this should look like: what kind of functionality and analysis would people want to see supported by a LSP implementation?

Syntax checking/validation (basically, show errors without having to run cue eval first)
Symbols for completion, documentation and jumping to definition

basically, I'd like the same editing experience for Kubernetes objects in CUE as you can get for YAML in VS Code

cueckoo commented 3 years ago

Original reply by @mpvl in https://github.com/cuelang/cue/issues/142#issuecomment-560804481

To get a better idea of what this should look like: what kind of functionality and analysis would people want to see supported by a LSP implementation?

find all locations contributing to a current position
expand implied fields into current struct, to allow editing them as if it were a form
grey out or otherwise mark fields that could be removed with trim (highly important fields)
remove unchanged implied fields (implied fields could be marked with @implied()
trim at cursor
hoist/relax (anti-unify) at cursor

cueckoo commented 3 years ago

Original reply by @rudolph9 in https://github.com/cuelang/cue/issues/142#issuecomment-562263864

Long term goal, I'd like to build something like Barlin.

This gets into some very meta concepts underpinning cue but theoretically a cue evaluator could be written in cue similar to the way Barlin utilizes a scheme interpreter implemented in MiniKanren which implemented scheme :thinking:

cueckoo commented 3 years ago

Original reply by @zchee in https://github.com/cuelang/cue/issues/142#issuecomment-569316852

I created packages that lsp for Go. (for my project, Neovim LSP plugin written in Go :D) It almost works IIRC. I'm using zchee/nvim-lsp (private).

https://github.com/go-language-server/protocol

If we develop lsp for CUE written in Go, it might be helpful that package.

cueckoo commented 3 years ago

Original reply by @myitcv in https://github.com/cuelang/cue/issues/142#issuecomment-582938287

We can likely reuse some of the work in golang.org/x/tools/internal/lsp/... to get us started.

Probably worth starting with simple diagnostics. That will allow us to flesh out a basic skeleton for what the LSP server should look like (including how to handle modules, a la Go modules).

AFAIK, syntax definitions fall outside the spec of LSP. I plan to implement a Vim plugin in Go using govim for AST driven syntax highlighting to try out the parser/AST.

cueckoo commented 3 years ago

Original reply by @metalmatze in https://github.com/cuelang/cue/issues/142#issuecomment-587090610

I've talked to @mpvl at FOSDEM and it might be worth checking out the Prometheus Language Server @slrtbtfs created as part of his internship in our team. It reuses parts of gopls but in a more general purpose way. @slrtbtfs should have more details about why things needed to be changed, in case you have questions. That should already get this going a bit quicker :)

cueckoo commented 3 years ago

Original reply by @slrtbtfs in https://github.com/cuelang/cue/issues/142#issuecomment-587434293

Hi,

I've had a quick look at cue, and it should probably be possible to reuse parts of the PromQL language server for the cue language server.

What is definitely reusable and turned out be very useful is the golang.org/x/tools/internal/lsp/proctocol package, which contains (almost) all the types defined in the LSP specification.

What could be useful for you, too, is forking the github.com/prometheus-community/promql-langserver/langserver/cache package, removing all the PromQL stuff there and replacing the yaml parser with the cue parser. This package basically takes care of all the statekeeping in the language server.

Then the only thing left to do is implementing the protocol.Server interface. You can start with leaving everything unimplemented and then add more and more language features over time.

Feel free to reach out if you have any questions.

cueckoo commented 3 years ago

Original reply by @mikelnrd in https://github.com/cuelang/cue/issues/142#issuecomment-596586362

Another more recent development related to language servers is The Language Server Index Format (LSIF).

Basically the idea is to spit out the analysis to a JSON file. Sourcegraph (a code search tool) lets you upload the LSIF file and gives you language-server-like code navigation (but without the language server even running!).

Perhaps cue could spit out the LSIF JSON data?

I've just grep'ed for my notes on the topic and thought I'd share them here in case they are helpful. See below.

Loving cue at the moment and the vscode-cue plugin. Can't wait for a language server!

.............................................................. https://code.visualstudio.com/blogs/2019/02/19/lsif

https://github.com/microsoft/lsif-node/blob/master/README.md https://github.com/Microsoft/language-server-protocol/blob/master/indexFormat/specification.md

vscode extension (to serve LSP from the file) https://github.com/microsoft/lsif-node/blob/master/README.md#lsif-extension

https://about.sourcegraph.com/blog/sourcegraph-3.9 https://about.sourcegraph.com/blog/writing-an-lsif-indexer https://about.sourcegraph.com/blog/sourcegraph-3.8 https://about.sourcegraph.com/blog/code-intelligence-with-lsif

https://docs.sourcegraph.com/user/code_intelligence/lsif

go get -u github.com/sourcegraph/lsif-go/cmd/lsif-go

https://github.com/sourcegraph/lsif-go https://www.youtube.com/watch?v=fMIRKRj_A88 https://youtu.be/fMIRKRj_A88?t=218

https://github.com/microsoft/lsif-node/tree/master/tsc

lsif-protocol: Protocol defined as TypeScript interfaces lsif-util: Utility tools for LSIF development lsif-tsc: LSIF indexer for TypeScript lsif-npm: Linker for NPM monikers

cueckoo commented 3 years ago

Original reply by @galli-leo in https://github.com/cuelang/cue/issues/142#issuecomment-624938305

Hi,

I semi-hacked together a working implementation (using promql as a base) over the past two days. You can see it running in action in vscode below :). Currently, I have the following working:

Diagnostics: parse errors, compile errors and even output of cue vet
Definition: navigate to definition (including multiple, see GIF)
Hovering: showing definition on hover + comments, works decentish, but could use some work
Completion: Extremely basic at the moment, would need a lot more work here

(There might be one or two crashes though 😅 )

Should I fork this project and add what I have up until know, then open a PR? Or should I just commit it to a personal repo, that's separate?

Some thoughts I had during my implementation, most of these are probably my fault for not being familiar with the codebase though 😛 :

~It is very annoying that compilation fails immediately, if there is any error (or maybe I haven't found the correct options yet?). At the moment I need to keep the last Instance of a document around, otherwise I only have the AST.~ Seems to be possible by using build.Instance instead. However, if a top level identifier is unknown, inst.Value() will still return nothing, so would need to access some internal stuff.
Navigating the AST and Instance is very annoying. I wrote my own Cursor for the AST, but nothing yet for Instance.
Matching between something in the AST and a corresponding Value is even more annoying, that code is very messy and should probably be reworked. Currently, I iterate through all Values in the Instance and check if val.Source() == node (node is an ast.Node).
There doesn't seem to be a way to get a Value corresponding to a field definition, which made some stuff a lot harder.
Mapping between source locations is also not trivial and seems to be the cause of one of the crashes.
There doesn't seem to be a distinction between an ast.Ident when used e.g. a: b (i.e. one is field definition label, the other field definition type?). Ideally, there was a way to know which one is which, since certain stuff (like auto complete or hover), have different info based on that (at least the way I implemented it at the moment).

Lastly, I wanted to say, that I really love this project! I think it's really cool and we plan on using it in our new infrastructure for all config management (hence why I decided to spend some time implementing this :)).

vscode

cueckoo commented 3 years ago

Original reply by @verdverm in https://github.com/cuelang/cue/issues/142#issuecomment-625010002

@galli-leo this is awesome!

Have you signed the Google CLA? If not, we can get you a link to the instructions

Cue accepts PRs from here or Gerrit, with some preference for Gerrit as it is the actual source of truth for the code. Gerrit uses a slightly different commit methodology which is pretty cool to experience if you are up for it. Depending on which method you prefer, we'll follow up with more details.

Are you aware the syntax is changing slightly? In particular, I see your example (really cool by the way) has the old style definitions with <ident> :: _. The newer syntax is #<ident>: _ and list comprehension is a little different too. I'm sure there are some others I am missing.

There are some other changes that may impact the lang-server implementation, and hopefully they will only make it easier. As a general FYI, Cue is going through some bigger changes right now on its way to stability later this year.

Have you joined the slack group? (link to join is here: https://cuelang.org/community)

cueckoo commented 3 years ago

Original reply by @galli-leo in https://github.com/cuelang/cue/issues/142#issuecomment-625274193

@verdverm

Have you signed the Google CLA? If not, we can get you a link to the instructions

Yep signed it already.

Cue accepts PRs from here or Gerrit, with some preference for Gerrit as it is the actual source of truth for the code. Gerrit uses a slightly different commit methodology which is pretty cool to experience if you are up for it. Depending on which method you prefer, we'll follow up with more details.

I prefer Github, since I don't have experience with gerrit. But I am also fine with using gerrit, if that makes it easier for you :)

Are you aware the syntax is changing slightly? In particular, I see your example (really cool by the way) has the old style definitions with :: . The newer syntax is #: and list comprehension is a little different too. I'm sure there are some others I am missing.

I am now ;) To be honest, I delved into this without having familiarized myself 100%!((MISSING)probably not even 80%!:(MISSING)P) with the language spec (in hindsight maybe not the best idea).

There are some other changes that may impact the lang-server implementation, and hopefully they will only make it easier. As a general FYI, Cue is going through some bigger changes right now on its way to stability later this year.

I saw the issues regarding that. Is there a central document detailing those changes?

Have you joined the slack group?

Yep! Will ask further questions there.

cueckoo commented 3 years ago

Original reply by @buremba in https://github.com/cuelang/cue/issues/142#issuecomment-725729167

@galli-leo looks great! Any plans to open-source the LSP?

cueckoo commented 3 years ago

Original reply by @myitcv in https://github.com/cuelang/cue/issues/142#issuecomment-725860986

LSP is close to the top of the list to be worked on in the near future... so this will get some attention soon.

cueckoo commented 3 years ago

Original reply by @alex-de-oliveira in https://github.com/cuelang/cue/issues/142#issuecomment-776642677

This would be absolutely invaluable

cueckoo commented 3 years ago

Original reply by @uhthomas in https://github.com/cuelang/cue/issues/142#issuecomment-788929894

I haven't seen any mention of Sublime Text 4 yet, so I'll share some quick thoughts.

Sublime Text 4 has a generic LSP implementation which works pretty well. I've been using it in conjunction with gopls for about a year now and both have improved significantly in that time.

It's an exciting prospect to have a similar LSP for CUE.

cueckoo commented 3 years ago

Original reply by @eadlam in https://github.com/cuelang/cue/issues/142#issuecomment-804401602

@myitcv I'm interested in helping with the LSP

cueckoo commented 3 years ago

Original reply by @myitcv in https://github.com/cuelang/cue/issues/142#issuecomment-804761896

I am now starting work on a first cut of cuepls. This work will complement the proposal for CUE modules which will be published this week.

This first cut of cuepls should hopefully support the work @jansorg and others have done in #250 on IntelliJ support.

Once we have the right "structure" in place (cuepls will live in the main CUE repo), I will message back here and create issues for aspects/features of cuepls that remain to be implemented. That way, people who would like to contribute can coordinate via an issue so that we don't have a race condition on people working on the same feature. We can also flesh out design issues, discuss whether changes to core CUE packages are required etc.

I will now tag all LSP-related issues with cuepls; adding help wanted and good first issue where appropriate.

That said, any contribution will of course be greatly appreciated! Because when we have an initial implementation of cuepls, actually using it and reporting bugs will be a massive help in and of itself.

cueckoo commented 3 years ago

Original reply by @samschlegel in https://github.com/cuelang/cue/issues/142#issuecomment-839168440

Also interested with helping with this as this is the main thing blocking our adoption of CUE, and I really want to adopt it 😄

cueckoo commented 3 years ago

Original reply by @eadlam in https://github.com/cuelang/cue/issues/142#issuecomment-840030836

This is also the main thing blocking CUE adoption at my org.

cueckoo commented 3 years ago

Original reply by @leoluk in https://github.com/cuelang/cue/issues/142#issuecomment-840050326

Note that while there's no LSP yet, there are a number of language/syntax plugins for various editors:

...and probably more. That might be good enough to unblock adoption while waiting for full autocompletion :)

The IntellIj one in particular is commercially supported by us and has a full parser/lexer and will highlight syntax errors.

cueckoo commented 3 years ago

Original reply by @myitcv in https://github.com/cuelang/cue/issues/142#issuecomment-840052052

Thanks for everyone's patience here. This is in progress; we very much understand the importance of good editor support for the success of CUE.

cueckoo commented 3 years ago

Original reply by @samschlegel in https://github.com/cuelang/cue/issues/142#issuecomment-840223275

Let me know if there's any way I could help!

Note that while there's no LSP yet, there are a number of language/syntax plugins for various editors:

Yeah I've been using those for some things I've been trying out, but in order to kick off replacing all of our Jsonnet we're going to need things like go to definition, type information, and autocomplete for both VSCode and IntelliJ

samalba commented 3 years ago

@myitcv Do you know if this is actively being worked on? @TomChv is looking to start a university project on LSP for Cue, with several students ready to code. But to avoid work duplication, this is on hold.

myitcv commented 3 years ago

@samalba as discussed, whilst my time on this largely got derailed by the migration in #1078, the gopls team have been working to refactor parts of that code base so that it can be more easily reused as an "LSP in a box". The plan is now to create a CUE instance using the core building blocks, on top of which we can start to flesh out CUE-based implementations of the LSP methods. I suspect that some of those methods will be discrete pieces of work where we would very much welcome community contributions (albeit coordinated via an issue first).

unicell commented 2 years ago

@myitcv Could you please provide some pointers to "LSP in a box" refactoring work mentioned above? Would love to learn some context here.

iFrozenPhoenix commented 2 years ago

@myitcv is there any progress comcerning this issue?

I know the golang implementation of the language server and from my point of view it is overcomplicated for this project (i have used the go language server for a language i've created). Only the very basic core and the protocol bindings seems to be useful.

The document symbols and the completion items can be relatively easy parsed with an ast implementation.

eadlam commented 2 years ago

We’re approaching the one year anniversary of this issue!

guettli commented 1 year ago

Sorry, I don't have time for developing a language-server-protocol.

But I could support by donating some money to a crowd-funding compaign. Related: https://github.com/cue-lang/cue/discussions/2358

cue-lang / cue

lsp: Language Server Protocol Support #142