Configuring autocomplete using gopls and jupyter lab

[marcoferrer]

Hi there. I may have just missed it, but Im trying to figure out how one would setup autocomplete using gopls and this kernel?

[janpfeifer]

It's already (partly) done. Try adding a cell like:

import "fmt"

%%
fmt.Println("Hello")

Execute it once -- so it registers the import "fmt". After that, in a line you can start with fmt.P and press tab and it will offer auto-complete. It also uses gopls for the contextual help (control+I).

Unfortunately, as before, it often only works after executing once successfully the cell with the import.

So the long story is that it should be improved, but I'm not sure exactly how gopls work, it probably could be improved.

If you are interested, let me know, the code in goexec/goplsclient/goplsclient.go.

cheers

[joelschutz]

This is a big dilema in my opinion and need a bit of work to get in a stable state. See, there are 2 ways to add completions to notebooks in the main editors(JupyterLab, VSCode):

• Via kernel(like gonb does)
• Via lsp(Language Server Prorocol)

The gopls is a very complete lsp and is capable of much more than just completion, but it needs to be synced to the workspace to work. Every change to a file must be notified to the server so it can keep track and do it's job, that's why you need to execute the cell once, as I undestand, a ghost file is updated with the cell content so the server can read and analyze it.

I don't really like this aproach, but until gopls is able to deal with notebooks directly that's the way to do it. It's a shame, because now lsp support notebook events and can be used for great effect with notebooks(check the python lsp on VSCode notebooks).

[janpfeifer]

Thanks for the explanation @joelschutz . I navigated a bit blind here -- documentation is sparse. And it is definitely not working well yet, and if I figure out how gopls works, I would definitely try to improve it.

GoNB works by regenerating a main.go file on-the-fly, everytime the cell is executed. It generates the main.go with all the definitions (functions, values, types, imports, consts) it memorized from previous executions. It then does goimports and re-parse the file for new imports (that are also included in its memorized definitions) followed by go get to automatically fetch the new imported dependencies. Fortunately this all happens so quick (at least when there is no new dependency to fetch) to feel interactive.

Now when auto-complete or contextual help is requested by Jupyter, GoNB will simply regenerate the main.go file (and map the cursor position accordingly). It always send an update notification (NotifyOpenOrChange) for the regenerated main.go and go.mod (just in case) files:

https://github.com/janpfeifer/gonb/blob/main/goexec/goplsclient/goplsclient.go#L111

The issues I observe are that:

• GoNB doesn't run goimports or go get on the new main.go before calling gopls: so if the user is typing fmt.P... (and hasn't explicitly imported fmt) it won't auto-complete. I wonder if I should change this behavior -- it will likely freeze a bit, specially if a package needs fetching...
• For packages one is editing locally on an editor on the side, GoNB doesn't know which files it needs to upload to gopls -- I work this way a lot, where I'm testing something using GoNB, but I'm actually writing a library code on an IDE on the side.
• gopls is also not being able to complete many non-go standard packages, even after the go get is executed. Maybe GoNB is doing something wrong, I'm not sure ... I went so far as to starting adding printf in the middle of gopls, but didn't manage to figure it out. GoNB is not sending a lsp.MethodTextDocumentDidOpen message to gopls for files on imported packages (probably located in some standard Go cache location).
• Also gopls sends lots of warnings/errors, and I haven't figure out yet what to make of them. If it doesn't return anything GoNB shows those collected warnings/errors instead of the contextual help (control+I).

If by any chances you understand how gopls work, pls let me know, I would love to improve this. Changing GoNB code is relatively easy.

cheers

[joelschutz]

I'm not expert in the topic either, but I'm happy to help. For what I could figure from the gopls code, it keeps a virtual version of of your workspace in memory so it can have context of what is going on the project.

• GoNB doesn't run goimports or go get on the new main.go before calling gopls: so if the user is typing fmt.P... (and hasn't explicitly imported fmt) it won't auto-complete. I wonder if I should change this behavior -- it will likely freeze a bit, specially if a package needs fetching...

I think that's a design decision, the way it works now(where you need to run the import so gopls knows about it) is how python usually works too for kernel based completion. If you wish to change it the workflow will change too, but I prefer that it runs this tools because that's how the IDE does it.

• For packages one is editing locally on an editor on the side, GoNB doesn't know which files it needs to upload to gopls -- I work this way a lot, where I'm testing something using GoNB, but I'm actually writing a library code on an IDE on the side.
• gopls is also not being able to complete many non-go standard packages, even after the go get is executed. Maybe GoNB is doing something wrong, I'm not sure ... I went so far as to starting adding printf in the middle of gopls, but didn't manage to figure it out. GoNB is not sending a lsp.MethodTextDocumentDidOpen message to gopls for files on imported packages (probably located in some standard Go cache location).

I'm not sure which event to send, but it seens like a problem on the sync between golsp and the new files. For the problem with the packages from go get maybe it's just a configuration option, because in VSCode it seens like it refresh the imports in background. For the local package, the gopls instance on the kernel may need to be notified of those changes too.

One tip that can help you find those events is to look for the traces in VSCode to findout how it interacts with gopls. First add those options to the settings.json:

...
"gopls": {
    "verboseOutput": true,
    },
    "go.trace.server": "verbose",
...

Then you can look at the traces on the Output, just paste workbench.action.showOutputChannels on the command pallete and it will show a list of all the services, find gopls and start digging. With those configurations it will show everything that is being sent and received by VSCode to the server.

I hope could help more, but that's what I got.

cheers

[janpfeifer]

Btw, do you know where the VSCode that talks to gopls is located ? (sorry I don't use VSCode usually, I've been using Goland)

[joelschutz]

I'm not sure what you meant, but to setup VSCode you can follow this guide: https://code.visualstudio.com/docs/languages/go

VSCode talks to the Language Servers directly, they invented the protocol after all, but this communication is configured by the extension. You can find the source for the official extenstion here: https://github.com/golang/vscode-go

If you are talking about where the code that communicates with gopls is located, probably in the official repo(https://github.com/microsoft/vscode). Be aware that it's written in TS and I don't recommend to go that route. LSP is a open standard and gopls follows it very well, no special hacks for VSCode.

Just to keep track, those are the documents that I read and recommend about LSP:

• https://cgi.cse.unsw.edu.au/~eptcs/paper.cgi?FIDE2021.3.pdf
• https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/

[janpfeifer]

Thanks for the links @joelschutz !

I've developed the connection using the specifications in MS site, plus lots of trial and error.

Now, before going through the VSCode extension for Go, I revisited the GoNB code, and added the calls to goimport and go get before calling gopls. More importantly, I also added go.sum to the list of files that I notify of change to gopls. With that gopls started fetching the information from the imported packages.

It's much better now.

But not perfect yet: for instance, auto-complete, when typing something mypkg. and trying to auto-complete, it fails because it cannot parse something ending in . ... I'll leave that for later though, since without parsing the contents of the cell, it's quite hard to regenerate the main.go (since I have to include all other memorized declarations from the other cells).

PR and new 0.6.1 release coming soon.

[janpfeifer]

The new release v0.6.3 improves the code-completion in many more cases. Still not perfect though -- local libraries, redirected in go.mod don't seem to be considered for auto-complete.

[AnotherCodeArtist]

Hi Jan!

Brilliant! First tests indicate that this is exactly what I needed.

Thanx again!

[janpfeifer]

One more release with improvements to gopls interface: this time to the "Context Help", a.k.a. InspectRequest (control+I in the colab). It also handles cases where cell is not yet parseable (just like the previous one).

I also fixed a bug I introduced in the last version, where go.mod / go.work was not being updated in go.pls.

Next I want to provide a way to inform GoNB (which in turn will inform gopls) of files being edited locally in a separated editor.

If that works well, that will make the 0.7.0 release.

Let me know if any of you see any other cases not working.

cheers

[janpfeifer]

v0.7.0 released. It allows tracking arbitrary files/directories, and automatically picks up redirect entries in go.mod (created with !*go mod edit -replace github.com/my/project=/home/myuser/my/project, and track those local files in disk.

That means that file changes done in a separate editor, in parallel to using Jupyter Notebook, will automatically be available in gopls

v0.7.0 also adds some definitions mangement special commands (%ls and %rm). But that is a separate topic.

I'll close the issue for now, but let me know if you find any more issues.