Implement snippets support for completions

[krassowski]

What are you trying to do?

Support LSP snippets to restore functionality of R languageserver. R languageserver moved some function completions to use snippets: https://github.com/REditorSupport/languageserver/pull/168. It is not currently supported and adds an annoying '$0' instead:

How is it done today, and what are the limits of current practice?

TBD

How long will it take?

ETA 1 week, need this every day.

[bollwyvl]

So snippets sound really cool (too bad it's forced on us, in this case).

The API for inline/line widgets in CM is pretty good (not quite as rich as prosemirror, to be sure).

I'd imagine either having:

• fully inline widgets, a la http://bl.ocks.org/jasongrout/5378313
  • they fully exist in the source
• line widgets that you fill out
  • they don't actually change the source, until you fill everything out and "commit"
  • used these a little on lintotype...

[krassowski]

Both are good options. Yes actually we should not be receiving the snippets as we do not declare snippetSupport

[krassowski]

Not sure if this should live in LSP, or should be directly contributed to the JupyterLab along with the completion API update.

[krassowski]

For convenience, the specs say:

A snippet can define tab stops and placeholders with $1, $2 and ${3:foo}. $0 defines the final tab stop, it defaults to the end of the snippet. Placeholders with equal identifiers are linked, that is typing in one will update others too.

and then:

Snippet Syntax

The body of a snippet can use special constructs to control cursors and the text being inserted. The following are supported features and their syntaxes:

Tab stops

With tab stops, you can make the editor cursor move inside a snippet. Use $1, $2 to specify cursor locations. The number is the order in which tab stops will be visited, whereas $0 denotes the final cursor position. Multiple tab stops are linked and updated in sync.

Placeholders

Placeholders are tab stops with values, like ${1:foo}. The placeholder text will be inserted and selected such that it can be easily changed. Placeholders can be nested, like ${1:another ${2:placeholder}}.

Choice

Placeholders can have choices as values. The syntax is a comma separated enumeration of values, enclosed with the pipe-character, for example ${1|one,two,three|}. When the snippet is inserted and the placeholder selected, choices will prompt the user to pick one of the values.

Variables

With $name or ${name:default} you can insert the value of a variable. When a variable isn’t set, its default or the empty string is inserted. When a variable is unknown (that is, its name isn’t defined) the name of the variable is inserted and it is transformed into a placeholder.

https://microsoft.github.io/language-server-protocol/specifications/specification-current/

[bollwyvl]

Is there a last-known-good version we'd want to pin on e.g. https://github.com/conda-forge/staged-recipes/pull/11280#discussion_r407234227

[krassowski]

v0.3.5 fixes this by respecting client capability, but I'm not sure if it was released.

[bollwyvl]

For the purposes of that staged-recipe: looks like most of the builds are up on conda-forge: https://anaconda.org/conda-forge/r-languageserver/files?version=0.3.5

So i think we're good to go asking for that pin, (with a link back here).

[bollwyvl]

@kpinnipa @jahn96 have expressed interest in working on this.

As this would change the behavior of the r-languageserver completion, we may well want to think about it on the 4.0 release train, rather than as a minor 3.x release.

As this issue is a bit old at this point, and I haven't spent much time thinking about it, perhaps @krassowski can fill out some more design ideas about what will be needed in the current architecture. My fear is that we may need #500 for this to all work properly... but we're going to need that eventually, anyway, for e.g. #184.

What i can offer is some old stuff pertaining to lineWidgets... one of the outstanding design decisions is whether we want inline widgets or separate input rows for filling in tabstop values.

If i get a chance to work on LSP stuff in the near term, it will likely be to put out the fire on #575, which is unlikely to affect this work much...

[jahn96]

Currently, in our code snippet extension, we have implemented a user friendly way of creating and inserting snippets. There is a gif in the README that shows the current status of the extension https://github.com/jupytercalpoly/jupyterlab-code-snippets. Since we think inserting snippets by dragging the snippet or clicking insert button is a little cumbersome, we want to simplify the insertion process by using autocompletion. Essentially, we hope that the user would be able to type the first few words from a snippet and then press ctrl to see user defined snippets that they can select and press enter to insert.

We are thinking of adding autocompletion for snippets into the lsp extension and having users of the code snippet extension install the lsp extension in order to autocomplete their snippets. Currently, a code snippet is defined as below: ICodeSnippet { name: string; description: string; language: string; code: string[]; id: number; tags?: string[]; }. We are planning to add more necessary fields as we go.

To start off, we are wondering what would be the good starting point to understand how we might start on integrating snippet autocompletion into lsp?

Thanks!

[krassowski]

Sounds like a great idea! We can do this by exposing a new CompletionManager token which would allow you to register your snippets as completion items. I already started refactoring the completion system to enable merging completions from arbitrary number of sources (e.g. kernel, LSP, kite) in #549 but we still need further refactoring to enable registration of new sources. In principle the CompletionManager would have register() method taking an object with:

• a callback to a function returning completions for a given user query (returning CompletionHandler.ICompletionItemsReply with a list CompletionHandler.ICompletionItem or our IExtendedCompletionItem)
• the metadata about the completion source (ICompletionsSource)

Ultimately I will be aiming to upstream the envisioned CompletionManager to JupyterLab core for 4.0 (it is on the roadmap and tracked upstream) but experimenting with it here in an extension would be very beneficial to figure out a good API first.

Figuring out the templating variables might need to wait as I do not have any bandwidth for that currently.

[jahn96]

Sounds good! Will take a look at the PR and start working on it.

[jahn96]

Hi @krassowski, @kpinnipa and I have started to look into refactoring to enable registration of new sources.

1. What should register() method do? Should it return the handler like how it's currently written in core(like this)?
2. Where is the handler's connector used in this lsp extension?
3. How can the metadata about the completion source (ICompletionsSource) be used in register()?
4. What is the user query? Is it the object that has ICompletable and renderer?

Thank you so much for your help!

[krassowski]

I will try to sketch this in more detail tomorrow evening. As for 1) probably not as the handler pattern causes a lot of headaches.

[krassowski]

Sorry for the delay. Working on it now. I will PR a draft today.

[krassowski]

Hi @jahn96 I started the work on #600. It seems to be moving well, I found solutions for most of the things that I was afraid would be difficult to disentangle (+ to do so in a way that does not break the current user settings). This is still an early WIP (and possibly broken), but I hope it gives you an idea on what would be needed on your side (this is implementing a custom ICompletionProvider and registering it with ICompletionProviderManager). Please let me know if it makes sense and if anything more would be needed.

[jahn96]

I think that makes sense. Thanks!

[jahn96]

To make sure if we have understood correctly, we are thinking of creating a plugin for this. Let me know if it is what you were thinking and if it makes sense.

[krassowski]

I will add documentation but in short making use of this mechanism will be:

1. Add @krassowski/completion-manager to dev-dependencies in package.json
2. Add ICompletionProviderManager as optional token in activate function of your extension
3. Create a class SnippetCompletionProvider which implements ICompletionProvider interface (see here for a draft - subject to change) with two required bits:
   • identifier (e.g. "snippet_completion_provider"), and
   • async fetch(request, context) method which returns the completions given position in editor
   • isApplicable() - I guess in your case it always returns true; I will probably make it optional
4. Create an instance of that class and register it one the instance of the completion provider manager token.

For (3) there will be examples here (subject to change):

• Kernel provider
• LSP completion provider

I will include a full example with code for each step in the docs... Or maybe create a minimal standalone example package that shows how to implement add a completion provider which always suggests the same thing.

[jahn96]

Awesome! This is great! Thank you so much!

[krassowski]

Because we now use provider API for completions, the snippets need to be implemented in JupyterLab directly first. The relevant issue tracking this is:

• https://github.com/jupyterlab/jupyterlab/issues/15026