A quarto Language Server?

[juliantao]

Is it a good idea to implement the Language Server Protocol for Quarto? I know that Quarto is not really a language, but a combination of YAML, markdown/markup and various specific languages (python, R, Julia etc). However, if a language server is available, one can benefit from features such as auto complete, diagnostics, hover, go to definition, etc, in any editor which supports LSP.

Even better, for code chunks, a user could opt to download corresponding LSP for the language in the chunks, to achieve all the smart features.

What is most attractive is that one can use the same LSP in any editor, be it rstuido, vscode, vim or other popular editors.

[jjallaire]

100% agreed! In fact after we land the YAML validation branch (https://github.com/quarto-dev/quarto-cli/tree/feature/yaml-validation) @cscheid is going to be working on a Quarto LSP. Will update this issue when there is something for you to kick the tires on.

[jmbuhr]

As a fellow neovim user I am looking forward to this as well. I am currently working on a publication using quarto and it has been so smooth! Just an amazing editing experience with neovim and quarto's live preview. I also came across your snippets @juliantao in your configuration (https://github.com/juliantao/biglvim/blob/main/snips/snippets/global.json) and thought I'd mention it here because it might have more inspiration for the quarto VSCode extension (https://github.com/quarto-dev/quarto-vscode), since luasnip and VSCode support the same snippet syntax. Oh, and your website built with quarto looks amazing (https://juliantao.github.io/big/)! Maybe this is something for the quarto examples gallery @jjallaire?

[juliantao]

Thank you, @jmbuhr ! I prepared two manuscripts and created many websites using Quarto and it was a great experience! I believe writing papers will be much easier and smoother when the rticle equivalent is ready for Quarto. I also use Quarto to prepare all my slides now.

In addition to the snippets, I found the following plugins particularly helpful when editing qmd files using neovim:

1. ekickx/clipboard-image.nvim. This plugin allows you to paste an image to the markdown file. In a nutshell, it saves the clipboard image to a defined image directory with customized file name and then pastes the corresponding markdown snippet to the qmd file. I found this especially useful in preparing slides when a lot of figures are from the internet or from cropping a pdf file.

2. dhruvasagar/vim-table-mode. Editing and formatting tables is much easier with this plugin!

3. dkarter/bullets.vim, which makes life easier when editing ordered or unordered lists.

4. Zoterocite. This is not a complete vim plugin, but a vimscript snippet to cite zotero items in vim with the help of the great better-bibtex plugin for Zotero.

5. vim-pandoc/vim-pandoc-syntax for syntax highlighting and jalvesaq/Nvim-R for competing and formatting the R code chuncks.

As for the website, please feel free to share. But I have to admit that I am only a starter in html, css and javascript, and really in programming in general... Some repetitive tasks such as the list of people can be made much easier by using a script. For example, I burrowed the R script carousel.R from @jjallaire for another website here.

Feel free to share more best practices in editing qmd documents in neovim!

[jjallaire]

@juliantao your website does look fantastic! Do you mind if we put it in the Quarto Gallery? (cc @mine-cetinkaya-rundel)

These snippets (https://github.com/juliantao/biglvim/blob/main/snips/snippets/global.json) also look terrific. Do you have any objection to our including a bunch of them in our VS Code extension?

Finally, we also have a fledgling vim plugin (https://github.com/quarto-dev/quarto-vim) which is basically just a fork of the vim-rmarkdown plugin bound to .qmd files. It would be really cool to add support for Quarto render/preview (as we are doing for emacs: https://github.com/quarto-dev/quarto-emacs). @juliantao or @jmbuhr if either of you is interested in contributing to the vim plugin that would be awesome.

[juliantao]

@jjallaire No problem at all! Please feel free to take anything you think is helpful.

Regarding the quarto-vim plugin, I'll see how can I contribute. @jmbuhr do you mind taking a lead on the effort?

[dragonstyle]

I was really impressed with the site- it was great to see Quarto in use! For repetitive tasks, you might consider checking out custom listings (if you haven't already):

https://quarto.org/docs/websites/website-listings-custom.html

The gallery page is a good example of this in action (other than the carousel :) )...

[juliantao]

Thank you, @dragonstyle ! I just checked out the document on custom listings, it does seem very powerful and flexible. I'll give it a try when I get time.

[jmbuhr]

Happy to contribute to the vim plugin, though I use mostly use neovim and the new hotness is writing plugins using their very good lua integration instead of vimscript. With neovim's native LSP and treesitter support there are already a bunch of plugins that make working with quarto pretty straightforward, so I think for this we mainly need a wiki with a quick minimal setup to use what is already provided by the very active community. Maybe we can put some effort into contributing to those or fork some of them. For example, the markdown syntax for treesitter doesn't work with the Rmd / qmd code-block syntax. Right now it feels like a bug, but if the maintainer wants to unterstandably stick with just the commonmark specification we could consider forking it to provide a grammar for qmd and Rmd files.

[jjallaire]

I'd be 100% fine if we just focused on neovim. It seems like maybe a combination of documentation and PRs to existing plugins might be enough? (we don't necessarily need our own plugin, but I wanted to at a minimum handle code cells correctly).

[jmbuhr]

As for the state of neovim for quarto developement: A lot of plugins now work great with quarto documents. For example, we get treesitter based language aware syntax highlighting for code chunks in md / qmd documents, which looks great. I wrote a small plugin with commands to render and preview qmd documents straight form neovim and added a list of helpful plugins: https://github.com/jmbuhr/quarto-nvim

One fundamental limitation as of now: you can only attache one language server to one buffer, so we can only get autocompletion and the likes for embedded code chunks once we have a language server for quarto, which can relay the relevant parts to the python and R language servers.

[jjallaire]

Great! A couple of comments:

(1) We will eventually take the LSP we have embedded in VS Code and make it available as a command e.g. quarto lsp. We can't do this right now because Deno (our JS runtime) doesn't yet emulate enough of Nodejs to make this possible (although its on the Deno roadmap for this year). In terms of completion for embedded chunks, that is actually NOT handled by the Quarto LSP proper, but rather by "middleware" made possible by the VS Code LSP client implementation (we intercept requests on the client and attempt to delegate them to any registered handlers). If you think about it this makes complete sense as there's really now way for an entirely detached LSP server to know anything about other LSP servers.

(2) For render/preview, we have over the past couple of weeks been doing this for VS Code, and as part of this we are attempting to push a lot of the complexity formerly embedded in RStudio into quarto preview. In short, the workflow for RStudio users is just "hit the render button" and the right thing will happen. For example, if you render a non-project file you'll get the render + quarto preview for a file, if you render a project file you'll get quarto preview for the project. For both cases if you render a file and you already have a preview server running that can handle that file it actually re-uses that preview server for the render (saving quarto startup time).

I think you would do well to emulate this in neovim. The key is to use an undocumented feature of quarto preview that allows you to send an HTTP request to a running quarto preview for another file -- if the quarto preview can render it (i.e. it's for the same file or another file within the same project) then it does so and returns status 200, otherwise it returns 404). The logic is here:

https://github.com/quarto-dev/quarto-vscode/blob/ef89c6777620f318faf2918ba66efca3a510be22/src/providers/preview/preview.ts#L77-L93

The HTTP request is a special UUID with a couple of query params as seen here: https://github.com/quarto-dev/quarto-vscode/blob/ef89c6777620f318faf2918ba66efca3a510be22/src/providers/preview/preview.ts#L149

One of the major benefits of this is that users get "one button" operation of everything (don't need to learn about orchestrating the various process states and commands) and it preserves the distinction between save and render (the former being cheap and the latter being potentially many minutes -- the default behavior of quarto preview of re-rendering on save loses this distinction). In this scenario we pass --no-watch-inputs to quarto preview and then get the auto-magic update of preview based on re-render rather than save.

@cscheid We haven't yet discussed this new endpoint for use in Emacs. Perhaps we could get this working in Emacs so you understand it fully then you and @jmbuhr could take a closer look at getting it to work in neovim?

-

[jjallaire]

Also, @jmbuhr I think the ideal thing would be to give you commit access to quarto-vim and make the improvements discussed above in that repo (sort of merging your current repo w/ some enhancements w/ the official repo). Would you be game for that?

[jmbuhr]

Yes, I'd be up for that. I suppose we would have to rename vim to nvim if the plugin uses neovim functionality like the lua and lsp integration, which we need.

1) Interesting, I hope this will also integrate with nvims implementation of the lsp client. To my knowledge emacs has this polymode thing where different parts of a buffer are recognized as different languages. Neovim has this for syntax highlighting and does understand the syntax tree of a document with embedded languages via treesitter, but I think as of now it is not possible to use this knowledge for other parts such as attaching specific language servers to only code chunks.

2) Makes sense, I will try to implement the same.

[jmbuhr]

I posed the question about attaching lsp servers to different parts of a buffer on the neovim matrix room (https://app.element.io/#/room/#neovim:matrix.org/$164862827440225lRuZr:matrix.org) and here is the tldr:

Q: "Is it (currently) possible to attach different lsp clients to different parts of a buffer (decided based on treesitter)? For example to get different autocompletions for code chunks of different languages and markdown in computational notebook formats such as Rmarkdown." A: "no and it probably will never be" ... "the same things that make the built-in client fast are what make have language servers attach to sub-ranges a pain" ... "the design of neovim's change tracking code means we would have to have a major regression in performance in order to accommodate region level change tracking"

Two possible solutions:

• "there are ofc hacks one could do, like synchronizing the text/decorations from a dummy buffer to the subrange buffer"
• A meta-lsp that delegates (like I assumed initially quarto lsp would do it). There is already an example, the vue languge servers does exactly that: https://github.com/johnsoncodehk/volar An approach like this might even make the quarto vs code extension less hacky, but I haven't looked too deeply into either, yet.

[jjallaire]

The first of their suggested solutions is exactly what we do in vscode: https://github.com/quarto-dev/quarto-vscode/blob/main/src/vdoc/vdoc.ts#L23. We basically create a virtual document w/ all lines NOT of the target region's language set to empty (that way all the line number positions continue to match). As long as you can get a callback when a completion request is about to made of the LSP you can get this to work, here's where we do that in vscode: https://github.com/quarto-dev/quarto-vscode/blob/main/src/lsp/client.ts#L74-L83

[jjallaire]

Yes, I'd be up for that. I suppose we would have to rename vim to nvim if the plugin uses neovim functionality like the lua and lsp integration, which we need.

1. Interesting, I hope this will also integrate with nvims implementation of the lsp client. To my knowledge emacs has this polymode thing where different parts of a buffer are recognized as different languages. Neovim has this for syntax highlighting and does understand the syntax tree of a document with embedded languages via treesitter, but I think as of now it is not possible to use this knowledge for other parts such as attaching specific language servers to only code chunks.
2. Makes sense, I will try to implement the same.

We'll get it working in emacs then circle back here to get you going in neovim. I have no problem renaming to quarto-neovim and just supporting neovim

[cscheid]

Circling back on jj's earlier comment, we now have that quarto preview functionality on the emacs mode, and I'd be happy to help @jmbuhr put that into the (neo)vim mode.

[jjallaire]

@jmbuhr Would also be happy to combine our current vim extension with your efforts (including focusing exclusively on neovim). Let us know how you'd like to proceed.

[jmbuhr]

Sounds great, I'm in! :)

[jmbuhr]

I have now written a preview function for nvim that detets a quarto project or a single file and acts accordingly: https://github.com/jmbuhr/quarto-nvim/blob/main/lua/quarto.lua

I haven't yet figured out how quarto render can re-use a running preview server, probably because lisp (in quarto-emacs) still looks very alien to me :see_no_evil: Though I suspect that most vim users would run the final quarto render straight from the terminal anyways.

I am now trying to wrap my head around creating an invisible buffer to run the correct LSP server on for embedded languages, since (neo)vim doesn't have emacs' polymode. This would be so cool if this works!

[cscheid]

@jmbuhr we don't actually run quarto render if there's a running preview server. The function we always call is quarto-mode--maybe-preview. The trick there is that if there's a running process already, and you're asking for a preview of the same file, then what we do is POST a refresh request with a special UUID: https://github.com/quarto-dev/quarto-emacs/blob/main/quarto-mode.el#L278-L283

If you want to set up a zoom call or something to talk about it, just reach out at carlos.scheidegger@rstudio.com, I'd be happy to help.

[jmbuhr]

That would be great, I'll drop you an email :)