Feature: live-preview from memory on keystroke

[dvdvgt]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the discussions and believe that my question is not already covered.

Feature Request

Overview

I would like to implement the capability to display a live preview of the current typst document that also changes on every keystroke akin to how live preview works in the web app, or, similarly, in VSCode for markdown documents. This is different from the already existing option to export the PDF on every keystroke because

1. exporting the PDF on every keystroke writes to the disk on every change, which seems wasteful. Whereas live preview, as described here, is to only read the preview image from memory, which is both faster and does not cause wear on the disk.
2. It can be more easily optimized as you could potential only render those images which are currently visible. Also, it should be possible to leverage incremental compilation.

Here's how it could look like and how I implemented it crudely for a proof-of-concept.

Implementation

The way I implemented it for a proof of concept is to expose a command preview that takes a file URI and returns a base64 encoded string containing the rendered document as PNG. Then, the extension issues a executeCommand("typst-lsp.preview", uri) on every change to the currently opened editor window. The string is then wrapped in a image tag <img src="data:image/png;base64,..." /> and displayed in a simple webview panel.

Of course, this is far from the best possible implementation. As I do not have much experience with the LSP or with async in Rust, I would be interested to hear your ideas on how to implement this efficiently. With some mentoring, I am confident that I can pull this off :) I am looking forward to your responses!

[Enter-tainer]

See https://github.com/Enter-tainer/typst-preview-vscode. It supports live preview (in an ugly way), incremental compilation, and update-on-visible functionality. I'm also thinking about how to incorporate this feature into this project!

[Enter-tainer]

incorporate this feature into this project

A key point is how to pass binary data(the framebuffer) between the lsp server and the webview. AFAIK, lsp works on json-rpc, therefore binary data is not possible. Maybe websocket is still needed? I'm not quite sure if this is in typst-lsp's scope.

[dvdvgt]

AFAIK, lsp works on json-rpc, therefore binary data is not possible.

That's true, but we can encode the binary data as a string using base64 and then send it to the webview.

[Enter-tainer]

AFAIK, lsp works on json-rpc, therefore binary data is not possible.

That's true, but we can encode the binary data as a string using base64 and then send it to the webview.

Indeed, but I'm afraid that encoding can hurt performance. The pipeline would be b64 encode -> transmission -> b64 decode. Also, b64 makes the data larger. I would always prefer raw binary data if possible.

Initially, I did a quick prototype by exporting png files, encoding them to b64 and sending them to frontend. But it's pretty slow. So I currently just send raw framebuffers.

Raw framebuffers can be huge. A single page of document can be as large as 8MB. I tried some tricks to make them smaller(so as to make the whole pipeline faster) but in vain, such as deflate-frame(takes longer time to compress data), rle encoding (decoding is slow in frontend).

[nvarner]

There's a PR that was also experimenting with this (#60). Someone there mentioned Base64 as well, and someone considered putting the Typst compiler into WASM and embedding it into the visualization. I don't like that approach, but if encoding and decoding really are slow, it may be a viable option.

[nvarner]

Using websockets is an interesting idea. I haven't tried serving over two interfaces from the same binary, but if it's possible, that may make the most sense.

[Enter-tainer]

The official web app follows the wasm approach. It doesn't require any kind of data transmission.(So it supposed to be fast, although wasm itself can cause some slowdown.) However, the wasm typst cannot read local fonts, making it totally unusable.😭

[nvarner]

I'm wondering about shared memory now. I worry about portability and fragility (so a websocket/base64 backup would be needed), but that would probably minimize the overhead.

[dmorawetz]

Wouldn't it be possible to output to a tmpfs location, and show from there as a first workaround? #118 would allow to output to /tmp, and pointing an auto-refreshing PDF viewer there would enable a near-real-time preview without touching the disk.

I don't have much experience with this area, just wanted to throw in my thoughts. I also don't know how performant this would be, but probably good enough until a better solution gets implemented.

[Enter-tainer]

I think the bottleneck here is the redundant pipeline: render to pdf -> parse pdf -> render pdf to screen. I don't think tmpfs can resolve this issue.

[Enter-tainer]

typst.ts https://github.com/Myriad-Dreamin/typst.ts proposes a way to directly render the document to canvas, this might be the fastest approach available.

[xsro]

I am trying to render only current page in png in my fork. The preview looks faster but still slower than official webapp.

[tmistele]

Fwiw, I've toyed around with rendering directly to a slint UI without a pdf/png roundtrip. Performance seems to be quite good. Since that approach involves a separate window being opened by slint it probably isn't suitable for vscode integration. On the other hand, the advantage is that it works with any editor with lsp support.

https://github.com/nvarner/typst-lsp/assets/29761456/a73b2393-ff0b-48ed-bcf1-fceb128047f8

[Myriad-Dreamin]

@tmistele what's the performance of slint on pageless document? I'm working on these case in typst-preview. You can test it by editting following file.

#set page(height: auto)

== Introduction

edit me

#lorem(25000)

[tmistele]

Yeah, performance is not so great for huge pages @Myriad-Dreamin . On each change the visible page is re-rendered which in this case is the whole document. Most of the time is spent in typst_render::render. Actually displaying the resulting image buffer using slint is still reasonably fast. Edit: I spoke too soon, a somewhat smaller but comparable amount of the time is spent in slint::SharedPixelBuffer::clone_from_slice. Maybe there's room for optimization there.

https://github.com/nvarner/typst-lsp/assets/29761456/d11b6da7-b954-472e-87ac-4026f712534a

[memeplex]

As a user of typst-preview, which does what it announces very well, I would like to see you joining efforts. typst-lsp relies on an external pdf previewer. This is slow due to the aforementioned reasons, but I believe another, maybe worse, problem is that previewers in VSCode marketplace have significant shortcomings: not frequently updated or plainly outdated or don't autoreload or lose position on autoreload or are a risky closed-source blob, etc. IMO the most sensible option is to install LaTeX Workshop in order to take advantage of its previewer. I've never tried working with both typst-lsp and typst-preview at the same time, I guess it may be workable but it seems like there is some overlapping. Maybe the only regret I have regarding typst-preview is the lack of control over the typst version. Ideally I would like to be able to provide another executable installed e.g. using brew. But perhaps this would hurt performance, I don't know.

[Enter-tainer]

they can work together. (and actually, they work pretty well!) Merging typst-lsp and typst-preview is not impossible but i think it will not happen in near future.

One main burden i can think of is that the design model of lsp and previewer is somehow different: in a vscode project, there is at most 1 lsp server, but there can be more than 1 preview panel. If typst-preview gives up multi-preview-panel support, we can merge them now. But i believe we can have better options!

[memeplex]

Ok, I don't mind having two extensions installed at all if there is a more or less well-defined boundary between them. Is it just a matter of disabling the auto-export feature of typst-preview? Moreover, with typst-lsp there is some syntax highlighting but the grammar may be provided by VSCode itself.

[Enter-tainer]

currently typst-preview only focus on low latency and realtime preview. typst-lsp does all other heavy stuffs, like syntax highlighting, auto complete, diagnostics, hover docs, ...

you can disable lsp's export-on-type feature if you find it annoying.

[memeplex]

Ok, I've been using lsp+preview and I'm very pleased with the experience ❤️.

To keep this on topic, I've opened a discussion regarding the "versions hell" that it entails (I've 3 different versions of typst installed in my system right now).