Mapping Between Typst Source and Output

winstonewert commented 2 months ago

Is there a feature/code that enables mapping between the typst source and positions in the output? It'd be nice to be able to map from a position in the source to the page and position on that page, and back again.

I'm using typst.ts on the browser client.

Myriad-Dreamin commented 2 months ago

The feature is already at the rust side (for typst-preview) but not exposed in the JS library.

winstonewert commented 4 weeks ago

For my own purposes, I've implemented a getMapping function which produces a javascript object allowing mapping between source and output positions. You can see it here.

Basically, I go through all of the glyphs on all of the elements of all the frames on all the pages of the document, and produce a collection of parallel arrays given page number, x, y, source file, offset. Then I can read through the array in javascript to map source to output and back again.

I'd like to contribute what I've done back, but I figured there would probably be some questions about design that would need to be hashed out before I try to put it in an actuall pull request. So I thought I'd get your thoughts here.

Myriad-Dreamin commented 3 weeks ago

It will be very expensive to export entire span mapping from rust side to js side, especially when document keeps changing. It seems you're building some editor and this will become a bottleneck on big documents.

Two aspect:

There are already 3 Mapping Query APIs in Typst-Preview for reference, which can be simplified, improved, and moved it to typst.ts.
- From Compiler provided Span to Source Code Location, see EditorActorRequest::DocToSrcJumpResolve
- This is used by click outline item and jump to either source code location or scrolling preview into the position
- From Source Code to Preview, see EditorActorRequest::DocToSrcJump
- From Preview to Source Code, see WebviewActorRequest::DocumentPosition
In https://github.com/Enter-tainer/typst-preview/blob/2517b5f4d8aa22fc11b7fb06e26e501dcd0debb4/src/actor/typst.rs#L20-L22, and https://github.com/Enter-tainer/typst-preview/blob/main/src/actor/webview.rs#L23-L24
https://github.com/Enter-tainer/typst-preview/blob/main/src/actor/typst.rs#L20-L22
Incremental or not:
- It will be simpler to have a getMapping API, as rust side doesn't have to keep any state.
- Holding state in some object can bring better performance, but the API will be more complicated. https://github.com/Myriad-Dreamin/typst.ts/blob/a429fb9257bffd7ac1acafe54ca4c59d7ac8541f/packages/compiler/src/lib.rs#L475-L475

winstonewert commented 3 weeks ago

So far, I appear to be getting away with exporting the whole mapping. My theory was that since I'm exporting a whole pdf anyways, it wouldn't be a big deal to export a mapping of it alongside that. But I did figure I'd likely need to optimize eventually. Exporting the mapping does have the advantage of being very fast to lookup new locations.

The actual mapping logic behind the code you pointed to appears to live here: (https://github.com/Myriad-Dreamin/typst.ts/blob/5f91f5a2207fbcefea8e0c0e8b4c441acbc35c30/compiler/src/service/compile.rs#L501), and I built my code as an adaption of that.

Can you explain how IncrServer is intended to work? I looked at it previously, but I wasn't sure how to do anything useful with that api.

Myriad-Dreamin commented 3 weeks ago

So far, I appear to be getting away with exporting the whole mapping. My theory was that since I'm exporting a whole pdf anyways, it wouldn't be a big deal to export a mapping of it alongside that. But I did figure I'd likely need to optimize eventually. Exporting the mapping does have the advantage of being very fast to lookup new locations.

Sounds great.

Can you explain how IncrServer is intended to work? I looked at it previously, but I wasn't sure how to do anything useful with that api.

Alright, it is in experiment state currently and doesn't get mentioned in the documentation, as I'm not sure how to expose it correctly. The usage of IncrServer is simple though. You can pass an additional IncrServer object to the compile API, so that typst.ts can store state/cache at rust side. Specifically, it stores the state/cache in the IncrServer object.

The goal of IncrServer is to ensure correct lifetime of these state/cache in rust. When IncrServer is destroyed, all related data are dropped at the same time. To avoid using the raw free function from the wasm-bindgen crate directly, I make a promise that destroys the IncrServer after resolved.

typstPlugin.withIncrementalServer(
  (srv: IncrementalServer) => {
    return new Promise((dispose) => {
      const updateDoc = () => {
        const v = await typstPlugin.compile({
          mainFilePath,
          incrementalServer: srv,
        });

        typstDoc.addChangement(["diff-v1", v]);
      };

      callback({ updateDoc, dispose })
    });
  }
);

You can see the experiment code here: https://github.com/mitex-rs/mitex/blob/c5e9f9bf2c5ef9a0f7be1c981f11026a0d732137/packages/mitex-web/src/tools/underleaf.ts#L103-L111

Myriad-Dreamin / typst.ts

Mapping Between Typst Source and Output #524