Open huwaireb opened 9 months ago
Yes, I have been thinking along those lines.
There are a few ideas for how this typst
package could be used:
npm install
with dependencies
having typst: "^1.0.0"
in it.compile()
that do exec("typst", args)
similar to how other CLI JS-API wrapper works (Docker, Pulumi, tar, apt, etc.)typst.compile("my typst text")
or something idk in your browserAll of these can be part of the same package.
The first one is relatively easy. Hence, it's the first one that I did. The second is just tedious JS -> arg mapping. and third is much tougher.
I've attempted and succeeded in compiling the typst-cli
package to WASIX and getting it to run on my local PC with wasmer run typst.wasm
https://github.com/jcbhmr/typst/actions but I'm still having trouble getting it to work with the Wasmer WASIX JavaScript browser API https://wasmerio.github.io/wasmer-js/
so that lead me to search "typst javascript" and you're absolutely right i did find typst.ts.
I've been trudging through the absolute monster of a codebase (there's like 10 rust crates and 10 npm packages in it lol) https://github.com/Myriad-Dreamin/typst.ts project to try and figure out how it was done! i dont need all that complexity; just enough to mirror the exposed cli API. particularly typst compile
.
typst compile: needs all the compiler infra. very tough. typst watch: idk if relevant in programmatic context? maybe. basically typst compile on loop tho. typst query: very tough. needs compiler infra plus metadata serialization. ugh. typst fonts: ezpz just use existing web fonts API in browser! https://developer.mozilla.org/en-US/docs/Web/API/CSS_Font_Loading_API https://developer.mozilla.org/en-US/docs/Web/API/Local_Font_Access_API typst update: not relevant. typst help: ezpz just print help text.
(this might be more for my benefit than yours) so far my understanding is this:
so to answer your question, yes, I have thought about it and yes I would like to add such functionality but I'm still a complete novice in WASM Rust world lol. the Node.js exec("typst", ...)
stuff; that I can do. the wasm browser stuff; that's more of an experiment and I'll see how it goes.
any input or ideas or help or code or education or resources or wasm magic is much appreciated lol. ❤️
Sounds great,
I'm curious to if you thought about merging step 1 and step 2 with the use of (napi-rs/deno-ops) to bind to the typst crates for cli/lib usage. To allow for a more cohesive, direct approach without the headache of managing Installations.
This would warrant a new cli(js) to act as a frontend to what will be the lib(ops/napi/wasm) that would for one wrap the typst crates.
This would entail separating the packages to be of that:
For shipping typst to the web, I have a general idea in regards to how it could be implemented (WASM, of course), but we'll leave that for later. Of course a custom parser is out-of-scope due to its unnecessary added complexity. As if SSR is the primary use of the end user this won't pose major problems in terms of the drawbacks of WASM such as bundle size. Since we can always rotate between the native addon (if implemented), and using WASM. Primary issue is of course ensuring, in the case of the browser & WASM, minimal bundle sizes are shipped.
However, I have to ask if you've evaluated the need for typst.js in the first place with the existence of typst.ts. To avoid unnecessary fragmentation of the community so early on. Existence can be always justified if you e.g have a different approach to the implementation consider LaTeX JS eco with the various different implementations.
I'm curious to if you thought about merging step 1 and step 2 with the use of (napi-rs/deno-ops) to bind to the typst crates for cli/lib usage. To allow for a more cohesive, direct approach without the headache of managing Installations.
Since napi-rs effectively builds platform-specific mylib.node
files (one for Windows x64, one for Linux x64, etc.) you'd run into the same "distributing a big fat binary" problem again. Their solutions is to, guess what, do #3 🤣
so effectively: instead of one typst binary for each platform, you'd end up with:
So I think that's a lot of work for not-so-much gain. Using exec("typst", ["compile",...])
is already what https://www.npmjs.com/package/docker-cli-js, https://github.com/minhhh/node-uncompress, etc. do: they wrap some fancy exec() calls in nice JS functions.
on another note lol: pulumi has a similar "distribute binaries as npm package" issue that might be related: https://github.com/pulumi/pulumi/issues/11744
However, I have to ask if you've evaluated the need for typst.js in the first place with the existence of typst.ts. To avoid unnecessary fragmentation of the community so early on. Existence can be always justified if you e.g have a different approach to the implementation consider LaTeX JS eco with the various different implementations.
"hey; what if i used wasix instead of trying to wrangle with wrappers?" <-- who knows how far that will get.
as of now the browser API is indeterminate. i have no idea. it could not exist! or it could be great! or it could just re-export typst.ts. or it could combine with typst.ts. who knows?!?
right now im just having a silly time trying to wrangle wasm and learn more about wasm/rust magic. typst seemed like a great project to do that since it compiles relatively cleanly to wasix using cargo wasix build
with minimal mods.
so far I've:
wasmer run ./typst.wasm --help
and had it workwasmer run ./typst.wasm compile test.typ
./typst.wasm
in the browser only to have the browser wasix runtime run out of memory and exit code 1. at least i think that's what it did. it was rather opaque and indeterminate.point being dont worry about fragmentation. right now typst.js https://npm.im/typst is just a redist installable version of the plain old typst binary. that's ok for now.
bottom line: my next step is to wrap those cli commands in pretty js/ts typed functions so you get autocomplete for them lol dont worry about browser wasm; im having fun experimenting with that. for now, use typst.ts as its the clear "already works" in-browser typst magic that works.
note to self: i was pointed to https://github.com/napi-rs/napi-rs/issues/796 wasm support seems to be on the horizon but not here until napi-rs v3? https://github.com/napi-rs/napi-rs/issues/1493
Since napi-rs effectively builds platform-specific
mylib.node
files (one for Windows x64, one for Linux x64, etc.) you'd run into the same "distributing a big fat binary" problem again. Their solutions is to, guess what, do #3 🤣
There's a huge difference between interfacing with a binary and library. Yes, you are distributing platform-specific software in all cases, however you miss the point of going the route of a library. When you wrap a cli you're essentially restricted with the interface, this is fine if of course the purpose is simple, you can just go with this unpleasant way of simply wrapping the cli. However, if you need granularity over the interface and reach over the underlying interface/functions which are not exposed, that is not the case. You'll have to either manually replicate these features in javascript, which is not going to end well, so you'll end up just wrapping the library anyways with napi.
Ah and did I mention that if you want to ship WASM you'll need to do it anyways 🙂? But yeah, at least you can distribute .wasm blobs instead, however as i will mention below, napi-rs supports wasm-bindgen as a target.
- recreating bindings for the cli interface through napi.rs to make sure arguments, env, etc. get forwarded to the main() rust function
This, sure is tedious. However since you'll have to implement the library wrapper anyways, is not the biggest problem, and you probably don't want to distribute both binary and library. Plus you get the advantage of having again as I mentioned control over the cli itself, features specific to typst in js could be introduced. A cli built on top of the runtime people are using to run typst offers its own merits.
- figuring out how to publish all those fat typst.node things (they're actually just dll/so renamed .node)
Handled by napi-rs, great cli offered.
- now maintaining a weird nodejs-only binding in addition to the wasm browser binding
napi-rs supports targeting wasm-bindgen, so you can avoid rewriting the bindings. And It's partially supported by the bun runtime, and there is work on going to get it to work in deno.
So I think that's a lot of work for not-so-much gain. Using
exec("typst", ["compile",...])
is already what npmjs.com/package/docker-cli-js, minhhh/node-uncompress, etc. do: they wrap some fancy exec() calls in nice JS functions.
Think again
youre right a direct nodejs <=> rust binding would be more powerful. in the future, that might be what happens! and the idea to avoid binding a typst.node and a typst.exe by just using a main.js => typst.node for the binary instead of a separate typst.exe is a good one.
but im not there yet. ill get there. still working on it. right now it works great with the cli wrapper! s othat's where it's at now. now that I've achieved getting it to have mirror with all the typst --help
commands available as js typst.*()
functions lol i feel good moving to next challenge! 👍
@jcbhmr @huwaireb web compiler (bundle size 17mb) and renderer (bundle size 290kb) separately runs in browser, which specially considers SSR frameworks, so it introduces extra overhead to understand. I created a typst.node yesterday, which does simpler binding to typst.
For benefit of a node binding rather than a binary shim to typst-cli, there is at least an advantage: you can run compiler with font loading once and comemo cache. Please see https://github.com/typst/typst/issues/2738.
exec('typst')
, there would introduce a 80ms (windows default fonts) ~ 100ms (user installing fonts) overhead to load font. With node binding you can load it once and run multiple compiler actions.
in packages/compiler/src/lib.rs there's a compile() function that takes in a path ...
It is done in by a hierarchical access model, indexing data by a standard unix path.
https://github.com/Myriad-Dreamin/typst.ts/blob/main/compiler/src/vfs/mod.rs#L125
type VfsAccessModel<M> = CachedAccessModel<OverlayAccessModel<NotifyAccessModel<M>>, Source>;
/// Create a new `Vfs` harnessing over the given `access_model` specific for
/// [`crate::world::CompilerWorld`]. With vfs, we can minimize the
/// implementation overhead for [`AccessModel`] trait.
pub struct Vfs<M: AccessModel + Sized> {
...
/// The wrapped access model.
access_model: VfsAccessModel<M>,
...
}
Interpreting CachedAccessModel<OverlayAccessModel<NotifyAccessModel<M>>, Source>
.
NotifyAccessModel
is for system notifying a vfs that some physical files are changed.
OverlayAccessModel
is for you shadowing memory content above NotifyAccessModel
correctly. There are some synchronization disaster detected long time ago, and I have solved it for typst-preview and get well-tested by preview users.CachedAccessModel
is internally for compiler storing cache within a single compilation.
Hello again 👋,
I'm curious in regards to your plans for
typst.js
@jcbhmr. From my understanding, currently it's only a wrapper around the typst binary distributed in npm. Are there any plans to go any further for e.g JS bindings? Like what is done by the typst.ts project. Or do your goals end here?