Embedder's guide to snapshotting

[bajtos]

_Continuing the discussion from https://github.com/denoland/deno/pull/18271#discussion_r1150218083._

It would be great to have a guide for embedders showing how to set up their build to create a snapshot of all JS files bundled in their custom runtime to get faster startup times.

I can probably figure this out by reading the source code of Deno core & cli, but it would be much nicer if there were a guide. Having said that, both the guide and snapshotting are pretty much "nice to have" for us right now.

Another possible solution is to add an example to deno_core to show this in actions.

Quoting from https://github.com/denoland/deno/pull/18271#discussion_r1150259267:

there's no official guide - but long story short - you need to define extension! in your build.rs script when creating a snapshot and include all the files you want. Looking at runtime/build.rs and cli/build.rs (the latter creates a snapshot from existing snapshot) is the fastest way to see it in action. If you could open an issue about it we can look into adding some examples to deno_core.

[orvitpng]

Are you looking to snapshot deno_core or deno_runtime. I have a project that is in active development if you'd like a makeshift example of snapshotting the deno_runtime, you can view orvitpng/evo:base_project. ext/src/interop.rs for making an extension, and runtime/build.rs for the actual snapshotting. I cannot promise that this code is perfect, but it is very similar to how Deno's cli handles things.

edit: Changed the URL as the project has been restructured. Either way, this code should be ignored because current usage has changed because snapshotting is a very volatile part of Deno.

[bajtos]

Are you looking to snapshot deno_core or deno_runtime.

deno_core plus a bunch of Deno extensions like deno_fetch plus some of our own extensions.

See https://github.com/filecoin-station/zinnia:

• ext/libp2p is our custom extension
• runtime contains all Deno bits
• cli wraps our runtime into a CLI binary

I have a project that is in active development if you'd like a makeshift example of snapshotting the deno_runtime, you can view orvitpng/evo:base_project. ext/src/interop.rs for making an extension, and runtime/build.rs for the actual snapshotting. I cannot promise that this code is perfect, but it is very similar to how Deno's cli handles things.

Nice! Thank you @orvitpng for sharing this as an example to inspire from 🙇🏻

[bartlomieju]

We'll do a live session on YouTube, next Wednesday and we'll talk about snapshotting. Feel free to put your questions here, we'll answer them and convert it into a blog post/manual page.

[bajtos]

We'll do a live session on YouTube, next Wednesday and we'll talk about snapshotting. Feel free to put your questions here, we'll answer them and convert it into a blog post/manual page.

Awesome! I may not be able to join the live session, but I am looking forward to watch the recording and read the content you create.

1. Let's say our custom runtime is following the architecture of Deno, where we have:

   • runtime crate wiring together deno_core, other Deno extensions and our custom extensions
   • cli crate that builds on top of our runtime and provides a CLI binary that can run arbitrary JavaScript files in our custom runtime

   Q: Where should we create the snapshots? In runtime, cli or both?

   • If we create a snapshot while building runtime, how can the cli crate initialise runtime from the snapshot?
   • If we should create snapshots both in runtime and cli, can the cli snapshotting step build on top of the snapshot offered by runtime? How?

2. I noticed that the file js/99_main.js gets a special treatment different from other deno_runtime JavaScript files like 90_deno_ns.js and 98_global_scope.js. Why is that needed?

   https://github.com/denoland/deno/blob/2184103a5ec7d7ae6fcfe2ecb79ad481498f5f8d/runtime/build.rs#L274-L280

3. What is the recommended snapshot configuration? The struct CreateSnapshotOptions does not have any documentation. In particular, when and how to use snapshot_path and startup_snapshot?

   https://github.com/denoland/deno/blob/2184103a5ec7d7ae6fcfe2ecb79ad481498f5f8d/core/snapshot_util.rs#L15-L22

4. IIUC, when using snapshotting: when building the Vec of extensions to add to the Deno core JS runtime, we call init_ops_and_esm during the build step and init_ops at runtime. Is that correct?

[crowlKats]

Hey @bajtos; I have answered your questions during yesterday's livestream. Regarding your 4th question: yes, you are correct; extensions during the build step are initialized with both esm & ops, and during runtime you just initialize the ops

[lambtron]

A link to the youtube of the livestream: https://www.youtube.com/watch?v=zlJrMGm-XeA

There is a table of contents at the bottom so you can skip to the end where Leo goes through the questions in this issue.

[bajtos]

Found the blog post, too: https://deno.com/blog/roll-your-own-javascript-runtime-pt3

Thank you for writing this up! ❤️