"deno compile" into executable

[matthewmueller]

I was wondering if there's any plan to support compiling deno scripts into a single executable?

I saw "Single executable" as a feature, but the code that follows seems to suggest it's deno as the single executable, not the programs executed with it.

It'd be awesome to support something like go build main.go or something like zeit/pkg.

[bartlomieju]

CC @ry

[kitsonk]

I believe this is still on the long term roadmap.

[ry]

I'm into the idea - and it's quite doable with our infrastructure. I'd estimate ~2 weeks of my time to get it working. It seems very cool, but it's unclear to me what the actual use cases are - do you have any? I also think there's some more important features that should come first.

[matthewmueller]

The main use case is to simplify deploys – you don't need to care about what's installed on the server because it's all present in that single executable, including the deno executable for the given architecture and OS.

You also tend to find issues at compile time, rather than runtime (e.g. we've all had something something this node module is not found after deploying to production).

[kitsonk]

You also tend to find issues at compile time, rather than runtime

I don't think a single binary helps or doesn't help in this case. Unless you are going to exercise all the code in tests, you aren't going to find issues. Building it into a binary doesn't help with that.

If you are using TypeScript modules, either local or remote with Deno, and you do a prefetch, all the types will be checked at least. Again, building this into a binary doesn't offer you anything more.

[hayd]

I think the use case is not server deploys but cli/gui apps where you want to lock down the deno version used and it's easier to distribute and call a single file.

The compile before deploy is solved with --prefetch if you include the cached deps.

[matthewmueller]

I don't think a single binary helps or doesn't help in this case. Unless you are going to exercise all the code in tests, you aren't going to find issues. Building it into a binary doesn't help with that.

I think you're right about this – good point @kitsonk!

I think the use case is not server deploys but cli/gui apps where you want to lock down the deno version used and it's easier to distribute and call a single file.

It depends on what you're doing, but a single binary definitely makes it easier in those cases too!

[matthewp]

Want to reiterate what @hayd says here. I like to create small single-use CLI tools that are essentially "done" and not having to worry about Deno version compatibility would be very enticing. Aside from security patches, it would be nice to have a binary that never needs upgrading.

[andyfleming]

A couple of use cases for me:

• CLI — Having a single executable makes distribution easy. Especially if cross-compiling is easy!

• Services built into simple build artifacts — This can be great for small container images, and generally nice for CI/CD workflows.

Support for this would push me over the fence of wanting to use Deno for real projects (once stable). Right now, it's not compelling enough for my use cases.

[phil294]

It seems very cool, but it's unclear to me what the actual use cases are

deno could solve an issue that Python and Node have: Ease of running foreign scripts for casual users.

The use case is simple: When my aunt asks me for a programm to organize her holiday pictures on her Windows machine, she does not want to download and install a 30 MB programming environment - she wants a 20 KB do_stuff.exe she can double click.

[ry]

Bert and I are actively working on this. Hopefully we’ll have a demo in a two weeks or so.

[ry]

FYI development has stalled on this feature while we are fixing other bugs in the system... but still on my back burner.

[andyfleming]

Thanks for the update, @ry. I'd be careful to wait too long on getting some version of this going. It seems like a feature that could quickly become infeasible unless we support it early.

[awkj]

by the way, I really found ry deeply affected by golang, study golang, use rust, write typescript.

[chenxyzl]

Thank you for your work.， Is there any progress? I look forward to it so much.

[ozanmuyes]

@awkj Good observation, I just thought the same thing. To me taking good parts from other languages/platforms and integrating something we use is great think, so thanks @ry. Haters gonna hate :sunglasses: (that issue gets me everytime).

[mnxn]

@satishbabariya I think that's a vast oversimplification of the amount of work needed to make a compiler. Implementing all the features of js/ts in a compiler is a monumental task and I would assume to be out of the scope of this project. A much more feasible way would be to embed the already existing V8 VM into the executable.

[kitsonk]

I think that's a vast oversimplification

Agreed! Many many people have tried to out perform V8 (and other advanced JavaScript runtimes). JavaScript (and therefore TypeScript) is such a permissive language that it take a lot of work to get anything that performs well. Also "TypeScript AST" also only solves part of a problem of creating what is offered by the runtime, like a highly performant garbage collector and a main event loop, just to start. All of that isn't expressed in a "TypeScript AST".

The snapshot building in Deno has been reworked and we are largely creating the internal Deno bundles from very similar code to deno bundle. We will likely unify that very soon, which leads to a path of creating something that can create a snapshot that includes the Deno runtime and the transpiled target program as one binary. For certain workloads it will greatly improve startup time too, I would suspect.

[andyfleming]

The snapshot building in Deno has been reworked and we are largely creating the internal Deno bundles from very similar code to deno bundle. We will likely unify that very soon, which leads to a path of creating something that can create a snapshot that includes the Deno runtime and the transpiled target program as one binary. For certain workloads it will greatly improve startup time too, I would suspect.

@kitsonk That's what I assumed the path would be for such a feature.

This would still be great to see!

[mehmetb]

For me, the major benefit of having this feature is to be able to make commercial versions of the code. Businesses usually don't want to expose their code to their customers.

In Node.js, there is a package called "pkg" which compiles javascript. I am not sure how they do it but I'm guessing they are forcing V8 to compile each function into bytecode and then put everything into a binary file along with node's executable.

[chenxyzl]

this is a discussion on compiling into single executable on dartlang I think it inspired the current discussion

[kitsonk]

No. This discussion predates that thread by about a year, and even before it was raised compiling user code into a binary was something i think Ry and I talked about.

[claytongulick]

NW.js does this by sort of "cheating". It's a single exe, but it's actually a very clever zip file that extracts itself and bootstraps node on first execute. I really appreciate the single exe deployment option with NW.js, and I've shipped box software with it, but would love to see deno take a better approach (if possible?).

[kitsonk]

Our approach (which I don't know if it is better or not) would be something like this:

• Deno bundles modules in a bundle.
• Deno loads the resulting bundle into a v8 isolate and takes a snapshot.
• Deno builds a binary that includes Deno and this additional snapshot.

The move to rusty_v8 should make this whole process easier, but it still isn't an easy process.

One of the biggest challenges is there are things that can be done to make snapshotting difficult. We have run into all sorts of strange things in building the internal snapshots for Deno, but we are doing some things that end users would never need to do. The challenge still is that snapshotting can throw some really strange errors that would be hard to provide back to users in a meaningful, actionable way. Something we would need to consider carefully.

[paulocoghi]

I believe that @kitsonk's approach is the way to go, and many NodeJS developers (like me and @andyfleming) may finally switch to Deno when this feature arrives.

[bmarkovic]

@kitsonik I think that's significantly more "compilation" than what zeit/pkg did last time I've inspected their resulting bundle/binary.

It simply bundled the files used by the entry script (all required JS is traversed and copied into a snapshot filesystem) and Node.js into a single executable which unpacks everything in RAM and starts it. I don't think they even bundled JS with something like rollup, they didn't mention it in their docs and my inspection of the binary confirmed as much. They did develop a separate tool called ncc later for bundling and you could feasibly use it prior, and feed the bundled script to pkg I guess.

Back then when I toyed with it as when you use strings CLI utility you could find bits of the JS source code inside and even filenames, and runtime traces would contain paths. Maybe it's different now with them, I didn't test later versions nor followed their development. We did use it to ship "boxed" code to people without them needing to bother with Node.js and it worked great. The "boxed" code would work on any Linux x86_64 so in a lot of ways similar to Go wrt deployment. Fantastic for containers as well.

I feel with grabbing a V8 snapshot you'd be skipping the lexing/parsing part of the process and startup would be faster. And for people that want their shipped code to be "obfuscated" and hard to reverse-engineer it would hardly get better than that with JS/TS.

[KyleTryon]

Adding a use case. I would love to be able to build and distribute small CLI based applications (that would accept arguments) specifically for CI platforms. A utility that has a dependency on a particular language/framework being installed is a compatibility limitation for many users who would otherwise make use of the tool if it could be shipped as its own binary.

A small example of this, for instance, is the AWS CLI v1 vs v2. The first version of the AWS CLI required Python to be installed, even in the case you are building a Node.js application. The new V2 CLI ships without the need for python to be installed locally.

[paulocoghi]

@kitsonk, is there anything the community could do to help you and other core developers on this topic?

[kitsonk]

It likely requires someone who understands V8 and Rust to a decent degree. In particular someone familiar with V8 snapshots. At a high level the process needs to be:

• Use the compiler APIs to generate a bundle of JavaScript.
• Load that JavaScript in a Deno CLI V8 isolate.
• Take a snapshot of that isolate.
• Write the Deno CLI binary and the snapshot out to disk in a single executable file (this is the part, I personally don't know how to do yet, though Bert or Ry might have an idea).

The first is basically available now, the second two should be possible given the infrastructure we have, the last as I indicate, I personally am not even sure of an approach right now.

It is certainly something that will get attention post 1.0. It is very much a desired feature. It just is on the complex end of the scale.

[mathiasrw]

Just to share the info: The #4402 PR now lets Deno load non-static snapshots. With the ability to loading snapshots at runtime we are one step closer making a single bin containing deno + snapshot + a bit of glue.

[mraerino]

This is a very interesting feature. I think before it can be attempted, some lower level primitives are needed.

First, making a snapshot of an isolate which is based on a snapshot. See this comment: https://github.com/denoland/deno/issues/1877#issuecomment-469116791 This is the LoC related to it: https://github.com/denoland/deno/blob/870622d3ccb3589eeb72a439717edf7e40d9b093/core/isolate.rs#L299

Secondly, how is Deno going to produce a binary of itself? What are common concepts for doing something like this? Do we even need the whole binary included or could we use a smaller binary that just supports the basics of executing JS? Where would that come from? I think a PoC approach could be to leverage the Rust compiler in order to compile bytes into a Deno binary. Example here: https://github.com/denoland/deno/blob/2b66b8a03e4f81cc158be40d07534f26fa762c2b/cli/js.rs#L4

[kitsonk]

@mraerino yup! You are thinking exactly along the same lines here. I think the first step would be to just be able to write out the snapshot to disk and bring it back in. Trying to have it be a single executable I believe is going to be difficult... but if we got to the point where deno run --bin=snapshot.bin or something, trying to inline would be another step.

[mraerino]

yeah, unless we load the cli utils from plain javascript we won't be able to snapshot the running isolate right now

[shanewholloway]

From my perspective, I see two complementary paths.

The primary path would be a Rust toolchain approach to support CI/CD based building with code signing workflows. The more we can make compiling a Deno + Snapshot look like normal compilation, the more workflows it can integrate with. Less magic, friendly to human audits, and it allows for downstream innovation by developers recombining the parts.

The second path is enabling Deno to create the binary executables itself, in the great tradition of self-extracting archives. I have a bit of experience with these in Python with py2exe and the like. Essentially the Snapshot payload is (re-)linked into a binary that looks for the Snapshot binary data at "the right place". In earlier days, that right place was aligned to the end of the binary, allowing you to simply concatenate the payload onto the executable. Unfortunately, the technique is dependent on the format of the binary: PE, PE/32, Mach-O, COFF, ELF, etc.

That said, the Rust compiler folks might have some modern tricks, insight, and code that could dramatically help!

[shynome]

Maybe we can lean from dotnet build single executable file mode pack all files to a single file and give this file to user, user click this executable file , then it will extract itself to one tmp dir and run the complied js file sorry for my bad english, I think this is a easy way to implement single executable file publish

[mraerino]

this is a rust tool that basicly does this: https://github.com/dgiagio/warp sadly, it's not released as a crate.

[ShivanshVij]

One thing to think about is closed source binaries. Tools like warp that simply compress and decompress the source code wouldn't work for scenarios like that and having closed source compilation would give deno a serious advantage over NodeJS

[mraerino]

warp would allow to compile-in deno snapshots which do not expose the JS/TS source code

[ShivanshVij]

Ah, my bad, it looked like Warp was decompressing into a local cache. In that case I think warp looks pretty promising.

[mraerino]

it looked like Warp was decompressing into a local cache

that's indeed what it's doing. but deno allows to snapshot the V8 state after the initial parse of the user script. so if we use that snapshot to run from the source code is not exposed

[ShivanshVij]

I'd be willing to take a shot at this, but I'm not the most familiar with Rust. Might be better to have a developer with more experience implement this.

[ozanmuyes]

Correct me if I'm wrong here's what I understand looking at the comments;

1. Write some code on your development machine
2. Once done invoke Deno to take a snapshot of the V8 isolate / state and then export it (though I'm not sure if it can be exported as a file, my initial thought for this step comes from this comment)
3. Use Warp (or similar tool / Rust code) to pack Deno, the snapshot and probably a launcher script (for packed Deno to load the extracted snapshot on the target machine)
4. Distribute packed, self-contained executable
5. On the target machine, once that self-contained executable ran, let the packer (Warp or, again something similar to that) unpack the contents (for the first-time or after an update) and run the launcher script to run Deno and make it load the snapshot (at runtime)

So basically the code can still be closed source and wouldn't be exposed. Although I'm not a Rust developer, I'm totally into Deno.

[Wallacy]

FWIW:

I just tried the Warp here with deno bundle:

$ mkdir bundle/
$ cp $HOME/.deno/bin/deno bundle/
$ deno bundle https://deno.land/std/http/file_server.ts bundle/bundle.js
$ cat <<'EOF' >> bundle/launch 
#!/bin/sh
DIR="$(cd "$(dirname "$0")" ; pwd -P)"
exec $DIR/deno run --allow-net --allow-read $DIR/bundle.js $@
EOF
$ chmod +x bundle/launch 
$ warp-packer -a linux-x64 -i bundle -e launch -o file_server

And then:

$ ./file_server 
HTTP server listening on http://0.0.0.0:4507/

It worked like a charm;

And yes, the extracted files are in $HOME/.local/share/warp/packages ; Its not "protected" like zeit/pkg ;

Maybe a binary instead a js bundle can help:

deno bundle --bin https://deno.land/std/http/file_server.ts snapshot.bin
deno run --bin  --allow-net --allow-read snapshot.bin

Warp is nice, uses a similar approach the gzexe, and because that needs to be 'extracted'. It would be nice to be able to configure this and run directly as in zeit/pkg;

[PhilParisot]

Total noob here but could there be a way for say two or more Deno executables to share snapshots between each-other on the same machine if the snapshots they use are somewhat similar, to save resources?

I could see the first Deno executable launched start a service that tracks snapshots, and every subsequent Deno executable checking first the tracking service before spinning up snapshots of their own.

At the OS level this could make a lot of sense. I could see this being used in Linux distros etc.

Just a thought.

[lukebrowell]

Would it be worth letting the OS handle the sort of copy-on-write optimisation you’re talking about PhilParisot?

[PhilParisot]

Hey @lukebrowell, like I said, total noob here, but my impression is that apps based on frameworks, for example electron, consume a huge amount of memory at runtime partly because they spin up their individual runtime environments (in electron's case, nodejs and chromium) for the sake of portability. They don't share their runtimes with all other concurrent electron apps on the same machine and I'm not sure why that optimization isn't done at the OS level.

It's a huge detractor.

Again, I could be totally wrong. Thoughts?

[paulocoghi]

@PhilParisot I believe the case is different for Deno and I vote to not have this concern about snapshot sharing.

On the examples you provided, the problem is in Electron design and the solution is not on the language or the way it compiles binaries, but to choose not to incorporate a huge browser engine inside the binary. Instead, it can use the system default engine or a smaller one, which is done on NeutralinoJS, Electrino, Quark, DeskGAP, Sciter, Azula and many others (more info here).

But, also, this is just my 2c. :smiley:

[ShivanshVij]

I'm in agreement with @paulocoghi, snapshot sharing is definitely a cool feature, however, this problem is more about how to have Deno output native binary as opposed to a complete GUI application like what electron provides. I'm sure in the coming days someone will figure out how to make electron work with Deno (I'd be interested in creating a project like that myself) and at that point, snapshot sharing would become essential

[Wallacy]

FWIW: In my very small example above, the bundle.js has ~180KiB, and Deno itselves ~51 MiB; Doesn't worth over-complicate the solution to share the snapshot or run-time if you dont "extract" the binary to any place. The best is make like zeit/pkg or nexe and just make a regular uncompressed binary.

You can always use the gzexe or other library to do this after; (gzexe compress deno to 20MiB)

[paulocoghi]

It seems promising to look and learn what could be eventually helpful on the approach of Tauri (github and website), that is made in Rust.

The generated binaries are extremely small.

Maybe @lucasfernog (hello from Brazil, Lucas!) can detail his approach :)