Qard
commented
7 years ago - A core
tarmodule might help here. - What about dependency deduping? It'd be a shame to throw that away.
- For native code, I wonder if just using embedded resources in
.nodefiles would be worth considering. It's possible everything could just be a.nodefile which includes some extra stuff in it, and things could even be precompiled as bytecode as part of that. I know.nodefiles are platform-specific, but that might actually be a good thing--one could create platform-specific builds with different dependency sets and less conditional branching.
bmeck
refack
ljharb
pmq20
prettydiff
This is a proposal for a feature that can create and use single files as a means to distribute packages be added to Node's core. In this document, use cases are presented but no implementation details are given. This is a rather large undertaking and should be delivered in smaller parts than the whole picture given here. EPs which include exact implementation details should be created as this work is agreed upon and split into smaller chunks.
This issue has been created as a successor to the node --install.
Use cases
Application users
People wishing to distribute their node.js applications face a tough time currently. There is no standard way in which to distribute these. Docker is increasingly common, but so is telling people to go install dev tools such as
npmon their machine and run something from the CLI. Compare this to Electron or Java.jarfiles which require double clicking a single file.This case notably has the following features of interest:
_third_party_main(native addon statics likeprocess._linkedBindingor through extraction)There are concerns here about how
node-gypfunctions not being compatible with statically linking in a dynamic manner for binaries. In existing research like noda the solution was to extract the.nodefile to disk as a temporary file and then load it. For truly dynamic distributables that cannot be statically linked, this approach is commonplace.Runtime plugins
The applications created by node.js have no standard for which to declare how application plugins should be created. Commonly this involves installing
devDependencies, but for true runtime determined plugins this functionality does not make sense. Examples of runtime plugins are things such as servelets or minecraft like addons that are not depended upon by the applicaiton intended to run the plugin.In well designed cases you can simply drop a
.dllor.jarinto a folder or onto a GUI and have an application know what to do. This should be easier and less home grown for the common case when developing on Node.This case notably has the following features of interest:
Verification of distributables
Due to security or connectivity, verification of the integrity of node.js modules is difficult. There is no clear standard for signing of the code, nor is there a distribution format to even create a digest from.
Offline distribution is also important in places with low fidelity connections or no connections. Rural locations, network limited secure environments, etc. all would benefit from having a standard of both distribution and verification.
This case notably has the following features of interest:
Isolation
It should not be possible for other modules to modify the manner in which snapshot internals are loaded. This allows a rudimentary guarantee of evaluation order and dependency resolution but does not intend to prevent mutation of global state.
It may be prudent to restrict the ability to load certain modules while within a distributable (
fs,child_process), doing so is potentially configured by the CA/Keystore standard.Mutability
In order for verification to occur, the internal state of a distributable must not have changed. If the only time required for verification is at install, extraction to disk is viable as any mutation after install is allowed.
If verification needs to be done when the runtime loads a distributable, an immutable state needs to be setup for a distributable.
Creation of reproducable dependency graphs
One of the largest problems with tooling in node modules is the lack of reproducable builds. Even with so called lock files, lack of an source of authority mechanism and signatures can lead to problems due to using naive things like checksums and name/version pairs. Creating a fully standardized bundle format that includes the entire assets of a module or application would prevent home grown solutions.
With a standard format in place it would also be possible to do a full audit of a dependency graph without having custom tools or having to recreate the entire dir structure on disk.
This case notably has the following features of interest:
Future Potential
The following I believe are best left to the future, but should be important for any design decisions.
Encryption
It should be possible to use encryption to generate a distributable that requires a key to be decrypted.
Privacy
In combination with the new bytecode system in V8 for example allows declaring the source string such that the JS debugger would not be able to see the source that generated the bytecode a modest level of code privacy can be obtained.
De-duplication
Sometimes, there are many duplicate files within a single application due to things like LICENSE files. While extracted to disk a solution is often to use hard links, it should be possible for a distributable to avoid declaring the same file body repeatedly.
Poly-distributables
When creating distributions, it is sometimes useful to include multiple variations of an application for testing, localization, etc. This may be due to lack of tooling on a target machine, or it may be to ease a specific workflow.
Distributables could allow some mechanism by which they can include multiple variants withing a single distributable for their internals based upon things such as: language, architecture, debugging, etc. This most likely is not intrinsic to the file format, but more likely revolves around inteligent entry points.
Shared library
This is the most complicated problem to solve. Many times, when creating applications, duplicate dependencies are extracted to disk in multiple places. Some attempts to avoid this such as using symlinks to a global cache exist. In order to alleviate both an audit, updating, and file size complexity this should be looked at. In particular, having a standard signature format is not enough.
The layout of a distributable is not something in scope of this proposed feature, that is left to package managers. Existing package managers can have complex relations where nested packages are not easily expressed using just symlinks due to how realpathing works in Node's module system.
I suspect, the creation of a well defined shared cache based upon signature of a distributable could be shared amongst all the package managers however.
UX Bikeshedding
All names/UX happily looking for improvement. I will be using
.distributablefor an opaque file extension.Runs the entry point to
app.distributableCreates a distributable, optionally using
--crtto declare signing information.Extract
app.distributableto the current directory. This is probably not terribly useful except to try and muck with source.Similar to
npm --global install, placeapp.distributablein a configured place and setup permisions and command shims.Automatically run on install or extract, this checks
app.distributableagainst Node's verification policy (signature may not be the only criteria).Create a single binary which runs
app.distributablewhen started. Does not include debugger or REPL by default.