search

overlookmotel / livepack

Serialize live running code to Javascript
MIT License
45 stars 1 forks source link

readme

NPM version Build Status Coverage Status

Serialize live running code to Javascript

Introduction

Most bundlers/transpilers convert Javascript code as text to some other form of Javascript code, also in text form.

Livepack is different - it takes a NodeJS app and produces Javascript from the live code as it's running. Essentially it's a serializer.

The difference from other serializers like serialize-javascript is that Livepack can handle function scopes and closures, so it's capable of serializing entire applications.

If you pack this with Livepack:

const externalVar = 123;
function foo() { return externalVar; }

...you get (externalVar => function foo() {return externalVar;})(123). The var in upper scope is captured. This works even with complex nested functions, currying etc.

What's included in the output?

Everything.

Whatever values are referenced in functions you serialize is included in the output. This includes code and objects that came from packages in node_modules.

The entire app is output as a single .js file, with no dependencies.

Why is this a good thing?

Faster startup

Many apps do time-consuming "bootstrapping" at startup (reading files from the filesystem, HTTP requests, loading data from a database).

With Livepack, you can run this "bootstrap" code, and then snapshot the state of the app at that point in time.

When you run the built app, the bootstrapping work is already done - it was done at build time instead of run time. Combined with the entire app's code being in a single file, an app built with Livepack should be faster to start.

Tree-shaking by default

Livepack doesn't perform tree-shaking exactly, but works more like a garbage collector - any value which is not referenced in the the objects/functions you serialize is omitted from the build.

This is more effective than tree-shaking as it doesn't rely on static analysis of the code, which can miss opportunities to discard unreferenced values.

Value-level code splitting

Most bundlers code split at the level of files. Livepack splits code at a far more granular level - individual Objects or Functions. This makes for smaller bundles. More details

Mix build-time and run-time code

Any values which are calculated before serialization are included in the build pre-calculated. ONE_GB = 1024 * 1024 * 1024; is serialized as the result, 1073741824. This is a trivial example, but any calculation of arbitrary complexity can be performed at build time.

Dynamic builds

This is the most compelling advantage, but the hardest to explain.

Javascript is a dynamic language, but our current build tools are static.

Livepack runs the code you give it and outputs the result. So your app can build itself. Data is code, and code is data - there's no difference between the two.

Design principles

Livepack has an emphasis on correctness. It will not always output the most compact code, but the output should perfectly reproduce the input (including function names, property descriptors etc, which tools like Babel often do not faithfully translate).

Beware ye!

Livepack is a new and experimental project. It works for NodeJS server-side code, but there are some major gaps for bundling client-side code.

Please see what's missing for more information.

The intention is to overcome the current limitations in future, and make Livepack work fully for client-side code. For now it's a proof of concept of a different approach - dynamic bundling - and what patterns it makes possible. Please see section on code spitting for examples.

Usage

Installation

npm install -D livepack

Requires NodeJS v18 or later.

CLI

npx livepack <input path(s)..> -o <output dir>

Inputs should be the entry point(s) of the app to be packed.

npx livepack src/index.js -o build
npx livepack src/index.js src/another_entry.js -o build

Entry points must export a function which will be executed when the built app is run.

This is unlike bundlers like Webpack and Rollup. You must export a function including the code you want to run when the built app is launched. Top-level code will be executed during build, not at runtime.

module.exports = function() {
  console.log('hello!');
};

or (see esm option below):

export default function() {
  console.log('hello!');
}

Resulting output:

console.log('hello!');

Promises

If your app needs to do some async work before serializing, export a Promise.

module.exports = (async () => {
  // Do async stuff
  const obj = await Promise.resolve({x: 1, y: 2});

  // Return a function which will be executed at runtime
  return function() {
    console.log(obj.x * obj.y);
  };
})();

Options

Option Usage Default
--output / -o Output directory (required)
--format / -f Output format - either esm or cjs esm
--ext JS file extension js
--map-ext Source map files extension map
--esm Enable if codebase being serialized contains ECMAScript modules (import x from 'x') Disabled
--jsx Enable if codebase being serialized contains JSX Disabled
--minify / -m Minify output Disabled
--mangle / --no-mangle Mangle (shorten) var names Follows minify
--comments / --no-comments Remove comments from source Follows minify
--entry-chunk-name Template for entry point chunk names (more info) [name]
--split-chunk-name Template for split chunk names (more info) [name].[hash]
--common-chunk-name Template for common chunk names (more info) common.[hash]
--source-maps / -s Output source maps. --source-maps inline for inline source maps. Disabled
--no-exec Output a file which exports the input rather than executes it. Exec enabled
--stats Output stats file.
Provide filename or true for livepack-stats.json.		 Disabled
--no-cache Disable instrumentation cache Cache enabled

Config file

You can set options in a livepack.config.json file rather than on command line. Config file can be in .json or .js format, in root dir of the app. If .js, must be CommonJS.

// livepack.config.json
{
  "input": "src/index.js",
  "output": "build",
  "format": "esm",
  "ext": "js",
  "mapExt": "map",
  "esm": true,
  "jsx": true,
  "minify": true,
  "mangle": true,
  "comments": false,
  "entryChunkName": "[name]",
  "splitChunkName": "[name].[hash]",
  "commonChunkName": "common.[hash]",
  "sourceMaps": true,
  "exec": true,
  "stats": false,
  "cache": true
}

input can be:

input: "src/index.js"
input: ["src/index.js", "src/other.js"]
input: {"index": "src/index.js", "another": "src/other.js"}

Then run Livepack with:

npx livepack

All of the above options are optional except input and output.

Programmatic API

There are two parts to the programmatic API.

  1. Require hook
  2. serialize() / serializeEntries() functions

Require hook

Livepack instruments the code as it runs, by patching NodeJS's require() function. This instrumentation is what allows Livepack to capture the value of variables in closures.

Your app must have an entry point which registers the require hook, and then require()s the app itself.

// index.js
require('livepack/register');
module.exports = require('./app.js');
// app.js
module.exports = function() {
  // App code here...
};

You must register the require hook before any other require() calls. The input file should be just be an entry point which require()s the app and exports it. Code in the entry point file will not be instrumented and so cannot be serialized.

The entry point file must require() the app, not import it. The app can be written in CommonJS or ESM (use esm option).

Require hook options

You can provide options to the require hook:

require('livepack/register')( {
  // Options...
} )
Option Type Usage Default
esm boolean Set to true if codebase being serialized contains ECMAScript modules (import x from 'x') false
jsx boolean Set to true if codebase being serialized contains JSX false
cache boolean If true, instrumentation cache is used to speed up Livepack true

These options correspond to CLI options, but sometimes named slightly differently.

Serialization

Use the serialize() or serializeEntries() functions to serialize. serializeEntries() is used if you have multiple entry points.

const { serialize } = require('livepack');
const js = serialize( { x: 1 } );
// js = '{x:1}'
const { serializeEntries } = require('livepack');
const files = serializeEntries( {
  index: { x: 1 },
  other: { y: 2 }
} );
// files = [
//   { type: 'entry', name: 'index', filename: 'index.js', content: '{x:1}' },
//   { type: 'entry', name: 'other', filename: 'other.js', content: '{y:1}' }
// ]

or ESM:

import { serialize } from 'livepack';
const js = serialize( { x: 1 } );
// js = '{x:1}'

Options

serialize() and serializeEntries() can be passed options as 2nd argument.

serialize( {x: 1}, {
  // Options...
} );
Option Type Usage Default
format string Output format. Valid options are js, cjs or esm (see below). 'js'
ext string JS file extension 'js'
mapExt string Source maps file extension 'map'
exec boolean Set to true to treat input as a function which should be executed when the code runs (as with CLI). Only for cjs or esm format. false
minify boolean Minify output true
mangle boolean Mangle (shorten) variable names options.minify
comments boolean Include comments in output !options.minify
files boolean true to output array of files (see below) false for serialize(),
true for serializeEntries()
strictEnv boolean true if environment code will execute in is strict mode (only relevant for js format) false for js or cjs format, true for esm
entryChunkName string Template for entry point chunk names (more info) '[name]'
splitChunkName string Template for split chunk names (more info) '[name].[hash]'
commonChunkName string Template for common chunk names (more info) 'common.[hash]'
sourceMaps boolean or 'inline' Create source maps. 'inline' adds source maps inline, true in separate .map files.
If true, files option must also be true.		 false
outputDir string Path to dir code would be output to. If provided, source maps will use relative paths (relative to outputDir). undefined

All these options (except files, outputDir and strictEnv) correspond to CLI options of the same names. Unlike the CLI, in the programmatic API exec and files options default to false and minify to true.

Output formats

Files

If the files option is set, the return value of serialize() will be an array of file objects, each with type, name, filename and content properties.

Use this if you want source maps in a separate file.

serialize(
  {x: 1},
  {files: true, format: 'esm', sourceMaps: true}
)

outputs:

[
  {
    type: 'entry',
    name: 'index',
    filename: 'index.js',
    content: 'export default{x:1}\n//# sourceMappingURL=index.js.map'
  },
  {
    type: 'source map',
    name: null,
    filename: 'index.js.map',
    content: '{"version":3,"sources":[],"names":[],"mappings":""}'
  }
]

Code splitting

Code splitting works differently in Livepack from other bundlers.

Livepack pays no attention to what files code originates in, and splits the output at the level of values, rather than at the level of files.

This produces an optimal split of the app, where each entry point only includes exactly the code it needs, and nothing more. It's more efficient than Webpack or Rollup's file-level code splitting.

Any values shared between entry points are placed in common chunks. By default, these are named common.XXXXXXXX.js, where XXXXXXXX is a hash of the file's content.

For example, if your input is:

// src/entry1.js
const { double, timesTen } = require('./shared.js');
module.exports = () => double( timesTen(10) );

// src/entry2.js
const { triple, timesTen } = require('./shared.js');
module.exports = () => triple( timesTen(20) );

// src/shared.js
module.exports = {
  double: function double(n) { return n * 2; },
  triple: function triple(n) { return n * 3; },
  timesTen: function timesTen(n) { return n * 10; }
};

Livepack will bundle this as:

// build/entry1.js
import timesTen from "./common.6N2RIGAZ.js";
export default ((double,timesTen)=>()=>double(timesTen(10)))(function double(n){return n*2},timesTen)

// build/entry2.js
import timesTen from "./common.6N2RIGAZ.js";
export default ((triple,timesTen)=>()=>triple(timesTen(20)))(function triple(n){return n*3},timesTen)

// build/common.6N2RIGAZ.js
export default function timesTen(n){return n*10}

src/shared.js has been split up. timesTen is used by both entry points, so has been placed in a common chunk. But double and triple are inlined into the entry points that use them, since they're not shared. So each entry point doesn't need to import any code it doesn't use.

You can customize how code is split to optimize caching (see below).

You may notice this output could be shorter - values are injected into a closure, when they could be accessed directly from outer scope. This is a current shortcoming of Livepack. It will be improved in a future release.

No import()

The other big difference from other bundlers is that Livepack doesn't at present support import(). It does, however, provide another mechanism to achieve the same goal.

Multiple entry points

If you have multiple entry points, use serializeEntries() or provide multiple input files to the CLI.

split()

You can customize how code is split with split(). Split will cause the value provided to be placed in a separate file and other files will import it.

This can be advantageous for caching - you may want to split off parts of your app which change infrequently.

const { split } = require('livepack');

const obj = { iAmABigObjectWhichChangesInfrequently: true, x: 123 };
split( obj );

module.exports = function getX() { return obj.x; };

Bundled output:

// index.js
import obj from "./split.V4ULTFDU.js";
export default(obj=>function getX(){return obj.x;})(obj)

// split.V4ULTFDU.js
export default {iAmABigObjectWhichChangesInfrequently:true,x:123}

If you want to specify the name of split point files, pass name as 2nd argument to split().

// Split off in a file called `my-big-object.XXXXXXXX.js`
split( obj, 'my-big-object' );

splitAsync()

splitAsync() is Livepack's version of import().

splitAsync() takes a value and returns an import function. Just like import(), this import function returns a Promise of a module object. The .default property of the module object is the value splitAsync() was called with.

When Livepack serializes an import function, it puts the value into a separate file and outputs () => import('./split.XXXXXXXX.js').

Example input:

const { splitAsync } = require('livepack');

const importDouble = splitAsync(
  function double(n) { return n * 2; }
);

module.exports = async function quadruple(n) {
  const double = (await importDouble()).default;
  return double( double(n) );
};

Output:

// index.js
export default(importDouble=>(
  async function quadruple(n){
    const double=(await importDouble()).default;
    return double(double(n))
  }
)(
  ()=>import("./split.LKCG7RVO.js")
)

// split.LKCG7RVO.js
export default function double(n){return n*2}

There's a few things to notice here:

  1. double has been split into a separate file
  2. importDouble is output as a dynamic import ()=>import("./split.LKCG7RVO.js")
  3. double didn't need to be defined in a separate file to be split off

All looks very weird? Maybe. But it does open up some patterns which usually aren't possible.

For example, you can dynamically create functions to be async imported.

Example with React

splitAsync() works well with React.lazy():

const people = [
  { firstName: 'Harrison', lastName: 'Ford', lotsMoreData: { /* ... */ } },
  { firstName: 'Marlon', lastName: 'Brando', lotsMoreData: { /* ... */ } },
  { firstName: 'Peewee', lastName: 'Herman', lotsMoreData: { /* ... */ } }
];

const lazyComponents = people.map(
  person => React.lazy(
    splitAsync(
      () => <PersonPage person={person} />
    )
  )
);

NB See here for a full runnable example expanding on this.

lazyComponents is an array of lazy-loaded components. Each will be output in a separate file, with the data for each individual bundled in. people isn't accessed from inside the function being split off, so it won't be included in the bundles, only each individual person object will be included in the file for that person.

Where it gets really interesting is that data passed in to each lazy component isn't limited to just data - it can also include functions.

So you could, for example, provide customized components for each person. e.g. include a Google Maps component only for the pages where you know the person's address, by adding a .MapComponent property to some of the person objects. Only pages where a map is needed would include the code for displaying a map.

More broadly, components can be created at build time however you like - create code according to data. It's far more flexible than the usual model.

Customizing chunk names

There are 3 options for customizing chunk names:

For each you can use placeholders [name] or [hash] within the name. e.g.:

commonChunkName must include [hash]. splitChunkName must include [hash] if any split points are not named.

These options should not include the file extension. Use ext option if you want to alter file extensions from the default .js.

If you include [hash] in entryChunkName, you may need to consult the files object returned by serialize() / serializeEntries() to get the eventual filenames of the entry points. If using the CLI, you can use the --stats option to output a stats file including this information.

What's missing

This is a new and experimental project. There are some major gaps at present.

JS features

Livepack can serialize pretty much all Javascript Functions and Objects. However, the following cannot yet be serialized:

NB Applications can use any of these within functions, just that instances of these classes can't be serialized.

with (...) {...} is also not supported where it alters the scope of a function being serialized.

Browser code

This works in part. You can, for example, build a simple React app with Livepack.

However, there are outstanding problems, which mean that Livepack is presently really only suitable for NodeJS server-side code.

Tests

Use npm test to run the tests. Use npm run cover to check coverage.

Changelog

See changelog.md

Issues

If you discover a bug, please raise an issue on Github. https://github.com/overlookmotel/livepack/issues

Contribution

Pull requests are very welcome. Please: