Asset Pipe

This is an initial and general overview of this system.

The initial purpose of this project is to be able to build assets bundles (javascript and css) based on assets which initially is located in multiple other micro services. A final asset bundle should be minified and not contain duplicate assets even if the same asset is used in multiple micro servies.

The data formats

Internally in this system we operate with two terms related to data format; "asset feed" and "asset bundle".

An asset bundle is generic term for a plain css or javascript file that has been produced based on multiple sources. An asset bundle can be executed in the browser.

An asset feed is type of feed that contains a set of resolved dependendt assets. In its simplest form it is an array with objects where each object has a unique ID and the source of an asset.

Example:

[
  {
    "id":"c645cf572a8f5acf8716e4846b408d3b1ca45c58",
    "source":"\"use strict\";module.exports.world=function(){return\"world\"};",
    "deps":{},
    "file":"./assets/js/bar.js"
  },
  {
    "id":"c86b4a60d1a97fab876be8d0e3739cd6643c4bc6",
    "source":"\"use strict\";var bar=require(\"./bar.js\");module.exports.helloworld=function(){return\"hello from server-micro-a\"},module.exports.hello=function(){return\"hello\"};",
    "deps":{"./bar.js":"c645cf572a8f5acf8716e4846b408d3b1ca45c58"},
    "file":"./assets/js/foo.js"
  },
  {
    "id":"d2377196700b98998945de5e91e163b8a79cf5c4",
    "source":"\"use strict\";var foo=require(\"./foo.js\"),bar=require(\"./bar.js\");console.log(\"server-micro-a:\",foo.helloworld()),console.log(\"server-micro-a:\",bar.world()+\" \"+foo.hello());",
    "deps":{"./bar.js":"c645cf572a8f5acf8716e4846b408d3b1ca45c58","./foo.js":"c86b4a60d1a97fab876be8d0e3739cd6643c4bc6"},
    "file":"./assets/js/main.js",
    "entry":true
  }
]

This format is the internal data exchange format in Browserify.

The repos

This system consist of multiple repos where each repo are the source of an npm package. They are:

asset-pipe-js-writer

This is a module for reading a CommonJS module entry point and resolving all the dependencies into an asset feed. This module is a Read Stream.

asset-pipe-js-reader

This is a module for converting a set of javascript asset feeds into one asset bundle. This module is a Transform Stream.

asset-pipe-sink-fs

This is a sink for writing and reading asset feeds and asset bundles to / from a local file system. This module has a write method which is a Write Stream and a read method which is a Read Stream.

asset-pipe-sink-s3

This is a sink for writing and reading asset feeds and asset bundles to / from Amazon S3. This module has a write method which is a Write Stream and a read method which is a Read Stream.

asset-pipe-server

This is a server which persist asset feeds uploaded to it. It can also build and and persist asset bundles based on the uploaded asset feeds. This server use the asset-pipe-js-reader module to make asset bundles out of asset feeds. By default it uses the asset-pipe-sink-fs module to handle persistence of feeds and bundles. One can swap out the asset-pipe-sink-fs module with the asset-pipe-sink-s3 sink for persisting to Amazon S3 or one can write alternative sinks for persisting to Redis or CouchDB etc.. This module can run as a standalone server but it does also provide a Express router which can be plugged into a second Express server if one want to run it with alternative configuration / setup.

asset-pipe-mid

This is a set of helper functions and a set of Express middleware which each node.js based micro service can use.

It contains a function for uploading an asset feed to the asset-pipe-server. This can be used to ex upload the asset feed for the micro service on start of the server.

It does also contain a function for triggering a build of an asset bundle in the asset-pipe-server. This can ex be triggered by a server which has the role of stitching content from multiple micro services into a whole page.

There is also an Express middleware for serving an asset-feed and for making an asset bundle. These middleware are to help with development of assets in the micro service without having to upload files to the asset-pipe-server in development.

This module use both the asset-pipe-js-writer and the asset-pipe-js-reader.

asset-pipe-cli

This is a command line client for making asset feeds and interacting with the asset-pipe-server. It is intended to be used in non-node.js environments or as a part of a build pipeline. It can ex sit in the build process and make an asset feed and upload it to the asset-pipe-server on deploy of a micro service instead of the micro service running this on start (as described under the asset-pipe-module).

This module use the asset-pipe-js-writer module.

asset-pipe-mock-servers

This is a set of mock micro services for testing. These mock servers use the asset-pipe-mid module to create asset feeds and upload them to the asset-pipe-server on start.

asset-pipe / asset-pipe-build-server

Document system overview #1