API Considerations

[max-mapper]

forgot to mention a previous attempt I made at this:

https://github.com/maxogden/snuggie

positives:

I'm pretty happy with the API https://github.com/maxogden/snuggie#usage

I think exposing the function requestHandler(req, res){}) functions is a good pattern as it doesnt force people to use a particular http stack

also the POSTing/receiving JSON seems to be the simplest way I can think of to achieve the goals of browserify-as-a-service. multipart uploads suck if you're not using an html form

negatives:

the main drawback is that it uses browserify V1 and browserify V2 doesn't let you pass an in-memory js entry e.g. https://github.com/maxogden/snuggie/blob/master/index.js#L46-L55

it also doesn't have any of the awesomeness that is in this repo in terms of handling multiple versions of modules

[max-mapper]

couple more thoughts:

it would be cool to break out the pure API stuff into a module called browserify-api or something like that, but still have your express server example as an easy to run demo

to play with snuggie you can goto http://creator.voxeljs.com/ and look at the network traffic when it compiles the game

[grncdr]

I totally agree with all of your points re: separating the various concerns into different modules, in particular the bundle building should be separated from the HTTP handler. This repo/package will probably end up holding just a single HTTP handler factory function, something like require('browservicify')(options) //=> function (req, res), where options will contain configuration that will be passed down to other components.

What other all of the other components are is TBD, but I think the current ./lib/bundler.js should be split into (at least) these modules:

• inline-dependencies - recursively resolve dependencies and inline all package.json contents (I wrote this today)
• prepare-package - given a package.json, resolve all dependencies to generate a unique signature, then install those dependencies in a new directory if required.
• browserify-bundle-factory - given a "bundle spec" (source files, browserify options, and a prepare callback), return a browserify stream

I want browserify-bundle-factory to be easily usable by other projects, so I'm not sure if it makes more sense to have it depend on prepare-package (simple installation, but adds a possibly unecessary dependency) or to require users to provide a callback (flexibility, but if prepare-package covers the most common use-case it's annoying to require module users to install both and pass it in themselves).

browserify V2 doesn't let you pass an in-memory js entry

I noticed that as well, do you happen to know if @substack would consider re-adding (or accepting a PR for) the ability to browserify from strings? Seems like it would be useful in lots of tools.

[thlorenz]

Cross posting from a similar thread by @jesusabdullah:

Packages

As mentioned in IRC, I'd like to make this work for packages on npm. Lets say I wanna use deep-equal in the browser, I'd do:

GET /bundled/deep-equal
GET /bundled-debug/deep-equal

Latter one triggers --debug mode and thus includes source maps.

What it would do is:

npm install deep-equal

## in non-debug mode
browserify package > bundle.js

## in debug mode
browserify --debug package > bundle.js

# Then send me that bundle

I'm not sure at the moment how to tell browserify to bundle the current package browserify package.json? I'm sure @substack will know ;)

Attaching to window

Since if someone gets this via a script tag, it would have to expose itself somehow. Something like window.bundled['deep-equal'] = require('deep-equal') included in the bundle may do the trick. It's kind of the anti- browserify-shim and somewhat hacky and ugly but will work.

Transforms

Two options to handle transforms

• ideally the package should define them somewhere (i.e. in a "transforms" field) so that they could be installed and then included when browserify is run
• alternatively they could be included in a query GET /bundled-debug/deep-equal?transfom=brfs

Problem with the query solution is that then the caller would have to know what the transforms are, although a link including the query could be posted in the readme of the package to make that easier.

I personally would favor either the solution of the transforms field or at least require the transforms to be included as dependencies.

In the worst case we could have a list of packages that are browserify transforms and run all of the ones that are included as dependencies (somewhat brittle though).

Solutions in the making

I started working on brundle which will allow applications that are not using browserify to consume browserifyable modules.

I believe it's more in line of what @grncdr would like to have, so different enough from snuggie and maybe solving a slightly different yet related problem. It is totally feasible however to use this on a server to provide bundled modules on demand.

[thlorenz]

So never mind about that brundle thing. @Raynos pointed me at browserify --standalone. So that problem had already been solved (at least as a cli tool).

[max-mapper]

had a chat with @substack and @kumavis in person yesterday, came up with an idea:

consider a gradient where on one end you have the browser running bundles and node doing all installs/bundling/caching/sandboxing and on the other you have no node server and the browser installs and bundles directly from NPM

we discussed options in this gradient and IMO the most flexible option is:

1. browser says to node "I need these modules at these versions"
2. if that module + version is already installed node sends an already cached browserify -r module output back for each module
3. if a module is missing node installs the module at the right version (or just extracts the tarball directly from npm -- this might be the easiest method) and runs browserify -r module and caches the output
4. when the browser receives these individual module bundles it stores them in indexeddb
5. when all bundles arrive the browser makes a bundle out of them (I believe you just have to concatenate in the right way and then put the entry at the end)

this approach has some benefits:

• easier to sandbox in node: disable pre/post install hooks on NPM and disable transforms in browserify
• once the modules are cached in the client no networking is needed to make new bundles

and some drawbacks:

• only works with newer browsers, but since this whole use case only really makes sense for heavy apps/development apps like in browser editors I think thats okay

thoughts?

[max-mapper]

also @substack said if you could pipe an entry to browserify then it would be sweet, and would fill the missing void where you used to be able to specify an entry from a string

[jfhbrook]

Neat. That sounds reasonable enough to implement. :)

[kumavis]

While getting the fetch-and-bundle system (npm+browserify) running entirely on the client is not impossible, the advantages are few. At this time, this work can be performed by a server using existing tools, as suggested above by @maxogden.

[kumavis]

The condition "only works with newer browsers" is only for the persistence aspect.

[max-mapper]

ah, right. and I guess localstorage is everywhere

[grncdr]

if a module is missing node installs the module at the right version (or just extracts the tarball directly from npm -- this might be the easiest method) and runs browserify -r module and caches the output

I've actually written a small module that just uses npms cache layout and symlinks all dependencies in place, combining this with the alternate version of create-dependency-stream that uses npm.commands.cache.read to resolve versions and fetch package.json contents, means you end up with all dependencies extracted at known locations with all of their dependencies already in place.

My approach right now has bee for browserify-bundle-factory to hash the dependency tree and build a new working directory containing a node_modules sub-directory full of symlinks into the cache, then the source code to be browserified is hashed to create a bundle directory name beside node_modules, the source code is written into that directory and browserified.

This means there's no redundant package storage, but it does involve a lot of disk I/O...

also @substack said if you could pipe an entry to browserify then it would be sweet, and would fill the missing void where you used to be able to specify an entry from a string

Exactly where I want to go with this next! He's added support for passing streams to browser-pack, so browserify itself (and probably browser-resolve) needs a way to pass a readable stream through with a filename. (e.g. b.require('./my-index.js', {stream: readableStream})), I don't know when I'll have time to get to that, and writing stuff to disk works well enough for the moment.

easier to sandbox in node: disable pre/post install hooks on NPM and disable transforms in browserify

Yep, there's no need to ever build anything because we can only browserify pure javascript anyways.

So, this is further along the gradient towards "the server does everything", but it wouldn't be a lot of work to move it back the other way. In particular I was planning to support an HTTP interface like GET /npm/package-name/semver that works how you described above, but I'm not too concerned about bikeshedding how the actual HTTP frontend works just yet.

@kumavis said: While getting the fetch-and-bundle system (npm+browserify) running entirely on the client is not impossible, the advantages are few. At this time, this work can be performed by a server using existing tools, as suggested above by @maxogden.

I somewhat disagree, but I'm going to leveraging as much of npm's code as possible to get things up and running. After that, I might return to the resolve-dependency-versions and get-package-jsons to build an in-browser package fetching and installation pipeline, just because I think it would be awesome.

ANYWAYS... I apparently had time to write all that out, so maybe I'll get back to pushing code now :wink:

• My symlinking module requires an alternate npm cache dir before doing anything.

[jfhbrook]

Hey dudes,

I hacked together v3 of browserify-cdn:

https://github.com/jesusabdullah/browserify-cdn

Would love feedback, code contributions, etc. Note that this isn't hosted anywhere right now. To run it, clone/npm install/npm start .

Cheers!

Edit: Yes, this is the third time I've written this. Hopefully THIS time it doesn't suck. ;)

[jfhbrook]

Also @grncdr I should've said something before, but...we should probably collaborate on this. You down? Let's chat on irc, I'm jesusabdullah in #stackvm on freenode.

[grncdr]

I'm down to collaborate, I'm on IRC and in #stackvm now (grncdr)