Dynamic remote plugin loading?

[deathcap]

Currently plugins have to be predetermined at compile time for inclusion with browserify (require('voxel-foo'), for browserify to recognize its needed). Some kind of dynamic or remote plugin loading mechanism could be worthwhile.

Notes from https://github.com/deathcap/voxpopuli/issues/23:

---

voxel-plugins allows for consistently and easily loading plugins, supports dynamic enable/disable, but you still need to pre-bundle the modules in package.json for browserify to include in bundle.js. Would be neat if plugins could be dynamically loaded across the web from anywhere.

NPM modules can load arbitrarily many other modules, so a resolution process is needed — some discussion here, including an idea to browserify NPM: https://github.com/substack/node-browserify/issues/314 browserify browserify

To load resources across the web: CORS, Access-Control-Allow-Origin: *. Tool for client-side OAuth to GitHub: Gatekeeper

requirebin + browserify-cdn gets very close to what I'd like, but the modules are "compiled" server-side using npm (may not be too much of a problem, though it would be nice to allow loading plugins directly from GitHub, gists, pastebin, etc. without server-side processing).

---

$script.get('http://example.com/foo.js', function(){}) using https://github.com/ded/script.js allows loading remote scripts (similar to script src=http://example.com/) + http://wzrd.in/ server-side compilation for dependencies?

[deathcap]

script src= would take care of executing the remote script (from raw.github.com, pastebin.com/raw.php, etc.) but the package information file package.json is JSON, not JSON-P, so it cannot cross domains without server-side processing or CORS

[deathcap]

Found this on npm: https://github.com/crcn/plugin.js "Simple plugin system for javascript" - supports remote plugin with "dnode" https://npmjs.org/package/dnode-plugin, appears to be focused on server-side node.js however.

[deathcap]

First part of this in https://github.com/deathcap/voxel-plugins/commit/76331741916f4bb7df3bb850598dd93737b1ce89 - instantiate() now accepts a plugin factory constructor, name, opts, can be passed in directly

[deathcap]

Found this for CORS: http://cors.maxogden.com/ "This API enables cross-origin requests to anywhere." via https://gist.github.com/maxogden/8795074 - demo: https://robwu.nl/cors-anywhere.html source: https://github.com/Rob--W/cors-anywhere/ - could host on https://github.com/joyent/node/wiki/Node-Hosting + and also: http://allow-any-origin.appspot.com/

[kumavis]

checkout requirebin.com if you haven't yet. it is using browserifyCDN. since processed modules are cached, things are fairly performant.

[deathcap]

http://requirebin.com https://github.com/maxogden/requirebin https://github.com/jfhbrook/browserify-cdn https://wzrd.in looks to be the closest to what I'm looking for, but maybe its possible to go further and "browserify npm" (using appropriate shims for web FS access, CORS for remote file fetching, etc.) so a remote service is not needed?

$ npm install npm
$ cat try.js 
var npm = require('npm');
npm.load();
npm.commands.install();
$ browserify try.js > b2.js

then loading into Chrome, crashes at:

/**
 * Creates a Cursor instance based off the given `writable stream` instance.
 */

function ansi (stream, options) {
  if (stream._ansicursor) {   // <-- *** cannot read property _ansicursor of undefined
    return stream._ansicursor
  } else {
    return stream._ansicursor = new Cursor(stream, options)
  }
}
module.exports = exports = ansi

from:

log.cursor = ansi(process.stderr)

process.stderr (and stdout) is undefined in browserify. Something that could be added on top of https://github.com/defunctzombie/node-process (to log to, say, the browser developer console or https://github.com/deathcap/console-widget instead)

more progress:

var Writable = require('stream').Writable;

process.stdout = new Writable();
process.stderr = new Writable();

process.stdout.write = function() {
  console.log('STDOUT', arguments);
};
process.stderr.write = function() {
  console.log('STDERR', arguments);
};

process.binding = function() {
  return {fs: ''}
};

process.argv = ['npm'];

var npm = require('npm');

npm.load();
npm.commands.install();

now crashes on fs.stat - browserify provides an empty fs object; need to replace it with https://github.com/mmckegg/web-fs . trying to replace builtins (better way to do this?):

var browserify = require('browserify');

require('browserify/lib/builtins').fs = 'node_modules/web-fs/index.js';

var b = browserify();
b.add('./try.js');
b.bundle().pipe(process.stdout);

but web-fs is not completely API-compatible with fs, it requires instantiation:

tmp $ cat create-webfs.js 
var webfs = require('web-fs');

module.exports = webfs();

require('browserify/lib/builtins').fs = 'create-webfs.js';

this now crashes in getEntry(this.entry…), where this.entry(=root parameter) is undefined, during fs.stat(p + ext …) on "/npm".

Standard error output from browserified npm is:

error reading version
 TypeError: undefined is not a function↵ info

error reading version
     at file:///tmp/b2.js:21991:25↵ info

error reading version
     at Object.<anonymous> (file:///tmp/b2.js:22414:3)↵ info

error reading version
     at Object.191.../bin/npm-cli.js (file:///tmp/b2.js:22416:4)↵ info

error reading version
     at s (file:///tmp/b2.js:1:254)↵ info

error reading version
     at file:///tmp/b2.js:1:305↵ info

error reading version
     at Object.<anonymous> (file:///tmp/b2.js:25:11)↵ info

error reading version
     at Object.1._process (file:///tmp/b2.js:30:4)↵ info

error reading version
     at s (file:///tmp/b2.js:1:254)↵ info

error reading version
     at e (file:///tmp/b2.js:1:425)↵ info

error reading version
     at file:///tmp/b2.js:1:443↵ info

error reading version
  [TypeError: undefined is not a function]↵

so there is probably much that would need to be done to make this work

edit: got a "browserified npm" to at least load (few commands work) in the browser: https://github.com/deathcap/webnpm

[kumavis]

I've played with these ideas before, a lot of exciting potential! see this long standing issue https://github.com/npm/npm-registry-couchapp/issues/108#issuecomment-73352201 and maybe sidestep it with http://cors.maxogden.com

[deathcap]

Ah, nice to see there is other interest in CORS-enabling the NPM registry, hopefully it happens!

Until then I tried a couple CORS proxies https://github.com/deathcap/webnpm/commit/944d03d2f865d48164ea65ef6e6d9a71c44b8704 https://github.com/deathcap/webnpm/issues/1, unfortunately there are some problems. http://cors.maxogden.com/http://registry.npmjs.org and http://cors-anywhere.herokuapp.com/http://registry.npmjs.org (which run https://github.com/Rob--W/cors-anywhere/) always return Access-Control-Allow-Origin: *, but this wildcard is not allowed in some cases; the server is supposed to echo back the same Origin header it receives, at least that is what CouchDB does with enable_cors. http://npm-registry-cors-proxy.herokuapp.com (running https://github.com/zeke/npm-registry-cors-proxy) also looks promising but it doesn't pass through the OPTIONS requests, failing in modern browsers. Maybe can find a way to get NPM to (disable the "credentials" flag in XHR? Chrome complains ''A wildcard '*' cannot be used in the 'Access-Control-Allow-Origin' header when the credentials flag is true.'), or worst case setup a new NPM CORS proxy.