Add UMD

[OliverJAsh]

This module should work in CommonJS, AMD, and browser globals. Currently it only works for AMD.

Ideally this wrapping would be done as part of the build process.

Re. https://github.com/guardian/scribe/issues/77

[pdufour]

+1

[josephmisiti]

+1

[marcelklehr]

Also, I would really appreciate support for component.

[simonsmith]

:+1:

If not, then exposing it as a global like Flight have done would be useful.

[mjackson]

:thumbsup:

@OliverJAsh I'd like to take this work on, but I need a little guidance. In your opinion, at which step of the build process should the conversion to UMD be done?

Given the fact that AMD is baked into all of the modules, one route might be to let RequireJS resolve all the dependencies and create the AMD module, and then bundle a minimal define implementation (perhaps almond). Then, wrap the whole thing in UMD's returnExports.

Another (and more future-proof) route would be to convert all existing AMD modules to ES6 modules. Then, use Square's ES6 module transpiler to output to AMD, CommonJS, or browser globals as needed. It's not UMD, but it avoids the need for a define shim and is just as flexible in the long run.

EDIT: This would also likely require a plumber-es6-module-transpiler plugin.

Of course, a 3rd option would be to try and find something that knows how to translate AMD to UMD, but I'm not aware of any such tools that are well-maintained and in wide use.

[Rich-Harris]

@mjijackson @OliverJAsh There's a library called amdclean which I find very useful for stuff like this. It takes the output of the RequireJS optimiser and removes all the AMD boilerplate (sample config code), exposing each module as a uniquely-named variable. Then all you need to do is wrap the whole thing up:

(function (global) {
  'use strict';

  /* amdclean'd r.js output goes here... */

  if (typeof define === 'function' && define.amd) {
    define( function () { return Scribe; });
  } else if (typeof module !== 'undefined' && module.exports) {
    module.exports = Scribe;
  } else {
    global.Scribe = Scribe;
  }
}(window)); 

This removes a fair amount of unnecessary code and means you don't need to include almond.

[benjismith]

For what it's worth, I'd love to use scribe in a node-webkit application, which uses node's native module system, like this:

"use strict";
var Module = (function(){
  var dependency = require("./some/deeply/nested/dependency");
  function blah() {}
  exports.blah = blah; 
})();

I started trying to do this myself, but I got somewhat tangled up in some of the deeply-nested dependencies where scribe plugins return functions that return functions (that also return functions?), and I wasn't entirely sure how to transform the code into a node-like module system without screwing things up. But if others are interested, I can try taking another crack at it.

[mjackson]

@Rich-Harris Thanks for your input!

I tried using amdclean as you suggested, and it looks good for the most part. The only problem is that it doesn't quite do the trick for modules that are already defined using UMD, like the EventEmitter module that scribe depends on.

For example, the EventEmitter module definition looks something like this:

(function () {

  function EventEmitter() {}

  // ...

  // Expose the class either via AMD, CommonJS or the global object
  if (typeof define === 'function' && define.amd) {
    define('event-emitter',[],function () {
      return EventEmitter;
    });
  }
  else if (typeof module === 'object' && module.exports){
    module.exports = EventEmitter;
  }
  else {
    this.EventEmitter = EventEmitter;
  }

}.call(this));

But amdclean transforms it into this:

(function () {

  function EventEmitter() {}

  // ...

  // Expose the class either via AMD, CommonJS or the global object
  if (true) {
    var event_emitter = function () {
      return EventEmitter;
    }();
  }
  else if (typeof module === 'object' && module.exports){
    module.exports = EventEmitter;
  }
  else {
    this.EventEmitter = EventEmitter;
  }

}.call(this));

which effectively conceals the event_emitter variable inside the closure.

@benjismith Is this similar to the problem that you ran into?

@gfranko Since amdclean is specifically meant for AMD modules, I don't suppose it should be responsible for anticipating scenarios where it's already wrapped as UMD, right?

[mjackson]

As a side note, I'm starting to believe that using AMD modules isn't a good idea for code that you intend to ship in multiple different module formats, unless you absolutely need to be able to asynchronously load modules at runtime, which I don't believe scribe does.

@OliverJAsh @theefer Please correct me if I'm wrong.

[gfranko]

@mjijackson You just need to set the transformAMDChecks option to false and it will not transform the UMD logic. By default, it expects to be used for web apps (which is why the transformAMDChecks option is set to true).

If terms of scoping problems for variables, you need to set the globalObject option to true. In the next major release of AMDClean, this will no longer be necessary since variables will all be hoisted correctly.

[mjackson]

@gfranko I see. Thanks for the help!

Unfortunately that option doesn't quite solve my issue in this case. Now, the output looks like this:

(function () {

  function EventEmitter() {}

  // ...

  if (typeof define === 'function' && define.amd) {
    var event_emitter = function () {
      return EventEmitter;
    }();
  } else if (typeof module === 'object' && module.exports) {
    module.exports = EventEmitter;
  } else {
    this.EventEmitter = EventEmitter;
  }

}.call(this));

which still conceals the event_emitter variable inside the UMD wrapper. It's almost like we need a umdclean (which may be difficult to create given the variance in UMD wrapper styles) for the EventEmitter module, and then we can use amdclean on everything else.

[gfranko]

@mjijackson If you don't want the event emitter module to be "cleaned", then you can include it in the ignoreModules option. Like this: ['event-emitter'].

Then none of your UMD logic would be touched.

[mjackson]

@gfranko Thanks for the input. It still doesn't resolve this particular issue, but it's good to know for using amdclean in the future.

[theefer]

I think using ES6 modules would be the preferred option, though we'd need to see if we can still use the es6-module-transpiler to transpile to AMD or CJS in spite of depending on non-ES6 code (e.g. EventEmitter).

[mjackson]

@theefer Agreed. I doubt es6-module-transpiler would mangle non-ES6 code, but can't say for sure.

[samvasko]

So what is the current situation? What is the best way to use it as CommonJS module?

[benjismith]

@mjijackson No, my issues were much more mundane: I'm primarily a Java programmer, so I'm accustomed to seeing a more straightforward import mechanism, one class per file, multiple class files organized into hierarchical packages, etc. I've also done a lot of node.js programming over the past year, and its module import mechanism is conceptually pretty similar to the stuff I'm already familiar with.

But things are a bit less predictable with client-side AMD modules. Some modules are implemented as functions that get immediately called to create a closure, while other modules define a function that's not called until later, creating a closure upon call. The lack of predictable structure provides a lot of implementation flexibility, but it looks a bit foreign to somebody coming from a different dev background.

I'd like to use scribe in a node-webkit project, so I need to use node's require implementation, but it's not always clear to me how to take scribe's closure and expose them as node module exports.

[OliverJAsh]

I want us to move to using ES6 modules internally, however the Plumber operation for Traceur does not currently output source maps, so I can’t use it in production just yet. Nonetheless, there’s no reason we can’t do the work to move to ES6 modules and keep that on a separate branch (without source maps). Hopefully it won’t belong before I give that a go, but if anyone else wants to pick it up, don’t hesitate!

If we use ES6 modules internal, my hope is that we can compile that source code to AMD/CommonJS/UMD-like source code.

Right now I’m wondering if it’s possible – when using ES6 modules – to import a module that uses AMD/CommonJS/UMD, as that will be the case for html-janitor and EventEmitter.

[OliverJAsh]

@benjismith I think you are speaking about the plugins. They are functions that return functions, yes:

var plugin = function () {
  return function (scribe) {
    …
  }
};

To enable a plugin, scribe.use simply expects one parameter that is a function, which will be invoked with the Scribe instance as its first and only argument. (We should have API docs to explain this; sorry. The code is very straightforward.)

So to use my example plugin plugin, you just do scribe.use(plugin()). In CommonJS that could look like:

var plugin = require('plugin');
scribe.use(plugin());

Although we don’t yet support CommonJS, unless you’re doing something funky.

Some plugins need to be configured, in which case we need to use higher-order functions so that you can pass in any options to the plugin function call. We made all the plugins higher-order functions as a normalisation step.

I hope that helps to clear things up for you!

[benjismith]

Thanks Oliver! That's a very clear explanation. I'll take a closer look and see if I can make it work in my setup :)

[janfoeh]

Is there a usable process available at the moment for converting ES6 modules into UMD? I could only find this year-old fork of Squares transpiler, which supposedly isn't always reliable itself.

Having spent some time with Scribes' sources, I am quite eager to migrate my own projects over to it from HalloJS. The only thing holding me back is the pain of dealing with require/AMD/CJS, which does not integrate very well with either the Rails Asset Pipeline, nor the rest of my projects.

So I would be grateful for any solution in the near future that does not involve me having to wrangle in UMD support manually and playing catchup with upstream from then on :)

[theefer]

Do you really need UMD if you have ES6 modules that you can transpile to AMD, CommonJS and globals, and distribute separate dist files for each case?

[OliverJAsh]

You could do ES6 modules => AMD => UMD, it’s just another step. Or what @theefer said.

[OliverJAsh]

Could also be interesting to see how Lodash generates global, AMD, and CommonJS versions. There’s a lot going on in here: https://github.com/lodash/lodash-cli/blob/master/bin/lodash

[janfoeh]

@theefer you're right, I am only really concerned with using Scribe via globals, and using UMD would just be a means to that end - whatever works, works. It simply seemed to be the simplest, least involved approach.

[benjismith]

@theefer and @janfoeh I agree with both of you. Browser globals will also work from within node-webkit, which is fine by me.

[theefer]

Btw I just fixed sourcemaps in the plumber-traceur operation. Still quite experimental, but at least it's not blocking us to try using ES6 anymore!

[OliverJAsh]

This is currently being held by up https://github.com/guardian/scribe/issues/111.

[akrymski]

I'm looking forward to using scribe in our project (post.fm) and wanted to chime in on this issue.

In my humble opinion, keeping it simple is the way to go. I’d take the same approach as Backbone: a single js file (with a UMD definition at the top) and underscore as a required dependency. Reasons:

• Scribe isn’t a huge library, number of lines is the same as Backbone’s if not less (1500 or so)
• Many people already include Underscore.js in their projects (or use lodash as an alternative) and Scribe can include any custom lodash methods missing from underscore API.
• No fancy packaging needed, no dependency on RequireJS/AMD/ES6/you name it
• Plugins can attach themselves to the ‘Scribe’ global (as jQuery plugins do) thus avoiding global pollution #146
• Works as an NPM & CommonJS module for browserify users #77

I’d have a folder with optional plugins in the same repo, with a standalone js file for each plugin (adding a plugin is then as easy as adding a script element).

PS I'm using browserify (CommonJS) and backbone in my project currently. I’d be happy to do this for you, however syncing with upstream would be hard so should be done in one go really.

[OliverJAsh]

We now have CommonJS support via Browserify. See https://github.com/guardian/scribe/pull/175. For an example, check out https://github.com/guardian/scribe/blob/master/examples/cjs.html.

Unless someone can provide a reason to implement support for globals, I’m happy to consider this issue closed.

[janfoeh]

My projects are quite small - low five digit KLOC, hundred-ish files and no more than one to two dozen dependencies. While I have tried JS dependency managers, I have found that they do not provide me with tangible benefits over plain IIFEs and a manually managed namespace, while introducing quite a lot of conceptual overhead and toolchain complexity.

While they certainly are a boon for some workflows and projects, for others like mine they are just process for processes' sake. While I cannot provide hard data, I have a hard time believing that the latter does not describe a significant percentage of use cases out there right now, if not the majority.

Using UMD would make Scribe accessible to them as well. While I respect opinionated projects, I would appreciate it if you would reconsider.

[TooTallNate]

@OliverJAsh @janfoeh I know that you guys are fond of Pluming.js for your dist build, however browserify has an option that does exactly this (that is, does UMD export style): the --standalone option. Perhaps that would make everybody happy.

See this blog post for a more detailed explanation: http://www.forbeslindesay.co.uk/post/46324645400/standalone-browserify-builds

[theefer]

So can you get Scribe via globals by using browserify and --standalone (since https://github.com/guardian/scribe/pull/175 added browserify support)?

[TooTallNate]

Try something like this from the root of the scribe repo:

$ npm install deamdify
$ browserify --global-transform deamdify --standalone Scribe . > build.js

Now build.js is a standalone copy of Scribe with "UMD" syntax, so it works in CommonJS, AMD, and in plain' ol <script> tag imports (will be global as window.Scribe in that case).

I think that final use-case is the one that @janfoeh was desiring.

[OliverJAsh]

So, theoretically, we could create the bundle using browserify instead of RequireJS, and that would leave us with a UMD compatible module? I think this would still be done in the form of a Plumber task (this doesn't really matter).

If it’s that easy, we should do it! :-)

[janfoeh]

@TooTallNate, @OliverJAsh thank you! It looks like this would indeed solve my problems. I'll give it a whirl as soon as possible.

[simonsmith]

Also needing a global version. Using Angular without CommonJS/AMD (quite common I expect, thanks to their take on modules) so the standalone Browserify options sounds great.

[OliverJAsh]

Let’s do this then: https://github.com/guardian/scribe/issues/83#issuecomment-43954522

Anyone want to give it a shot?

[samvasko]

@OliverJAsh I will give it a try :package:

[janfoeh]

Well, I finally got back to my Scribe-related project and managed to spend a couple hours on this. It was a bit of a pain, but I now have a converted UMD builds of Scribe and all official plugins i could find.

@TooTallNate, @OliverJAsh would you mind if I put up a scribe-umd repo as a quick fix in the interim? Judging by this thread this might be of use to some.

[theefer]

@janfoeh How are they built?

[callum]

I've been fiddling a little bit to see how Scribe might work with ES6 modules. Here's a untested first pass at converting the module syntax:

https://github.com/callum/scribe/commit/2357d5b696a2b365b364b328dc63dc68c49c914d

I wanted to experiment with various aspects of ES6 modules on a smaller scale, so I've scrappily implemented es6-module-transpiler on my own fork of scribe-common:

https://github.com/callum/scribe-common

I'm coming at this from a CommonJS angle, but in its current state, if you installed the module with npm, you could:

var element = require("scribe-common/dist/cjs/element");

element.isBlockElement();

As @OliverJAsh points out above, I'm not sure how dependency management works; whether you're able to use npm for libraries such as Lo-Dash or other. (https://github.com/lodash/lodash-es6).

If scribe-common is written in ES6 modules, how does scribe then consume that module? Via npm, Bower, jspm? I don't know.

Hope this is somewhat helpful.

[OliverJAsh]

I believe you can use jspm to consume a variety of module formats. We should give it a go! On 13 Sep 2014 19:52, "callum" notifications@github.com wrote:

I've been fiddling a little bit to see how Scribe might work with ES6 modules. Here's a untested first pass at converting the module syntax:

callum@2357d5b https://github.com/callum/scribe/commit/2357d5b696a2b365b364b328dc63dc68c49c914d

I wanted to experiment with various aspects of ES6 modules on a smaller scale, so I've scrappily implemented es6-module-transpiler on my own fork of scribe-common:

https://github.com/callum/scribe-common

I'm coming at this from a CommonJS angle, but in its current state, if you installed the module with NPM, you could:

var element = require("scribe-common/dist/cjs/element"); element.isBlockElement();

As @OliverJAsh https://github.com/OliverJAsh points out above, I'm not sure how dependency management works; whether you're able to use NPM for libraries such as Lo-Dash or other. (https://github.com/lodash/lodash-es6 ).

If scribe-common is written in ES6 modules, how does scribe then consume that module? Via NPM, Bower, JSPM? I don't know.

Hope this is somewhat helpful.

— Reply to this email directly or view it on GitHub https://github.com/guardian/scribe/issues/83#issuecomment-55501373.

[callum]

Trying to get my head around a stack that uses jspm and is able to build to AMD, CJS and globals. @OliverJAsh what ideas do you have around this?

[callum]

I've got a working example of a jspm/SystemJS implementation with ES6 modules working in this commit https://github.com/callum/scribe/commit/978f8eeab157ebf43c0a8215bec1075ec27def69. It's just the regular example sans toolbar for now, but it works brilliantly.

[callum]

External dependencies are the pain point at the moment. I tried es6-module-transpiler on the above and it chokes at things like this:

import flatten from 'lodash-node/modern/arrays/flatten';

Only jspm/SystemJS cleverness knows how to resolve that because of its internal configuration. (https://github.com/callum/scribe/blob/es6-modules/examples/jspm/config.js#L11-L12). I can't think of a way to normalise imports across AMD, CJS and globals. Let alone how you would manage versioning between them.

There is an NPM resolver for es6-module-transpiler, but it requires that libraries are authored in ES6 module syntax. (https://github.com/caridy/es6-module-transpiler-npm-resolver).

One option would be to factor out external dependencies, but that's probably a narrow minded solution. https://github.com/jakearchibald/es6-promise doesn't have any, for example.

There's also the option of bundling external dependencies, either using jspm or Browserify bundling. Not ideal, but it would work.

@guybedford you mention tooling for plugins at the end of your JSConf talk, is that relevant to this problem? Would appreciate your input.

[guybedford]

One idea I've been trying to push for a while is the idea of a module bundle. That is, you build your internal modules into a bundle, but still create a UMD file that has external dependencies.

I've been suggesting this for both the Traceur and ES6 Module Transpiler projects, but haven't been able to convince anyone yet that it is worth doing. Relevant posts are at - https://github.com/esnext/es6-module-transpiler/issues/140#issuecomment-49571317, https://github.com/google/traceur-compiler/issues/844#issuecomment-45453350.

Let me know if you think something like that sounds like a possible direction here. Very keen to see work done along these lines myself.

[callum]

Have you any thoughts on how those external dependencies are referenced? I've done something similar with Browserify in the past, using https://github.com/pluma/literalify. The idea being that you bundle your internal modules as you say, and swap your external require calls with window globals.

The ideal solution is something that transpiles to different formats, referencing external dependencies in a manor that each format understands with regard to package managers (npm for Browserify, Bower for RequireJS or other).

I don't know how well bundling would suit that requirement.

As a side note, I've just seen this: https://github.com/polyfills/es6-module-crosspiler

[guybedford]

My ideal workflow for this would be the following:

1. Write in ES6:

   main.js

   import { f } from './dep';
   import _ from 'lodash-es6';
   f(_);
   export function apiFunc() {}

   dep.js

   export function f() {...}

2. Compile into a "module bundle" that is still ES6:

   bundle.js

   import _ from 'lodash-es6';
   $$dep$f(_);
   function $$dep$f() { ... }
   export function apiFunc() {}

   note that we've inlined the private modules (./), while leaving in the public third-party modules as dependencies. This is what I mean by module bundle. The square ES6 module transpiler already does the private concat process.

3. Now, I have one ES6 module, that I can compile for the environment I want:
   • For npm & Browserify I can compile this ES6 module into CommonJS, rewriting lodash to a suitable name for npm perhaps (simple mapping).
   • For AMD & Globals / a UMD pattern I can compile this ES6 module into the good old UMD monster (https://github.com/umdjs/umd/blob/master/amdWebGlobal.js)

etc.

Let me know if that makes sense? These workflows are where we should be working towards, but not where we are today, so it is great to be discussing this stuff.

[callum]

I like the idea. Your third point is the tricky part to my understanding, because as an author, you want to ensure that consumers are using consistent versions for dependencies. Sort of like "we don't know if you're using CommonJS or AMD, but you should be using Lo-Dash 2.4.1". I'm not sure how you propose the rewriting/mapping would work, but I'm interested to know more about that.