Babel modules roadmap

[julik]

Currently Babel is switched to replace ES6 module declarations with CommonJS requires, whereas Sprockets does not provide a runtime to handle those requires.

Is there some roadmap as to how modules are going to be wired into Sprockets, or does it have to be handled by some module you have to BYO?

[metaskills]

Kind of curious about this myself. I've been looking into sprockets-es6 and I have no idea how to bridge the module declarations with my work.

[tf]

Also really interested.

[andyl]

I'm also interested in this.

[schneems]

Let's pretend I didn't understand half of the words you used in this issue. Also let's pretend that the original author of sprockets isn't here anymore and i'll likely have to work with whatever solution we decide to go with forwards. Let's be patient and go slowly.

What's babel?

Babel is a compiler for writing next generation JavaScript.

Okay. That seems like a thing we should want to support.

What is a common js require?

Looks like it's a way to import explicit functions from http://wiki.commonjs.org/wiki/Modules/1.1

What's a ES6 module declarations?

Looks like it's a way of importing module like behavior http://wiki.ecmascript.org/doku.php?id=harmony:modules_examples.

We already have a babel processor. It looks like it currently supports es6

  def test_compile_es6_features_to_es5
    input = {
      content_type: 'application/ecmascript-6',
      data: "const square = (n) => n * n",
      metadata: {},
      load_path: File.expand_path("../fixtures", __FILE__),
      filename: File.expand_path("../fixtures/mod.es6", __FILE__),
      cache: Sprockets::Cache.new
    }

    assert js = Sprockets::BabelProcessor.call(input)[:data]
    assert_match(/var square/, js)
    assert_match(/function/, js)
  end

The test suite uses babel-transpiler from https://github.com/babel/ruby-babel-transpiler. How does sprockets know to convert es6 files?

  # Babel, TheFuture™ is now
  require 'sprockets/babel_processor'
  register_mime_type 'application/ecmascript-6', extensions: ['.es6'], charset: :unicode
  register_transformer 'application/ecmascript-6', 'application/javascript', BabelProcessor
  register_preprocessor 'application/ecmascript-6', DirectiveProcessor.new(comments: ["//", ["/*", "*/"]])

It looks like sprockets gets its ES6 support directly from babel. How are commonJS requires implemented? Is it a different file extension & mime type or is it some other layer on top of es6? If it's a file extension then once the babel-transpiler supports commonjs then we can register that in the same way. There will probably be some plumbing needed in the processor itself.

[andyl]

We would like to use CommonJS export/require with CoffeeScript.

Currently we're getting the job done with the Browserify-Rails gem. It works, but is slow as heck, and the setup is complex. IMHO it would be a big improvement for Rails if CommonJS was incorporated directly into sprockets.

I can't imagine building any JavaScript front-end these days without CommonJS - I believe it should be part of Sprockets.

[tf]

Babel does not implement common js requires itself. All it does it rewrite ES6 imports (i.e. import foo from 'foo') to common js requires (i.e. var foo = require('foo')). There are more examples in the docs. Tools like webpack include a minimal runtime in the final js output which defines a require function, and concatenate the contents of all js files inside an array to allow lookup via this require function.

[julik]

Well that is the thing - Sprockets does not provide a loader, and it seems that this is something that pretty much anyone would expect in a modern setup. And that loader should work both with precompiled assets and expando assets. I know it is a tall order, but...

[schneems]

Sprockets does not provide a loader,

Can you give me more info, what exactly is a "loader", this term has a sprockets specific context

  # The loader phase takes a asset URI location and returns a constructed Asset
  # object.

Something that could help an implementer could be a PR with a failing test. Here's an example of an es6 test https://github.com/rails/sprockets/blob/7b913165d54bef952b8f83509e731a90987321bb/test/test_environment.rb#L268-L272

You would need to add your own asset.

[elia]

@opal would benefit from a loader too, currently we monkeypatch javascript_invlude_tag to append the final load statement that will bootstrap the app.

I looked into that in the past and I suspect sprockets pipelines can be used for something like that.

https://github.com/opal/opal-rails/blob/master/app/helpers/opal_helper.rb

[lucasmazza]

@scheems a Loader would be a 3rd party JavaScript library responsible for polyfilling the import/require features from ES6 that aren't currently supported by browsers. The Babel transformer can transpile these keywords to plain JavaScript code but the application still needs to provide the mechanism to load the JavaScript files as expected through loader like almond.js for AMD modules or System.js.

Folks using sprockets 3 and the sprockets-es6 plugin can reconfigure the ES6 transformer to use a specific module format supported by Babel on their apps with something like the following:

Rails.application.config.assets.configure do |env|
  # If you are using Sprockets 4, use `Sprockets::BabelProcessor`
  # instead of `Sprockets::ES6`.
  es6amd = Sprockets::ES6.new('modules' => 'amd', 'moduleIds' => true)
  # Replace the default transformer to transpile each `.es6` file with `define`
  # and `require` from the AMD spec.
  # Just be sure to add `almond.js` to the application and
  # require it before requiring other assets on `application.js`
  env.register_transformer 'text/ecmascript-6', 'application/javascript', es6amd
end

Ideally, Sprockets could choose a module format by default (like system), and ship with a Module Loader polyfill for the same format so developers can use ES6 modules without the extra steps of recofinguring the transformer or adding these 3rd party libraries manually, but that could be something done by an extra plugin rather than live in this codebase.

Hope this helps to shed some light on some of the terms from the ES6 world here :smile: I'm using this setup on my current project and I believe that some other users of sprockets-es6 might have done a similar setup on their apps.

[elia]

related: #52

[jcoyne]

@lucasmazza Thanks so much. Exactly what I needed! However the syntax is thus:

Rails.application.config.assets.configure do |env|
  es6amd = Sprockets::BabelProcessor.new('modules' => 'amd', 'moduleIds' => true)
  # Replace the default transformer to transpile each `.es6` file with `define`
  # and `require` from the AMD spec.
  # Just be sure to add `almond.js` to the application and
  # require it before requiring other assets on `application.js`
  env.register_transformer 'application/ecmascript-6', 'application/javascript', es6amd
end

[pftg]

@lucasmazza :+1:

[tf]

Any update here?

Discourse seems to be using some vendored variant of ember-cli/loader.js to be able to transpile to amd modules.

I really think Sprockets should provide something like this out of the box. Otherwise one will always have to install additional components to obtain a real ES2015 environment.

[tf]

While looking at maccman/sprockets-commonjs, I realized there are two more aspects that would need to be addressed:

• How can we prevent having to also //= require files to ensure they end up in the bundle?
• Webpack is really well integrated with npm to import modules from third party packages. Is there anything sprockets might be able to offer here (via Rails Assets or custom -rails asset gems)?

[rescribet]

@tf My first thought of preventing double requires would be to let sprockets be aware of the package.json convention, aka adding npm as a source for JavaScript dependencies.

[tf]

I'm not sure I agree. From my perspective, one of the main advantages of the asset pipeline comes from packaging up assets along with other Rails app components in gems and not having to deal with two package managers. If I was to use npm anyway, I do not see much value over switching to the js ecosystem completely and using webpack directly.

[rescribet]

Yeah, I think assets from other parts of Rails should definitely be accessible. But having to repackage every JS lib in a gem, or completely switching just to be able to use npm packages is also quite cumbersome.

I was thinking more of a sprockets implemented require('foo') that enables people to replace all //= require foo calls. It would check the assets directory structure for bundled assets, and fall back on npm. Making the jQuery gem for example, unnecessary but not incompatible. Though I'm not sure whether such a thing would be possible.

For anyone interested, a lot of discussion has been held over at react-rails about the require() topic, and spawned a side project dedicated to including webpack (w/ React) into Rails.

[mockdeep]

Would love to see node-style requires in sprockets proper. We're using browserify-rails right now and slowly moving our sprockets requires over to node style with module.exports. So far we've been pretty happy with this approach (aside from the compile times...) and it seems like functionality that would make sense to have in sprockets itself. Would still need to figure out how to handle things like .jsx transforms, though.

[jcoyne]

Wondering if it makes sense to have a dependency on babel-transpliler give the maintainer is not upgrading it to the latest version of Babel (see https://github.com/babel/ruby-babel-transpiler/issues/288). I haven't yet looked at this library, but it seems like it may be a promising alternative: https://github.com/fnando/babel-schmooze-sprockets

[guilleiguaran]

I worked with @Liceth updating sprockets-commonjs to sprockets 4.x, the processor is this: https://github.com/Liceth/sprockets-commonjs/blob/sprockets-4/lib/sprockets/commonjs.rb,

The modules can be used like this:

/* foobar.module.es6 */
function foo() { return 'foo'; }

function bar() { return 'bar'; }

export { foo, bar };

/* main.es6 */
//= require foobar.module

import * as lib from 'foobar.module';
console.log(lib.foo());
console.log(lib.bar());

I think this can be an acceptable default solution for this since it works without requiring Node.js/NPM in the system like other alternatives.

[rmacklin]

For anyone who ends up on this issue and wants to use something like the aforementioned babel-schmooze-sprockets but in an application on sprockets 3 (babel-schmooze-sprockets only works with the sprockets 4 beta), I'll mention the similar sprockets plugin I created: https://github.com/rmacklin/sprockets-bumble_d. It supports sprockets 3 and has some extra stuff geared specifically toward migrating large codebases to ES6 modules (since that was my use case at work) but it also works for general purpose babel transpilation.

[wnewbery]

There been any more progress / documentation on this (or similar, e.g. where does TypeScript stand)?

I looked at https://github.com/rmacklin/sprockets-bumble_d linked above, but seems, and if I understand correctly that wont tell sprockets the proper dependency order even between ES6 files, and similar for the other tools?

Ive also looked at the CommonJS libs/loaders to make define/require work before, but honestly when I start getting hundreds of files, I like how Sprockets in development config can put them all as a series of separate script tags with digests and no extra overhead (so I get 99% of "from memory cache" and not several seconds of "304 Not Modified" as the loader dynamically gets files with the non-digest names).

As I understand from ES6 modules/imports, etc., dynamic loading is not supported there, so all the imports can be determined statically? So something like this should be possible and performant?

// page/my_class.es6   (allow page/my_class.es6.erb as well for image_url etc.)
import * as lib from 'lib/bar'
...
export MyClass;

Use bable (or any similar tool, maybe seperate gems) to get CommonJS ES5 (or even better, the exports and requires as a separate object/output).

var lib = require('lib/bar');
...
module.exports = MyClass;

Then convert this for Sprockets (unless its possible to directly tell sprockets the requires list in the API and skip the "//=" stuff)

//= require foo/bar
(function() {
  var lib = window.$modules.foo$bar;
  ...
  window.$modules.page$my_class = MyClass;
})();

So in the HTML simply:

<!--As before: devlepment-->
<script src="/assets/lib/bar.self-00112233.js?body=1"></script>
<script src="/assets/page/my_class.self-00112233.js?body=1"></script>
<!--As before: production / precompiled-->
<script src="https://cdn.my-asset-host.com/assets/application-00112233.js"></script>

[schneems]

Have you tried setting this up with webpacker & rails 5.1 ?

[schneems]

I'm pruning issues for Sprockets 4 release. We can still discuss here, but I want to keep the tracker as primarily actionable bugs. I'm going to close this for now.

[lancecarlson]

@schneems Looking at the es6 support section of https://github.com/rails/sprockets/blob/master/UPGRADING.md, does that mean es6 exports/imports don't work yet? When I try to import and export modules, it seems like it's attempting to require('the_module') and I don't see require defined anywhere. Is there a right way to do this? I guess this is happening because the babel-transpiler is just doing whatever it deems reasonable in this case.