Meteor 1.3 *early beta* (ES2015 modules, better file load order control, and more)

[avital]

We're starting the process towards Meteor 1.3, with module support, better NPM integration, better file load order configuration, and more.

Meteor 1.3 gives apps full ES6 (ES2015) modules support and better NPM support. Apps and packages can now load NPM modules that are installed via npm install on client and on server. Amongst other benefits, modules let you control file load order in much more precise way than naming your files in special ways.

To get feedback early, we decided to go ahead and release a beta release that doesn't yet have all of the features ready, but does have almost complete support for modules.

Try the release by running meteor update --release 1.3-modules-beta.8 in your app directory.

@benjamn wrote a great document explaining how to use modules in Meteor.

Please start trying this release out! You can start experimenting with switching your apps or packages to use the new module features. Let us know what works well or doesn't.

Take a look at the list of changes in Meteor 1.3, and the remaining tasks.

[guilhermedecampo]

Thanks for that! Looking forward to test modules! :+1:

[fakenickels]

Great work! Really excited about it, really cool that we can start getting rid of creating wrapper packages for npm stuff. :+1:

[vjau]

This seems to work great, thank you mdg.

[nicolasparada]

Nice.

[GeoffreyBooth]

Before you release the new module system, please make it compatible with CoffeeScript. In CoffeeScript, import and export are reserved words, so to write code that uses them requires using backticks, which looks really awkward. Ember.js apps built with Ember CLI have this same issue; such code looks like this:

`import Ember from 'ember'`

DummyController = Ember.Controller.extend()

`export default DummyController`

A simple solution is to follow the example of the ember-cli-coffee6 package, that does some simple string parsing to fix Coffee-style import/export statements before the regular Coffee compiler evaluates them.

[ciwolsey]

@avital Is import/export supposed to work from jsx? I've added the react meteor package but when using import/export from within jsx I get an error about usage of a reserved keyword.

[ccorcos]

@benjamn How do you feel about relative vs absolute file imports? It frustrates me when I have to write something like import Something from '../../lib/some/thing'. If I move the lib/some/thing file, then its going to be a huge pain in the ass chasing down all the relative paths to it and fixing them. And if I move the current file, the link breaks as well. In my webpack configuration, I resolve the root path to the root of my project so I never have .. in my imports. If I move a file, I can findAllAndReplace that absolute path with the new path trivially.

I saw your really nice presentation on ES6 modules and I'm curious what you think about this. Will this be supported in Meteor? How would this sort of thing work when publishing packages?

[benjamn]

@ccorcos there's been some debate about whether we should support module identifiers starting with a /, and your comment makes a strong case for "yes." The good news is that support for absolute paths is already implemented, so the debate is actually about whether it should be disabled, or remain enabled.

Now for the caveats…

In Node, an absolute module identifier corresponds to an absolute path from the root of the file system. Because we definitely don't want to include path prefixes like /Users/ben/dev/... in the client-side JS bundle, absolute module identifiers clearly need to work a bit differently from Node.

Instead, absolute module identifiers refer to the root of the virtual directory structure installed by meteorInstall. So, in your example, you could do

import Something from '/app/lib/some/thing';

if you wanted to avoid ../ paths. Though this behavior is admittedly not obvious, it's probably something we could commit to and document.

The situation with packages is similar, but somewhat more complicated-looking, because Meteor packages get installed with meteorInstall the same way npm packages are installed:

import PackageThing from '/node_modules/some-meteor-package/some/thing';

On the other hand, because of this structure, you probably never need to use an absolute module identifier to refer to a package, because you can just treat it as a top-level npm package:

import PackageThing from 'some-meteor-package/some/thing';

In other words, the ../../relative/path/problem is mostly something for apps to worry about, and /app/some/thing should help you avoid ../ paths.

[ccorcos]

@benjamn thanks for the explanation. Feeding off your presentation, I'm curious about this from a more generic sense -- how can this be compatible within Meteor and the rest of the NPM / Javascript community.

For example, within my webpack config, I simply tell it where it resolve absolute paths from:

      resolve: {
        root: [
          path.resolve(__dirname),
        ],
        modulesDirectories: [
          'node_modules'
        ]
      }

Then I don't even have to use the leading slash (which is ambiguous with the root of the filesystem as discussed):

import Something from 'app/lib/some/thing';

And by specifying the modulesDirectories, I can get away with this as well:

import PackageThing from 'some-meteor-package/some/thing';

The big caveat left in both the Meteor and the large Javascript community is that if I produce a package that uses these absolute paths (relative to the package's root), and then I start using it within another project, the packaging system needs to know what the package absolute paths are relative to the package root and not the project root. I believe this is still not a solved problem within the Webpack community.

@benjamn I'd also like to hear your thoughts about webpack's approach to bundling static assets (CSS, SCSS, Images, Fonts, SVGs, etc) within Javascript.

[raix]

:+1:

[TankOs]

Won't this feature also make it possible (or make it easier) to introduce bundlers, like Webpack? I'm especially concerned about build times, which would greatly benefit from tools like Webpack.

[trusktr]

@GeoffreyBooth Why are import and export reserved word in coffeescript? Is this a relic of times before ES2015 module syntax became a thing? Anyway, you can continue to use coffeescript, just add the coffeescript package. This works because compilers (provided by packages like coffeescript, ecmascript, less, etc) all do their transpilation work first, and then the new modules package handles the module aspects of the resulting JavaScript files at the end. And since coffeescript is an official package, it will have support for compiling to CommonJS format no doubt. If the import/export syntax isn't in that package yet, then just use the CommonJS syntax (something like ember = require 'ember').

[trusktr]

@benjamn

import Something from '/app/lib/some/thing';

if you wanted to avoid ../ paths. Though this behavior is admittedly not obvious, it's probably something we could commit to and document.

The situation with packages is similar, but somewhat more complicated-looking, because Meteor packages get installed with meteorInstall the same way npm packages are installed:

import PackageThing from '/node_modules/some-meteor-package/some/thing';

On the other hand, because of this structure, you probably never need to use an absolute module identifier to refer to a package, because you can just treat it as a top-level npm package:

import PackageThing from 'some-meteor-package/some/thing';

I don't think this is a the best way to go, for some reasons:

1. Meteor package names can now collide with NPM package names. Perhaps require a colon in the name in order to differentiate, and meteor packages would always be prefixed with meteor: in this case.

   import PackageThing from 'some:package/some/thing'
   // or
   import PackageThing from 'meteor:package/some/thing'

   This isn't part of NPM's spec, but it seems something like this is needed in order to have zero chance of collision.

2. The meaning of an absolute path is not clear. A better bet would be for / to mean the app root in app code, and the package root in package code. Instead of writing

   import Something from '/app/lib/some/thing'

   we can write

   import Something from '/lib/some/thing'

   which is more intuitive (people won't go off wondering what else is in the root if it isn't the app itself).

Additionally, some things that might help:

1. Treat the app as a package. We would be able to do

   import Something from 'app/lib/some/thing'

   without the leading slash.

2. Allow a user configuration somewhere (perhaps modules.json) that lets us map names to paths (including paths that start with module names). For example

   {
    'someLib': '/lib/some/thing',
    'otherLib': 'npm-package/some/thing',
    'anotherLib': 'meteor:package/some/thing'
   }

   Now we can do

   import Something from 'someLib'
   import otherThing from 'otherLib'
   import anotherThing from 'anotherLib'

   which would be useful for things we import repeatedly (and is simliar to what @ccorcos describes Webpack having, and what Browserify, JSPM, and etc have).

[trusktr]

@ccorcos Also brought up a good point about the absolute paths and the meaning of it changing when using libraries in different environments. I think that the use of absolute paths should generally be discouraged in Meteor's case.

I'd also like to hear your thoughts about webpack's approach to bundling static assets (CSS, SCSS, Images, Fonts, SVGs, etc) within Javascript.

This is what packages the provide compilers will continue to be used for. For example, the coffeescript package will continue to compile coffeescript to javascript before Meteor 1.3's modules package does it's thing. So, these packages in the Meteor community (like coffeescript) are essentially equivalent to what loaders are in Webpack.

@TankOs

Won't this feature also make it possible (or make it easier) to introduce bundlers, like Webpack?

Nope, not at all. We can do that already, right now, without Meteor 1.3 (for example, my rocket:module package uses Webpack to let you write ES2015 modules). Webpack compiles code into a a compiled form that you run in the browser, which is what Meteor 1.3 is now doing too. You would either use Webpack (directly, or with packages like rocket:module), or use Meteor 1.3's modules package, but not both because both are tackling the same problem.

[TankOs]

@trusktr I see, interesting. I'll have a look at your implementation for getting some ideas. Thanks! :)

[ccorcos]

@trusktr

Treat the app as a package. We would be able to do import Something from 'app/lib/some/thing' without the leading slash.

I like this approach. You'd effectively need to symlink your project into node_modules, but then you can reference files with absolute paths just like any other package! Using a symlink is sort of a hack, but work would solve all of these absolute path issues at once :+1:

So, these packages in the Meteor community (like coffeescript) are essentially equivalent to what loaders are in Webpack.

I guess your right, actually. But then we have a similar problem as before where we have to create all these package shims. Theres a huge selection of webpack-loaders that are really nice to use:

1. Suppose you share a bunch of code between 5 different applications. Each component can require('components/styles/this_component.css') in JS for the CSS related to that component. Thus, your components are more modular.
2. The react-svg-loader is pretty cool and lets you inline SVG into a react component so you can style it with CSS.
3. The file-loader, url-loader, image-loader let you minify images, inline small fonts and images, and revision larger images, replacing with a url instead of inline.
4. There are all kinds of advanced techniques for gaining web performance like showing a small blurred inline photo while the larger one is loaded asynchronously. And the people building the tools for this are building them on top of webpack.

[trusktr]

You'd effectively need to symlink your project into node_modules

You wouldn't have to, the modules package would handle that behind the scenes.

I agree, there's tons we can do with Webpack that we won't immediately be able to do with Meteor 1.3. One thing (as you allude to) is that Webpack laoders don't just transform code, they also specify how to write import/require statements. So, even if Meteor 1.3 has compiler packages, there would still need to be an something additional implemented to let Meteor 1.3's modules package know how to handle certain imports (SVG files for example).

[trusktr]

Still, the name app conflicts with the app package on NPM, but it doesn't look like anyone is using that and it hasn't been updated for 4 years. Maybe meteor:app could represent the app, so

import Something from 'meteor:app/lib/some/thing'

?

[ccorcos]

What if it were just the name of your package?

[trusktr]

Then that's what it would be! ;]

[GeoffreyBooth]

@trusktr, import and export are reserved words in CoffeeScript because they were reserved words in ES5, per this thread where I and others have pleaded with CoffeeScript’s maintainers to add idiomatic support for ES2015 modules. Unfortunately it doesn’t appear that the other CoffeeScript maintainers have any interest in addressing this issue right now.

I’m glad that ES2015 is here and catches up to CoffeeScript with regards to the features that tempted many developers to CoffeeScript years ago. Many of those developers have switched back to ES2015, and more power to them; I’ve been using CoffeeScript for years primarily for the easier-to-read syntax, for the breath of fresh whitespace I get from fewer braces and parentheses and semicolons, and I don’t intend to stop coding in CoffeeScript just because JavaScript no longer sucks so much.

So please give us CoffeeScript diehards a way to use ES2015 modules in Meteor. Backticking ES2015 code in JavaScript is a pretty shitty solution, ugly and prone to error because you’re asking people to mix ES2015 and CoffeeScript in the same file. I think ember-cli-coffees6’s approach, letting me write import and export as idiomatic CoffeeScript that it fixes before the CoffeeScript compiler sees it, is probably the easiest “good” solution that you can do without hacking the CoffeeScript compiler. Better still would be if I could just use CommonJS require statements and module.exports like Node or Browserify, and Meteor pulls the packages together the same as if I had used import; at least the CommonJS approach would mean that only one transpiler is required (coffeescript) and not three (the Meteor version of ember-cli-coffees6, then coffeescript, then ecmascript). But I can see that supporting both CommonJS syntax and ES2015 syntax might be too much of a burden, so if that’s the case please implement something like ember-cli-coffees6.

Or if anyone knows @jashkenas or @michaelficarra or @satyr or @lydell or @alubbe or any of the other CoffeeScript maintainers, please convince them to stop ignoring this issue.

[lydell]

Just to clear some confusion regarding CoffeeScript:

• That import and export are reserved words in CoffeeScript is not relevant here (except that CoffeeScript could add support for them in a backwards-compatible way, which is good). CoffeeScript's parser simply does not support the export and import syntax, and the compiler is not able to emit such code (yet?).
• All of the pinged people in the above comment, such as me, might not necessarily:
  • consider themselves maintainers (but rather, for example, people who have managed to get several PRs merged)
  • share the disinterest in adding export and import syntax to CoffeeScript
• Basically, jaskenas is the one who decides what gets into the/his language.

(Also, I don't see how using backticks is prone to error (but I agree that it is ugly):

`import {a} from 'a'`

b = ->
  a()

`export b`

The code in backticks are just "static declarations".)

Unsubscribing.

[alubbe]

Personally, I believe CoffeeScript should be a whitespace-based improvement of JavaScript, which should now include the finished ES6 syntax incl. import/export. That's why I built and lobbied for generator support - but as @lydell mentions, it's up to @jashkenas

If you really want to see it in CoffeeScript, I'd suggest to build it yourself and open a PR.

[prodogs]

Anyone else having trouble with 3rd party libraries. created a shell (Meteor create) runs fine then added a compatibility directory and inserted go-debug.js and it hangs building

[ccorcos]

What are the entry point on the client and the server?

[prodogs]

Didn't define them simply created new project and dropped go into compatability directory

[JeremyBYU]

Got this error recently. I emptied my meteor/local directory except the database and then reran meteor. Havent figured out how to fix it yet.

ReferenceError: dirPath is not defined
    at ImportScanner._tryToResolveImportedPath (/tools/isobuild/import-scanner.js:273:27)
    at /tools/isobuild/import-scanner.js:96:36
    at Array.forEach (native)
    at Function._.each._.forEach (/Users/natty/.meteor/packages/meteor-tool/.1.1.11-modules.0.1mkz1hj++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/lib/node_modules/underscore/underscore.js:79:11)
    at ImportScanner._scanDeps (/tools/isobuild/import-scanner.js:95:5)
    at /tools/isobuild/import-scanner.js:68:24
    at Array.forEach (native)
    at ImportScanner.getOutputFiles (/tools/isobuild/import-scanner.js:66:22)
    at Object.fullLink (/tools/isobuild/linker.js:969:8)
    at /tools/isobuild/compiler-plugin.js:686:28
    at /tools/utils/buildmessage.js:359:18
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:352:34
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:350:23
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at Object.enterJob (/tools/utils/buildmessage.js:324:26)
    at PackageSourceBatch._linkJS (/tools/isobuild/compiler-plugin.js:685:18)
    at PackageSourceBatch.getResources (/tools/isobuild/compiler-plugin.js:586:48)
    at /tools/isobuild/bundler.js:753:37
    at Array.forEach (native)
    at ClientTarget._emitResources (/tools/isobuild/bundler.js:736:19)
    at /tools/isobuild/bundler.js:515:12
    at /tools/utils/buildmessage.js:359:18
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:352:34
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:350:23
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at Object.enterJob (/tools/utils/buildmessage.js:324:26)
    at ClientTarget.make (/tools/isobuild/bundler.js:506:18)
    at /tools/isobuild/bundler.js:2199:14
    at /tools/isobuild/bundler.js:2288:20
    at Array.forEach (native)
    at Function._.each._.forEach (/Users/natty/.meteor/packages/meteor-tool/.1.1.11-modules.0.1mkz1hj++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/lib/node_modules/underscore/underscore.js:79:11)
    at /tools/isobuild/bundler.js:2287:7
    at /tools/utils/buildmessage.js:271:13
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:264:29
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:262:18
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:253:23
    at [object Object]._.extend.withValue (/tools/utils/fiber-helpers.js:89:14)
    at Object.capture (/tools/utils/buildmessage.js:252:19)
    at Object.exports.bundle (/tools/isobuild/bundler.js:2180:31)
    at /tools/runners/run-app.js:585:36
    at time (/tools/tool-env/profile.js:238:28)
    at Function.run (/tools/tool-env/profile.js:391:12)
    at bundleApp (/tools/runners/run-app.js:575:34)
    at [object Object]._.extend._runOnce (/tools/runners/run-app.js:628:35)
    at [object Object]._.extend._fiber (/tools/runners/run-app.js:880:28)
    at /tools/runners/run-app.js:402:12

Was wondering if something changed with the beta release that broke it. Or was it just me?

[ccorcos]

Didn't define them simply created new project and dropped go into compatability directory

Hmm. So the entry point is effectively deduced?

@JeremyBYU I also get an error just doing import React from 'react'

[JeremyBYU]

Okay I think I narrowed it down to an issue with the package.json I use to install my node modules. I think this is the offending text:

  "dependencies": {
    "formsy-react": "git+http://github.com/christianalfoni/formsy-react#4931256"
  },

It seems if I specify a git url and commit and not a version number that the 1.3 module system goes haywire. Should be simple to check. Create a new project, add a package.json with a dependency to a git commit, npm install, then run meteor. You should get a similar error. Be sure that the the meteor/local folder is empty before running meteor, if its not, you probably wont get the updated package.

[benjamn]

@JeremyBYU that's a bug that will be fixed in the next beta release

@avital can we do that today?

[prodogs]

awesome - will check again when available to see if things have changed.

i am assuming that with 1.3 you don't have to set an explicit entry point other than defining routes.

[JeremyBYU]

Oh awesome! Thanks you guys.

[ccorcos]

Ok. Maybe I'm entirely missing something, but I can't get this to work. Here's a simple application that just imports React on the client:

meteor create chatroom
cd chatroom
meteor update --release 1.3-modules-beta.0
rm chatroom*
mkdir client
cd client/
touch main.js
echo "import React from 'react'" >> main.js
cd ..
npm init
npm install --save react
meteor

And I run into this error:

Uncaught ReferenceError: process is not defined

Are there any example apps I can take a look at?

[JeremyBYU]

Hey @ccorcos , Heres a small sample app that I made LINK. Check out the readme. It should answer your question about react.

Thanks

[avital]

@ciwolsey Filed at https://github.com/meteor/meteor/issues/5827. We haven't yet updated the jsx package. It might just get unified with the ecmascript package.

[ccorcos]

@avital it seems alike a meteor.config.js file might be in order -- then we could set the specific babel plugins we want -- stage 2, jsx, etc.

[mitar]

+1 for support with CoffeeScript.

[avital]

Regarding CoffeeScript, is the request simply "let us use import/export with using backticks (`)?"

[benjamn]

New release!

To update to the latest beta release, run the following command in your meteor project:

meteor update --release 1.3-modules-beta.1

Changes:

• Fixed typo where self was undefined, reported by @lampe and @jiku: #5812, #5799, fb73d9e3704556018310b7c873b4e7a95fac807c
• Made import and export work in .jsx files, though you'll want to use ecmascript instead of jsx now: #5827, 86b743f96346b66a7b79586823bbb92eeed8a83d
• The jquery package once again defines $ and jQuery globally, since some legacy code depends on that behavior: #5819, f241cbe44fb604396d39ef7839c940e727412569
• Fixed a bug that prevented loading directories as modules: fb73d9e3704556018310b7c873b4e7a95fac807c
• Now possible to import or require files with a .json file extension: #5810, 32fcced142eb6c03746333a29bc1bfb85040584e
• process.env.NODE_ENV is now defined as either "development" or "production", which makes some NPM packages (e.g. react) happier: dbebcd2d904c725b2634b6c26c5c1f29a79ecb25

[laurentpayot]

With this new beta (1.3-modules-beta.1) I get:

=> Errors prevented startup:                  

   While determining active plugins:
   error: conflict: two packages included in the app (jsx and ecmascript) are both trying to handle *.jsx

jsx package is automatically added when removed, and removing ecmascript leads to build errors as expected...

[mitar]

Regarding CoffeeScript, is the request simply "let us use import/export with using backticks (`)?"

I think what I would like to see is a non-backticks solution. But that is probably out of scope for Meteor to patch.

[vjau]

Same problem as @laurentpayot. But the jsx package isn't even the package list, i don't know where it comes from.

[vjau]

I guess the problem is that the react package implies jsx.

[bySabi]

cc @avital @benjamn @stubailo 1.3 beta related problems goes here or we must create a separate issue like we did on: https://github.com/meteor/meteor/issues/5831, https://github.com/meteor/meteor/issues/5832, https://github.com/meteor/meteor/issues/5824, https://github.com/meteor/meteor/issues/5819 ?

[vjau]

When doing import * as React from "react" (in login.jsx on line 1) i get following error from the browser console :

Uncaught TypeError: babelHelpers.interopRequireWildcard is not a function
meteorInstall.app.client.imports.components.blaze.login.jsx @ login.jsx:1
fileEvaluate @ install.js:183
require @ install.js:75
(anonymous function) @ app.js?hash=64c87f0e53426ff260637397dfc3bbb3dcd4e519:20410
(anonymous function) @ app.js?hash=64c87f0e53426ff260637397dfc3bbb3dcd4e519:20415

[JeremyBYU]

I seem to still be having issues that I referenced above with this new Beta.

Okay I think I narrowed it down to an issue with the package.json I use to install my node modules. I think this is the offending text:

"dependencies": { "formsy-react": "git+http://github.com/christianalfoni/formsy-react#4931256" }, It seems if I specify a git url and commit and not a version number that the 1.3 module system goes haywire.

There are no errors during the build this time, however I am unable to reference formsy-react. I have tried every combination I can think of (require, import, relative reference, etc.). If I change the dependency to a version number then everything works great again. Same steps to replicate as stated above.

EDIT: This might just be an issue with formsy-react at the specified commit. I tried this with moment and didnt have an issue. I'll keep investigating.

[laurentpayot]

Created #5849 for "error: conflict: two packages included in the app (jsx and ecmascript) are both trying to handle *.jsx"

[jiku]

@benjamn 1.3b1 solved it but uncovered some new issues. See comment on #5799.

[dcsan]

does this include fast rebuilds as per

METEOR@1.2.2-faster-rebuilds.0

i looked at the changelog but its not obvious...

[benjamn]

If you've been having problems with the jsx and ecmascript packages conflicting, please update to the latest version: react@0.14.3_1. In this version, the jsx package is implemented entirely by the ecmascript package, so there is no conflict.