Node style modules or UMD?

[ericelliott]

npm is the largest programming language package repository there is. It's about 5x as large as Bower and much more active. An overwhelming majority of developers today are using Browserify, which uses node-style modules and bundles them for the browser. (Source: ModuleCounts.com)

Is it possible to test with node-style modules? Are there any docs which show node-style equivalent examples?

I really want to recommend this on my Essential JavaScript Resources doc, but I can't do that if it looks like it's AMD-only. Alternatively, if you'd like to support ES6 modules, that would be fine, too.

[vladikoff]

That is in the Wiki: https://github.com/theintern/intern/wiki/Writing-Tests-with-Intern#commonjs-code

[csnover]

Thanks for writing.

As @vladikoff mentioned there is information about how to load CommonJS code for testing in the wiki.

An overwhelming majority of developers today are using Browserify,

I know Browserify is popular with some Node.js-first developers but this is not a true statement generally.

I can't do that if it looks like it's AMD-only.

Why?

Alternatively, if you'd like to support ES6 modules, that would be fine, too.

Why is this OK but AMD is not? What would it mean to “support” ES6 modules?

[ericelliott]

1. Yes, it is true. Denying it doesn't change the fact that despite browser JS being a much larger use case than Node, npm is 5x larger than Bower and growing at a much faster clip than Bower - faster even than Bower and every other browser-only package manager combined. Bower has lost the package manager wars, and AMD has lost the module format wars.
2. AMD is quickly losing ground and I'm not going to recommend tools that are heavily dependent on AMD. It just confuses people when it's not immediately obvious how to integrate tools with the rest or their code.
3. ES6 modules are the future of JS modules. I'm happy recommending future proof tooling and teaching people how to use it today. What I won't recommend is tooling that will be obsolete in a year or two.

[dylans]

1. npm !== cjs, just like bower !== amd. These are dependency management systems, not module systems. You can npm install packages with AMD modules. At this point, it will be at least five years before ES6 modules gain wide adoption, and frankly, it's a vastly inferior format to what you can do with AMD today, so it will hopefully undergo significant changes in ES7 or else no one will actually use them. For example, ES6 modules do not really allow you to, at run-time, selectively choose to load a module or not. This is something you need with a testing tool, to, for example, determine if you are in a browser environment at the time the tests run, to determine if you will include that test which cannot run in a command-line environment for example.
2. Node made the deliberate choice to not support AMD, as they believe async module loading is not important (whereas it is essential for client-side dev). Intern is a tool to test applications, and that includes browsers. We don't require an author to write their code with AMD, just their tests. The syntax is frankly more terse and efficient than either CJS or ES6 if you have more than 2 dependencies in your module.
3. See point 1. If you really think things become obsolete in a year or two on the web when a standard doesn't meet the needs of developers, I'd like to point to the past 20 years of web development as evidence to the contrary.

[dylans]

@ericelliott my point being, you cannot only live in the future, sometimes you need to live in the present, especially when work on the future is not yet better than what you might use in the present. You're free to disagree of course, but arguing that npm is 5x more popular than bower is a bad statistical argument, and also arguing based on popularity over quality is not really very interesting. Open source software is not a zero sum game where only one thing can win, and only one approach is useful, and dismissing the work of the past ten years of people who have worked on the AMD format is unfair.

[ericelliott]

1

I am aware that module format and package repository are not the same thing. However, the overwhelming majority of packages in the npm registry are node-style modules. The overwhelming majority of packages in Bower are AMD modules. So my point still stands.

"You can programmatically import modules, with a syntax reminiscent of AMD modules:

System.import('some_module')
    .then(some_module => {
        ...
    })
    .catch(error => {
        ...
    });

System.import() enables you to:

Use modules inside <script> elements (where module syntax is not supported, consult Sect. “Further information” for details). Load modules conditionally. System.import() retrieves a single module, you can use Promise.all() to import several modules:

    Promise.all(
        ['module1', 'module2', 'module3']
        .map(x => System.import(x)))
    .then(function ([module1, module2, module3]) {

    });

2

As support for ES6 is increasingly adopted by the Node/io.js community (io.js 1.0.0 will have good support for the latest V8 engine, including many ES6 features), this will change, and ES6 modules are already quite usable with current ES6 transpilers.

The syntax is frankly more terse and efficient than either CJS or ES6 if you have more than 2 dependencies in your module.

No, it isn't. The additional function wrapper requirement and extra indentation adds distracting syntax noise.

3

If you really think things become obsolete in a year or two on the web when a standard doesn't meet the needs of developers, I'd like to point to the past 20 years of web development as evidence to the contrary.

I'd like to point you to 20 years of outdated technology alternatives to recommended web standards: Java applets, Flash, a thousand non-standard HTML, CSS and JavaScript extensions, the Prototype library that was all the rage before jQuery took over... How about ASP.net, C#.net, etc... all relics of a bygone era now.

Building on web platform standard technology is the safest bet when it comes to future proofing libraries and applications, especially when it comes to something as core as module formats. People are so sick of the module format incompatibility problems, I'd bet that ES6 modules will be clearly dominating within the next 3-5 years.

In other words, building a large enterprise app with this test framework today pretty much guarantees that the entire test suite may need to be rewritten after a couple of years worth of pumping tens of thousands of tests into the test suite. I'm just not going to recommend that course of action to anybody. It would be like telling them to start using Angular custom elements instead of standard web components when we all know that Angular custom element directives will be incompatible with Angular 2.0 a year from now.

If I'm wrong about this, feel free to publicly shame me with a link to this conversation 3 years from now, and I'll make a public apology and send you a big chocolate cake. =)

[csnover]

Hi,

"You can programmatically import modules, with a syntax reminiscent of AMD modules: […]

FYI, TC39 punted on module loading and dropped the loader from the ES6 spec at the end of September, just after that article was written. Anything you read about Loader or System object is not real right now, and there will not be any module loader in the ES6 spec. As such, ES6 does not solve the module problem. TC39’s solution is to make vendors deal with it (which is exactly what the CommonJS project tried and failed to do 6 years ago, and is where the Node.js and AMD module formats originate).

the Prototype library that was all the rage before jQuery took over...

Applications written using Prototype.js still run just fine today. I know this because I wrote one eight years ago and it is still running fine.

How about ASP.net, C#.net, etc...

ASP.NET and C# are very much alive and well, used as the primary language to develop quite a few Windows-only and cross-platform applications & games. In fact within the last 3 months Microsoft open sourced .NET core, so there is quite a lot of activity there.

building a large enterprise app with this test framework today pretty much guarantees that the entire test suite may need to be rewritten after a couple of years worth of pumping tens of thousands of tests into the test suite.

Objectively, the only standardised module format that actually works in browsers without an extra compile step is AMD. It won’t just stop working or become unsupported one day, unlike your Angular example. It is also extremely well suited for being automatically converted to any future module format since the wrapper is dead simple (unlike the Node.js module format which is not fully compatible with the static analysis used for ES6 imports). People can write tests in AMD format today and they can convert them to the One True Module Format tomorrow with zero effort. This is not a big deal. This is not a reason not to use or recommend Intern.

Obviously it’s your prerogative what you want to list or not list on your list but let’s stick to the facts (on both sides). I would also recommend seriously finding another forum to have this discussion on. Thanks!

[dylans]

1. As far as I know, you cannot easily do this with ES6 (and intentionally so, as part of the spec):

define([
    'app/has!mobile?app/MobileUi:tablet?app/TabletUi:app/DesktopUi'
], function (AppUI) {
    // ...
});

Why does that matter? In a test suite, I want to be able to efficiently do this:

define(['has!host-browser?./some/test/suite'], function() {})

Neither of your examples solve this incredibly common use case.

1. Distracting syntax is your opinion, it's still less to type.

[ericelliott]

People can write tests in AMD format today and they can convert them to the One True Module Format tomorrow with zero effort.

Point taken.

define(['has!host-browser?./some/test/suite'], function() {})

Another interesting point.

Is there a good source in the docs that addresses these issues? I suspect I'm not the only one getting turned off by the AMD. It sounds like you guys have considered these issues and have a well reasoned argument for sticking to your guns in spite of popular trends.

I'd love to learn more about your thoughts and approach on this, and discuss them with the broader community. I'm working on a next-gen build process tool suite, so I'm taking a broad survey of available options. You can find more in the Cloverfield repository. So far concentrating on task runner config automation (Discussion here), but spinning up code quality tools like lint and automated tests are next on my todo list.

I've created a new thread to discuss testing issues. I'd love to call out more details there.

[dylans]

@ericelliott https://groups.google.com/forum/#!msg/amd-implement/AbAkImN23F0/wlkvjXwER0sJ is a bit out of date, but not that much has changed

[ericelliott]

@dylans After several years using BOTH AMD and Node style modules, it seems to me that the use cases solved by AMD are edge cases, and universally, I have never had a problem authoring all of my modules in Node style modules and producing UMDs explicitly for the exceptional cases where I actually want conditional / asynchronous require.

None of the points in that message resonated with me as particularly interesting. Is there a specific point you'd like to call out and discuss?

[dylans]

Interesting !== necessary to solve complex application development problems. Everything in AMD that's not supported by ES6 modules are things we've needed through years of building large scale apps. I cannot easily distill years of this stuff into a few talking points, other than to say that ES6 solves the easy to solve problems.

Your main point was that ES6 is the future, and we're saying it's not yet there. For so many specs/standards that have been introduced over the years, I can point to a graveyard of things that are not up to spec (things like isContentEditable, or even the current state of web components unfortunately). Just because something is new or standard doesn't mean it's immediately better.

Your next point is that you prefer CJS + Browserify, which is of course your choice given the open web we live in. We prefer a module format that works today, and solves a number of on demand dependency loading scenarios that are common in our development efforts, that ES6 chooses to ignore. We prefer not having to compile to something to get it to run in a browser, when that code is intended to run in a browser.

Async dependency loading, being able to on-the-fly determine which dependency to load, being able to work with module loader plugins to change what resource gets returned, and doing all of this without a compile step is what AMD is about. AMD is also about reducing reliance on global variables and about minimizing object allocation, which are pretty important problems that occur for large apps.

In the context of Intern, AMD is really excellent with its approach to dependency management, being able to mock/stub simply by undefining a dependency, mapping the dependency, and reloading the dependency. For example, see http://www.sitepen.com/blog/2014/07/11/testable-code-best-practices/ and http://www.sitepen.com/blog/2014/07/14/mocking-data-with-intern/

So, you have decided you don't like AMD, but your reasons are that you either don't like the syntax, don't get its benefits, you haven't found a need, or you don't think it's popular or cool. I don't care about popularity, I care about solutions that work and work well.

You're placing the burden of proof on us, without given a technical reason why you think AMD is inferior. Is there something in a browser that AMD cannot do that ES6 modules can do today?

[ericelliott]

We prefer not having to compile to something to get it to run in a browser, when that code is intended to run in a browser.

For large applications, I really don't see the point. If you're not compiling anyway, you're hurting your user experience unless all your users are running the latest browsers with HTTP2/SPDY protocol support (seriously doubtful).

Async dependency loading, being able to on-the-fly determine which dependency to load, being able to work with module loader plugins to change what resource gets returned, and doing all of this without a compile step is what AMD is about.

Yep. All I'm saying is that in my experience, it's not actually beneficial to remove the compile step. In fact, a lot of projects I've seen started out without a compile step (because AMD lets them), and they go months or even YEARS without bundling their assets for better perf. IMO, that's a pit of despair that hurts users and slows page loads, which hurts the business bottom line.

Almost universally, avoiding the compile step is harmful to real-world projects.

AMD is also about reducing reliance on global variables and about minimizing object allocation, which are pretty important problems that occur for large apps.

All module formats do these things.

In the context of Intern, AMD is really excellent with its approach to dependency management, being able to mock/stub simply by undefining a dependency, mapping the dependency, and reloading the dependency.

Maybe none of that is the module format's job. For instance, I get all that conditional goodness by using feature toggles and switching toggle states, and that doesn't just serve my testing needs, it serves the app and business needs, as well.

So, you have decided you don't like AMD, but your reasons are that you either don't like the syntax, don't get its benefits, you haven't found a need, or you don't think it's popular or cool.

I don't like the syntax, I get all the benefits you mentioned without authoring everything in AMD, I have an occasional need for async loading, but when I need it, I still use node-style modules as my authoring format.

popular or cool.

Popular matters, because you get network / ecosystem benefits. Cool matters because programming should be fun, and AMD isn't. ;)

In the edge-cases where you actually need async dependency loading, one-off UMD bundles + curl.js does the trick. I don't hate AMD. It has its use-cases, but IMO:

• It sucks as an authoring format.
• It sucks as default behavior.
• It hurts the UX when you skip compile steps (and many AMD users do, it's the default mode for using AMD in most of the docs).

I have experience using AMD over the course of several years in very large apps (for instance, Adobe Creative Cloud uses AMD for front-end code. I served on the creative cloud team for more than a year).

I get what AMD is about. I don't dislike it because I misunderstand it. I dislike it as a default authoring format because there are choices I know from experience to be better.

You think I don't understand? I've fully explored both AMD and Node modules for several years for browser use in production environments with 100kloc + code bases. Have you?

[dylans]

For large applications, I really don't see the point. If you're not compiling anyway, you're hurting your user experience unless all your users are running the latest browsers with HTTP2/SPDY protocol support (seriously doubtful).

Yes, I agree a compile step is absolutely necessary for production, but should be an optional step during development if at all possible. And since we're talking about writing tests, I don't want to have to compile a test suite in order to run it.

Maybe none of that is the module format's job. For instance, I get all that conditional goodness by using feature toggles and switching toggle states, and that doesn't just serve my testing needs, it serves the app and business needs, as well.

From my perspective it is, but I of course understand that it doesn't have to be. But in the case of Intern in particular, it's very ideal to be able to unload, remap, load, unload, unmap, load as part of the setup/teardown process for a test suite.

I don't like the syntax, I get all the benefits you mentioned without authoring everything in AMD, I have an occasional need for async loading, but when I need it, I still use node-style modules as my authoring format.

I actually like the syntax. I like it's terseness and I like that it's strict. That doesn't mean that I don't also like CJS syntax when I'm writing Node.js code, or even the TypeScript module syntax.

Popular matters, because you get network / ecosystem benefits. Cool matters because programming should be fun, and AMD isn't. ;)

In the words of Hamlet, "for there is nothing either good or bad, but thinking makes it so". I think AMD is fun. ;)

As one of the creators of Dojo and a long-time JS dev (pretty much since the beginning of the language), I realize that there is nothing I can do to predict or influence popularity. The only thing I can control is engineering quality and make my life more productive. I find AMD to be highly productive and efficient.

On the other hand, I find UMD to be completely not fun at all,, and I'm not a fan of the browserify process. I mean, we're creating code to primarily run in browsers, and to suggest that I need to go through a process to make my primary target environment work feels wrong to me.

I have experience using AMD over the course of several years in very large apps (for instance, Adobe Creative Cloud uses AMD for front-end code. I served on the creative cloud team for more than a year).

While I've of course heard of Creative Cloud, I've never looked at the code, so I don't know how well that worked for the Adobe team. I will say that I find AMD to be far more refined in an environment like Dojo than in one based on jQuery, but I don't know what the CC team used.

You think I don't understand? I've fully explored both AMD and Node modules for several years for browser use in production environments with 100kloc + code bases. Have you?

Of course I have. Our team at SitePen (and I) have been doing this for a very long time, and have worked with everything.

We live in a far from perfect authoring environment, where we make compromises every day, and live with them. As I've tried to explain, what bugged me the most is when you slighted Intern because you think ES6 is awesome, which it is not. Intern is a well engineered test stack, and we believe it to be the best architected product on the market today, regardless of the module format.

In terms of what Intern uses internally, the Intern package uses AMD, but the leadfoot and digdug subprojects are authored as CJS. Why? Because those dependencies run inside of Node.js only, whereas Intern's test running code is intended to run in browsers as well. So it's not like we have to use AMD everywhere because we believe it's the only module format that matters for everything, but rather we use what we believe is the right format for the right situation.

And really, if ES6 modules really get there, or CJS+Browserify really does "win" such that the writing is truly on the wall for AMD, the time it would take to get Intern working to support other module formats for writing tests is not insurmountable, and it's very easy to convert from AMD to other formats. But we do believe that AMD is the best option today for an authoring format for tests that should run in a browser and on the command-line.

[ericelliott]

Yes, I agree a compile step is absolutely necessary for production, but should be an optional step during development if at all possible. And since we're talking about writing tests, I don't want to have to compile a test suite in order to run it.

Most of the JS I write is isomorphic. When I run unit tests on my local machine, I use standard Node modules and don't have to bundle. When I run the browser tests in the cloud (typically with Sauce Labs), I prefer to bundle first so that the code being tested is actually the final code that will ship to production.

Failing to test your bundles is a big mistake, IMO. More on that point...

it's very ideal to be able to unload, remap, load, unload, unmap, load as part of the setup/teardown process for a test suite.

Doing this takes you further from testing the actual code that is bundled and shipped in production. I'd rather test the code my users are actually going to get delivered to their browsers.

In the words of Hamlet, "for there is nothing either good or bad, but thinking makes it so". I think AMD is fun. ;)

Diversity is a good thing. =)

On the other hand, I find UMD to be completely not fun at all...

I haven't hand-written a UMD wrapper for more than 3 years. I let browserify -s do that for me, so I'm authoring purely in Node style modules all the time. =)

While I've of course heard of Creative Cloud, I've never looked at the code, so I don't know how well that worked for the Adobe team.

Creative Cloud uses AMD and Backbone for front end modules. It's a team of small teams all working together building lots of mini apps that come together as one large application with a large set of features for creative collaboration.

An interesting aside, when I left a year and a half after the AMD module system was put in place, they were still not bundling, and it was hurting page load times. IMO, that's a good argument to use a format that forces you to bundle from the start. UX should be a high priority, and page load times are one of the biggest barriers to great UX.

when you slighted Intern because you think ES6 is awesome

I meant no disrespect to intern. I'm here because I think it looks interesting and I'd like to be able to say that I can recommend it. For me, anything that forces you to author in AMD is a non-starter. Your team should be aware that I'm not alone in that, and consciously enter into the decision to cut yourselves off from a potentially large number of users because you think it's worth the compromise.

If you're making the choice with your eyes open, aware of the consequences, and you believe it's the right choice, I admire that. Integrity is important. =)

it's very easy to convert from AMD to other formats.

Yes it is. Keep up the good work, and good luck! =)

[dylans]

I prefer to bundle first so that the code being tested is actually the final code that will ship to production.

We create a lot of open source libraries and toolkits, and need to be able to test things in isolation. One of my mantras is that code that cannot easily be decoupled is usually poorly architected (the converse is not guaranteed, but often the case). Also, our test suites for projects like Dojo take so long to run, usually exceeding the timeout limits imposed by travis-ci. Fun stuff...

Doing this takes you further from testing the actual code that is bundled and shipped in production. I'd rather test the code my users are actually going to get delivered to their browsers.

Sometimes the point of a good test is to isolate a production problem. Front-end developers historically are blamed for every bug on a brittle platform. Having nice mocks makes it easy to point out that a bug is with an API change or a data service problem, etc., making it easier to track down an issue quickly.

An interesting aside, when I left a year and a half after the AMD module system was put in place, they were still not bundling, and it was hurting page load times.

Yikes. ;)

If you're making the choice with your eyes open, aware of the consequences, and you believe it's the right choice, I admire that. Integrity is important. =)

Thanks!

[csnover]

anything that forces you to author in AMD is a non-starter

Just as Intern’s choice of module format will not force you to rewrite all of your tests in the future if/when the ES6 module format becomes viable, use of Intern also does not force you to author anything in AMD.

You’ve already said your feeling is everyone should be forced to run builds to run their code, so if if it is really too much to stomach writing define(function (require) {}) around your test code, just author your test modules using your preferred module syntax and use a build to convert them into AMD modules so they can be loaded by Intern! Write them in TypeScript. Write them in CoffeeScript. Write them in ES6. Write them in AtFlowHaskellLispMonadicScript. It really doesn’t matter.

Intern is not prescriptive but you keep talking about it like it is. It is not. It provides a very sane and reasonable set of defaults that work across multiple platforms that people target for testing. If you want to not follow those defaults it does not prevent you from it.

You can write your own test interfaces. You can write your own reporters. You can choose your own assertion library. You can choose your own continuous integration service. You can choose your own module loader with useLoader, as long as it can also consume AMD-formatted modules. If you really want to be loathsome, you can dump in a second different unit testing system. In Intern 3, you’ll even be able to write your own entire custom bloody test executors.

There is no other JavaScript testing framework on the planet that gives you anything near this level of flexibility (and I know, because I researched all of them before I started the project, because I didn’t want to write another testing framework!). You would have to cobble 3 or 4 other libraries together on your own just to get what you get out of the box with Intern, and this fixation over how it uses the only module format that works natively in browsers today is just…I don’t even know what to say. Double facepalm.

[ericelliott]

We create a lot of open source libraries and toolkits, and need to be able to test things in isolation.

All of my modules are tested in isolation, and bundled, and it happens automatically.

Also, our test suites for projects like Dojo take so long to run, usually exceeding the timeout limits imposed by travis-ci.

Yep. I'm planning to run our module tests isolated and in parallel. Aiming for complete runs of a very large codebase in about a minute, distributed in the cloud, testing all of our target platforms simultaneously.

Sometimes the point of a good test is to isolate a production problem.

True enough. That's why I test both isolated and bundled, and that's a breeze with Node modules. =)

[ericelliott]

Just as Intern’s choice of module format will not force you to rewrite all of your tests in the future if/when the ES6 module format becomes viable, use of Intern also does not force you to author anything in AMD.

Good point. Funny I didn't think of it! =)

It provides a very sane and reasonable set of defaults that work across multiple platforms that people target for testing.

As I've mentioned before, most of the code I test is isomorphic. What is your recommended solution for testing Node code in Node with Intern? Do you have a good example case I can point people to?

You can choose your own module loader with useLoader, as long as it can also consume AMD-formatted modules.

Intern is in the module loading business? Can you point to a good example of this for me?

this fixation over how it uses the only module format that works natively in browsers today is just…I don’t even know what to say. Double facepalm.

Haha. Well, I'm not the only one who's having doubts about Intern for this reason, but you're right, we could author our tests in Node style modules and bundle them to UMDs automatically with Browserify. Problem solved. I think I'll give that a try and kick the tires a bit.

I'm glad you responded! =)

[neonstalwart]

What is your recommended solution for testing Node code in Node with Intern? Do you have a good example case I can point people to?

https://github.com/neonstalwart/level-perstore is something i've written where the tests are for node code in node with intern. i have a number of other modules doing the same. it's really not all that difficult or painful - the benefits far outweigh the costs in my mind.

[ericelliott]

var Q = require('dojo/node!q')

Is that how you require the Node modules?

[ericelliott]

@neonstalwart - Thanks for the link. If you wanted to author your tests in Node style modules too, how would you go about that?

[neonstalwart]

it would be simple enough to write one AMD module that loaded all your node tests - i believe this kind of thing is possible but i have not tried.

// load-tests.js
define(function (require) {
var test = require('intern!object'),
    // cross over from AMD to node once
    suite = require('intern/node!./tests');

suite(test);
});

// tests.js (node-style)
module.exports = function (test) {
    require('./foo-test')(test);
};

// foo-test.js
module.exports = function (test) {
    test({
        // ...
    });
};

[ericelliott]

Nice! We should test this out and maybe add it to the docs for people who prefer to author in Node style modules. =)

[neonstalwart]

at that point though... you're writing a function wrapper for every module so that you can pass the test object to them. sounds a lot like AMD :open_mouth:

[ericelliott]

at that point though... you're writing a function wrapper for every module so that you can pass the test object to them. sounds a lot like AMD

Not really.. couldn't tests.js just loop over all the modules in some directory and require them all in one go? No extra wrappers necessary?

[neonstalwart]

this thread has gone way beyond an "issue" - @csnover is probably about ready to :boom:

@ericelliott i guess you could. each test could just be an object and then tests.js could call the equivalent of test(require('./foo-test'))

i'd love to hear if it works for you - i'm sure it would be a useful pattern for some people.

[ericelliott]

this thread has gone way beyond an "issue" - @csnover is probably about ready to :boom:

Haha, I really appreciate your patience. =)

Maybe this thread can be used as a reference to add documentation around workflows for node style modules. At the very least, it's made me more comfortable with Intern.

[aramk]

A note regarding the loading of Node.js modules in the format @neonstalwart mentioned: I had the issue where using require('lodash') in my Node.js module wasn't working correctly. It turns out that define() was defined while lodash was loading, preventing it from defining itself with module.exports. I redefined define = null and reverted it after all tests ran to get it working.