Transform `examples/js` to support modules

[ghost]

The main source code has recently been ported to ES2015 modules (see #9310); however, some parts are still relying on global namespace pollution and have thus become unusable.

Specifically the things in /examples/js/ haven't been transformed to support modules yet; this makes them currently unusable from within environments such as webpack or SystemJS.

Edit, 11-20-2016: They can be used, but it requires a lot of setup: SystemJS, webpack. If you need assistance with these: Sorry, but this is the wrong thread to announce it! 😄

[ghost]

Side note: You could theoretically use these from within webpack by injecting the global THREE variable (and others, depending on what you're trying to load) using imports-loader, but the problem still remains for SystemJS and native module evaluation that will eventually land in browsers.

[mrdoob]

I don't know how this can be fixed. We can't turn those files into ES6 Modules because, not only browsers don't support them yet, we want to support old-ish browsers too.

So, as far as I can see, the "hack" for Webpack and SystemJS is the price to pay for now?

[andrevenancio]

Either things like examples/js/postprocessing/ become a module of themselves or we need to come up with some kind of plugin logic.

@mrdoob what if github.com/threejs becomes a thing? You can create different repos inside of it which have a similar build system as the current threejs, which can be used via modules or people can just download the legacy code in the build folder. Again with the postprocessing in mind, it will have THREE has a dependency, and it would be both a module or you can use the /build/EffectComposer.js for legacy code?

[mrdoob]

@mrdoob what if github.com/threejs becomes a thing?

I have a hard time maintaining one single repo already 😕

[andrevenancio]

the idea was to give you and the maintainers more control over the releases, a bit like https://github.com/airbnb or https://github.com/facebook I'm happy to pick up some bits and bobs as I need them but you'll end up with github.com/randomuser/effectscomposer github.com/randomuser2/orbitcontrols :(

[GGAlanSmithee]

Does it really make sense to make the examples be modules though? If I understand it correctly, the examples are consumers of three, not a part of it. They are there to provide you with example usages of three.js.

I think a more correct way to go about it would be to move as much as possible to the core of three.js and rely on tree shaking to eliminate dead code. (I understand this will be problematic for users that does not want to use a build pipe of their own though..)

[andrevenancio]

We're not saying we should have the examples as modules. We're saying that some of the files referenced in the examples folder aren't yet modularised, like OrbitControls.js or EffectsComposer.js or many others that are very often used in demos, prototypes and even in production. Those files, should be, as I was suggesting, modules of their own, outside of three.js

[satori99]

I agree in regards to OrbitControls and the the other control classes. They are too useful to be just examples.

[killroy42]

I wrote a small universal module loader, which supports various module systems and falls back to global namespace. It makes it all work, just by including the script. Should I do a demo with it for the examples?

[mrdoob]

They are too useful to be just examples.

What does that mean? Do you mean that they are too commonly used?

[satori99]

Yeah exactly. Especially OrbitControls. It might just be me, but I end up including that file a lot.

[danrossi]

I've ported OrbitControls over to a module, you can do something similar. My version has some changes when to dispatch start events on movement.

It can then include it within Three.Js main app file ie

export { OrbitControls } from '../three-vr-orbitcontrols/src/OrbitControls.js';

I've had to include other examples that are not modules yet, you concat them in with rollup in the rollup script like so

var fileList = [
    "../three.js/examples/js/effects/StereoEffect.js",
    "../three.js/examples/js/effects/AnaglyphEffect.js",
    "../three.js/examples/js/controls/VrControls.js",
    "../three.js/examples/js/controls/DeviceOrientationControls.js"
];

var out = fileList.map(function(filePath){
    return fs.readFileSync(filePath, 'utf-8');
});

var footer = out.join("\n");

export default {
    entry: 'Three.js',
    dest: 'build/three.js',
    moduleName: 'THREE',
    format: 'umd',
    indent: '\t',
    plugins: [
        glsl()
    ],

    outro: outro,
    footer: footer
};

https://github.com/danrossi/three-vr-orbitcontrols

I've done something similar for the VREffect but have refactored and cleaned it up also.

https://github.com/danrossi/three-vreffect

[killroy42]

Perhaps we could start a small initiative to modularize these commonly used helpers from the examples. Clean them up and improve the quality (For example I rebuild the FPS controller to include keyboard controls, events, etc).

[mrdoob]

What will happen to the people that don't use modules?

[ghost]

I find myself in the same position as @satori99, often needing a way to use Projector or CanvasRenderer, both being far from just an "example" with a good 1000 lines each.

@mrdoob People that don't use modules could just be pointed to versions of the project that had a pure ES5 codebase, so that they can pull the code via a <script> tag via global namespace pollution.

Essentially, the jQuery project did the same thing to gradually save themselves from supporting legacy browser versions (i.e. make a cut somewhere, support-wise).

[mrdoob]

@mrdoob People that don't use modules could just be pointed to versions of the project that had a pure ES5 codebase, so that they can pull the code via a <script> tag via global namespace pollution.

So... maintaining two versions of the same code base?

Essentially, the jQuery project did the same thing to gradually save themselves from supporting legacy browser versions (i.e. make a cut somewhere, support-wise).

That's different. One thing is stopping supporting legacy browsers. What you guys are suggesting is stopping supporting new programmers.

[GGAlanSmithee]

Before we continue this discussion, I think it is important to highlight that this is "only" affecting people that are using a bundler already (emphasis mine):

Specifically the things in /examples/js/ haven't been transformed to support modules yet; this makes them currently unusable from within environments such as webpack or SystemJS.

Users that just want to include examples in a page can still do so (correct me if I'm wrong)

<script type="text/javascript" src="three.js"></script>
<script type="text/javascript" src="examples/js/OrbitControls.js"></script>
<script type="text/javascript" src="myapp.js"></script>

The group of users that does use a bundler is ofcourse very large and not unsignificant (I am one of these), but given that these users are already using a bundler, it might not be crazy to require them to do some manual step if they want to include examples in a custom build of three.

For example, something in the lines of what @danrossi have done, but we could modify the ´npm build´ script to take an additional, optional, list of examples to include in the bundle.

[danrossi]

My build is quite out there.

I have had to copy the rollup script config. the package json file into a seperate directory outside three.js. The Three.Js main app file which becomes out of sync with head commits of course and need to be updated manually.

the paths look like

project
   build-threejs
   three.js

I then reference the exports like this within the main app file. They have to be changed from the original reference path.

export { EventDispatcher } from '../three.js/src/core/EventDispatcher.js';

I've then included my refactored modules from examples.

The examples that have not been converted have to be appended after the build output with the example code above.

I've stripped the size by 200kb by removing exports that are not needed also from within this modified main app file.

[GGAlanSmithee]

Yeah, I guess concatenating files like that is a bit hacky and also negates some of the possible benefits..

@kdex, is your proposal to turn each example into an umd bundle? That would make it work in all env. at the cost of added noise..

[ghost]

So... maintaining two versions of the same code base?

My intent was to eventually kill off the way that three.js ships bundles that mess with global namespace. It might be too early for that, given that <script type="module"> and the System global are not quite there yet, but I don't think that any project should promote this way of shipping code to the user in the future anymore. Hence, what I meant to say was that three.js should eventually just work like so:

<!--
 Use the bundle if you need this to work with legacy browsers.
 This will inject `window.THREE`.

 WARNING: 0.80.2 is the last version that comes with `window.THREE`.
 If you need a newer version, please consider migrating to using modules.
-->
<script src="three.js"></script>

So that in the future, users could migrate to something along the lines of:

<script type="module">
    /* This would be the version that supports `examples/js` to be consumed */
    import THREE from "./vendor/three/three.min.js";
    import CanvasRenderer from "./vendor/three/examples/js/renderers/CanvasRenderer.js";
</script>

Another (maybe cleaner?) approach worth discussing might be to move examples/js into the core. Pros:

• No breaking change in terms of deployment (see outlined approach above)
• Three can be consumed from bundlers and browsers
• Full support for import syntax without hacking globals in with something like imports-loader

Cons:

• For people using <script> tags only, the consumed bundle size will increase by a lot

If there's more cons, feel free to bring them up; but as for the bundle size: This only concerns users that want to inject three into global namespace anyway. If they see that this hurts site performance, it will as well act as an incentive to migrate to a build environment with something like rollup or tree shaking and create personalized bundles themselves.

@GGAlanSmithee brings up another good idea: With UMD, the bundle size won't increase for legacy environments, it allows being used with import, allows the examples to be transformed to use ES2015+ and prevents them to be moved to the core. On the other hand, this also makes three.js's build pipeline a little more complicated.

[mrdoob]

So that in the future, users could migrate to something along the lines of:

<script>
    /* This would be the version that supports `examples/js` to be consumed */
    import THREE from "./vendor/three/three.min.js";
    import CanvasRenderer from "./vendor/three/examples/js/renderers/CanvasRenderer.js";
</script>

That was super helpful to see! Thanks! Can't wait for browsers to start supporting modules.

[andrevenancio]

mrdoob commented 7 hours ago What will happen to the people that don't use modules?

they will still be able to use script tags in their html with references to either a CDN or a local copy like so:

<html>
    <head>
        <title>Three.js</title>
    </head>
    <body>
        <script src="https://cdn.rawgit.com/mrdoob/three.js/dev/build/three.js"></script>
        <script src="https://raw.githubusercontent.com/mrdoob/three.js/dev/examples/js/controls/OrbitControls.js"></script>
    </body>
</html>

One of the advantages, for me, of having three.js as an npm module, means I avoid having to add a script tag per library in the html, or merge vendor scripts, copy files around from src to dest folders etc.

But having three.js as a module but not being able to import in the same way, OrbitControls, PostProcessing, ExploderGeometry etc, its not convenient.

We can easily have the community helping exporting every utility on the examples folder to a modularised version with backward compatibility (as three.js currently is). But I reckon its something you should do (at least create the official repositories, so people can contribute to them), or we'll end up having a couple of different versions of OrbitControls like we currently do in npmjs.

[ghost]

@andrevenancio If every example becomes its own official npm module written with ES2015 modules, they would essentially all do import THREE from "three"; in order to provide functionality by using what this THREE instance provides it, right? This is both useful as well as dangerous.

If some guy's personal three.js project does an import THREE from "three"; too, there is a chance that the versions of his project's THREE instance and, say his three-canvas-renderer module's THREE instance are different and not thus not only prevent a proper rollup, but can also introduce weird bugs that are hard to debug (I ran into failing subclass checks before, having introduced two vastly different three.js files by accident via <script>. I'd imagine that the same could happen by dividing the examples into npm modules).

Wouldn't it be safer if we could have all that stuff in one place, e.g. something like

import THREE, { CanvasRenderer } from "three";

?

This would guarantee that the THREE instance for CanvasRenderer also matches the instance of THREE you're importing, as they both originate from the same npm module. This ensures that the rollup can be fully leveraged and should prevent weird bugs.

At least, that's the npm side of it. On the GitHub side, it's really a matter of personal preference if you want to split the examples up into different repositories.

[andrevenancio]

Again, I never said examples should be their own npm module. What I said, and keep trying to explain, is that some utilities that only can be found in examples/js/ should be moved elsewhere.

Example, OrbitControls, or everything in the postprocessing folder.

And yes, I agree with you, we should be able to import it such as

import THREE, { OrbitControls } from 'three'

[danrossi]

The build could generate each file into place. into examples/js/ and can be imported as individual modules in src/examples/ or contrib/src/ ?

[borismus]

I'd also really like this to be possible, both via

require('./node_modules/three/examples/js/controls/VRControls'); 

and via modules:

import VRControls from './node_modules/three/examples/js/controls/VRControls'

Could either manually add JS boilerplate to the end of each example (doesn't scale):

if (module && module.exports) {
  module.exports = ModuleName;
} else {
  THREE.ModuleName = ModuleName;
}

Or do something more clever with a build step. Having all of the example code in a unified 'three' bundle seems not particularly friendly for non-npm users, unless I'm missing something.

[mrdoob]

There is no way to do a ES6 module code that also works with non-ES6 code without transpiling, right?

[GGAlanSmithee]

@mrdoob the es6 modules format is still not implemented in any browser (or node) so it will require transpiling. Even when support is added to modern browsers there is the question of backwards compatability, so es6 modules will probably have to be transpiled for a long time

[satori99]

I did an experiment with all this new module stuff to see how small the result could be after tree shaking and minification. I included some example classes by copying the files and adding an export/import declarations manually. This was easy enough to do, and worked fine on the first try.

It is a hassle, but doesn't this come down to whether the examples/ files are just examples, or have some of them become too useful to be considered as just example code? I'm thinking of the post-processing and controls code mostly.

[danrossi]

@GGAlanSmithee chrome and firefox both run es6 natively. It was a surprise. If you try and change the rollup mode from "umd" to "es" . It should still run. I do believe it has to be transpiled for a very long time sadly unless two versions are offered. At least the rollup transpiling is very small compared to just using babel.

[GGAlanSmithee]

@danrossi oh I did not know, thanks for telling me, I thought it was still some ways off. I agree that transpiling will still be required for a long time.

[danrossi]

Cecil mentioned me but the message isn't here strange. You can't choose es for three.js because it's not entirely pure es6 yet. It's just using es6 module imports that rollup is converting.

Once you do, you can run babel through rollup.

Back to the examples. I see no reason the sources can't be in es6 format. And then transpiled into place. The files in the target directory examples/js is the transpiled files. The example sources could go into contrib/js ?

This means they could be imported into an es6 module project.

import Module from 'three.js/contrib/js/Module';

[cecilemuller]

We did have a config for building an es bundle (it's just the glsl transform that prevents using src directly), the eventual issue was merely the existence of a second bundle might be too confusing for users apparently :/

[zhanghaowx]

Thanks to @kdex for pointing out import-loader could be a workaround. It worked perfectly for me:

import "imports?THREE=three!loaders/OBJLoader";

[mrdoob]

Thanks to @kdex for pointing out import-loader could be a workaround. It worked perfectly for me:

import "imports?THREE=three!loaders/OBJLoader";

Oh? How does that work?

[ghost]

@mrdoob It's a Webpack thing. :smile: I mentioned import-loader. The plugin allows you to inject a global variable into your module's scope. So the global (!) namespace of your execution context still remains completely clean, but during "compilation", Webpack will be able to figure out where to lookup the binding for a module that only comes with a global variable.

Basically, this plugin currently acts as the bridge between projects that only offer you a global variable and proper ES2015 modules, at least in the webpack world.

The syntax that you can see above is weird. It roughtly translates to "Hey, imports loader! In this module, please assume that there's a global variable named THREE, and its value is whatever the npm package three exports. Also make sure to import three/loaders/OBJLoader.js, but don't bind it to any variable." (which should exploit the fact that it modifies the THREE object, so there's no need for a variable binding).

Beware: All of this only covers webpack. AFAIR, jspm currently doesn't come with such a plugin, and once browsers handle ES2015 modules natively, we're back to square one.

[ghost]

By the way: Earlier in this thread, I said that moving everything into the core might be safer, since it allows importing components as:

import THREE, { CanvasRenderer } from "three";

After some reflection (and comparison to what big libraries tend to do), I'm not too sure about that anymore. I'm not sure how efficient the best tree-shaking algorithm can be, but compare:

import THREE from "three";
import CanvasRenderer from "three/CanvasRenderer";

The difference is that by importing two exports from "three", your browser still needs to know the entire three module to know what the exports are. Now imagine what happens to the file size if we cram every export in there. :confused:

I've seen big libraries like material-ui tackle this problem as shown above, too, and they tried the "single-module" way once, too.

[danrossi]

Good news. you don't have to turn the entire library into Es6 to turn the modules into Es6 and pass through babel. This is one module turned into an Es6 module and explains the extra rollup configs.

If at any stage it does turn into Es6 code, eslint is a good code checker and helps check for code problems.

https://github.com/danrossi/three-vr-pointerlockcontrols

[killroy42]

It's not very hard to do hybrid module/ global file actually. Will try to wrap my FPS controls into a forward/backward compatible file.

[danrossi]

@killroy42 if you mean outside of the main build that is possible. My requirements is to build them in. Which I then concat at the top of my Es6 project.

My guess the main app for the module would just have something like

export { PointerLockControls } from './src/PointerLockControls';

[killroy42]

I already do that in my projects, although using a loader stub that manages it. But I think you can create a hybrid .js file that can be used in either environment.

[danieldelcore]

I know how mrdoob feels about the accessibility of the project to newcomers. So hopefully a newbie's perspective helps here.

There are so many useful scripts nested within the project that I don't think belong directly under the THREE namespace, but rather a complementary one (or more). I feel that it would be beneficial to break these things out into seperate modules and house them under an organisation so that they may be maintained separately by the community. It will will take the load off the core repo and seperate concerns in regards to Documentation, PRs, and issues.

For example:

• editor
• examples
• utils
• threejs.org
• etc.

For me as a newbie, the problem wasn't that everything was hard to get started, it was more that i had to delve into the codebase and try to figure out if a script was there and right for my needs. Separating the repos would also give contributors a chance to write some readmes and tests, whilst allowing three to scale.

[mrdoob]

For me as a newbie, the problem wasn't that everything was hard to get started, it was more that i had to delve into the codebase and try to figure out if a script was there and right for my needs.

Not sure we're talking about the same type of newbie... A newbie for me is someone that wants to do a website without modules and/or build processes. Just a html and a bunch of js files.

Separating the repos would also give contributors a chance to write some readmes and tests, whilst allowing three to scale.

I don't think you're taking in consideration the amount of work that this would bring me...

[andrevenancio]

completely understand where you're coming from, but why change three.js to a modular approach if we can't use all the plugins that depend on the global THREE variable? :( (well we can use them if we add a script tag to the html, but then we have different logics of importing files that complement themselves)

[mrdoob]

completely understand where you're coming from, but why change three.js to a modular approach if we can't use all the plugins that depend on the global THREE variable? :(

Are you suggesting that we should revert the modules change? ;P

[andrevenancio]

YES for my own benefit and NO because I'm afraid of the community going after me and hunt me down! :)

[table315]

@kdex I'm current working on a Angular2+three.js app in TypeScript using angular-cli(the build system is webpack). I'm trying to load a model with the MTLLoader. After I installed the imports-loader and imported the three and MTLLoader as the way that you have suggested:

import * as THREE from 'three';
import "imports?THREE=three!../../node_modules/three/examples/js/loaders/MTLLoader.js";

When I do:

 var mtlLoader = new THREE.MTLLoader();

It says:

Property 'MTLLoader' does not exist on type 'typeof THREE'.

Looks like MTLLoader still not be able load into the THREE variable. Any Idea?

[ghost]

@table315 You're not the first person to come to this thread looking for a way to import examples in a non-ES5 environment. 😕 Let's just make one thing clear for future readers:

This is not a general support thread. This thread is merely a discussion about the project's build pipeline and how we should enhance it for the future. If you need assistance with imports-loader, you can always look for examples or open an issue here.

Other than that, you might want to check everything after THREE=three!. This string looks wrong and differs a lot from what I've showed above. Lastly, please note that you can use GitHub to search for code examples, see here.

[backspaces]

Near the beginning of this issue, there was mention of a hack for system.js that will let it import OrbitControls.js (systemjs can import old fashioned global modules) such that it "sees" THREE, thus install itself inside the THREE namespace.

What's the trick?

I use import * as THREE from 'three'; with no problem .. shouldn't I be able to load OrbitControls within the THREE scope? The problem is, naturally:

THREE.OrbitControls = function ( object, domElement ) {...}

.. so I guess I should be able to somehow make THREE visible to an import of OrbitControls, right?

[ghost]

@backspaces Was there really? I think SystemJS uses System.config for setting up globals as dependencies in modules. If it helps, here's an issue on how to use three.js with SystemJS.

Unfortunately, it currently doesn't get any nicer than this, and making it work will require additional configuration. Hence this thread. 😄