Require.js

[thehydroimpulse]

This has been discussed a little bit and it seems that require.js would be a valid candidate.

I've seen a bunch of projects on github (running node) using require.js on both the server and client. For example https://github.com/chaplinjs/chaplin .

An example file from chaplin: https://github.com/chaplinjs/chaplin/blob/master/src/chaplin/lib/router.coffee

It makes the code extremely clean and verbose while doing it's job. With the ability to lazy-loading all the dependencies, this would add some optimizations.

Though implementing this on the both the server and client would require a lot of revamping (though it may not be that bad) starting with the client could be easier. We could then split the client into several smaller modules which would simplify development on Tower as building the client code would be extremely quick; only building the modules that code within has changed.

http://requirejs.org/docs/node.html

[lancejpollard]

I'm down for this, it would be great for all the reasons you mention (allow for lazy loading modules, would make breaking up the client scripts possible, simplifying development and optimizing, etc.).

Feel free to start a branch and start messing with it. I want to get to this at some point but will probably hold off personally until some more of the API stuff is solid (and 0.4.3 is reached, though I may break 0.4.3 into smaller parts lol, it's getting big). The sooner we start on this the better though, it's going to require modifying the way we write code in Tower to some degree I imagine.

[thehydroimpulse]

Ok sounds good. I'll start with the DiC or Centralization which might solve the lazy-loading. But i'll see if and how require.js would fit into the system.

[lancejpollard]

Sweet, sounds good.

[thehydroimpulse]

I believe I found a good alternative to require.js that would provide an EXTREMELY fast lazy-loaded (by default) system. It's used by many of the ember.js guys, in their development environment.

So right now let's say everything is concatenated into a single file. When a user visits the webpage, the script is downloaded into a single HTTP request. Just downloading contains two parts; latency, and download. On, let's say, an iphone 3, the latency and downloading, accounts for over 1/4th or 1/3rd of the total time.

After downloading a Javascript file, you need to parse it which also takes into account for 1/6th, if not a 1/5rd of the total time. The last part is the evaluation, which takes most of time.

Without Caching

This is all done when loading a page and before any actions can occur.

With Caching

Loading JQuery

As you can see, this is a huge drawback! All this evaluation and parsing is done on EVERY SINGLE PAGE REQUEST!

The Solution

While lazy-loading the downloading isn't the solution, as there are many problems that can occur and you end up paying twice as much in bandwidth then a single file.

The solution is to instead, store each module within a string:

modules = []
modules['Tower.Controller'] = "console.log 123"
modules['Tower.Something.Else'] = "console.log 345"

Parsing strings are a LOT more efficient then straight javascript. This will also lead to lazy-loading of both parsing and evaluation which is awesome.

Without HTTP Caching:

With HTTP Caching:

Comparing

Comparing this new technique with the original, single file, load everything model you get the following data on an Ipad2

Result

You can get speed boosts by a factor of over 10!

This could also be built in conjunction with the IoC system to provide a natural flow and simplicity.

[thehydroimpulse]

Oh and the dependencies are ALL resolved on runtime. You could even have specific dependencies or modules for specific platforms (mobile, pc, etc..). You could stay with the require statement and also form it into the IoC.

# Create a new instance of the IoC container that will store each dependency and each file. 
Container = new Container()
Container.set 'Tower.Controller', Container.file 'tower/controller' # Load contents of file.

# Global objects:
window.Tower = Container.alias 'Tower'
window.App    = Container.alias 'App'

# *.coffee
# Because we previously set a file to load, we can use it in two ways.
require './tower/controller' # This would trigger the Container to resolve this file and parse it.
# Or:
App.PostsController extends Tower.Controller # This would also trigger the Container to resolve this file and parse it if it's not already loaded!

Any thoughts / ideas?

[thehydroimpulse]

More info: http://calendar.perfplanet.com/2011/lazy-evaluation-of-commonjs-modules/ http://tomdale.net/2012/01/amd-is-not-the-answer/

[lancejpollard]

Now that is fucking awesome. Very interesting idea!!

It seems strange that the browser somehow doesn't have this built in, but I guess it makes sense (the 'loading everything as a string and evaluating on some user action' part).

We definitely should test this out and see how it works, at first glance it seems like this could be a pretty hardcore optimization.

Along the lines of caching/optimization, we should consider integrating the HTML5 cache manifest to the equation: http://www.html5rocks.com/en/tutorials/appcache/beginner/.

I'd love to see a test/example of this on the tower codebase (or some other thing I can clone and test) to get a better feel for it. Have you started messing around with the lazy evaluation stuff you're describing yet?

[thehydroimpulse]

I have a working example here that I made really quickly: https://github.com/TheHydroImpulse/resolve.js

I also added the sourceURL option that when appended to code, will appear as separate files in dev tools. So in this example, the controller.js will appear as a separate file in the chrome dev tools.

[thehydroimpulse]

Each file needs to have the following structure (if it's a coffee-script class) to use the previous implementations (as of now): http://jsbin.com/ijiquz/1/edit

module = (function() {
    var controller;
    controller = (function() {
        function controller() {
           // alert('Controller constructor has been loaded.');
        }
        controller.prototype.get = function(route) {
            console.log(133);
        };
        return controller;
    })();
    return (controller);
});

module = new (module())();

module.get(); // 133

The module wrapping function isn't actually apart of the code but when you call: new Function(); it will wrap the code with this function.

[thehydroimpulse]

So the module loader I made works 100% and is quite powerful tbh. You can clone it and see the awesomeness working. I wanted to try loading jquery or underscore with this, but it's hard putting these huge libraries within quotes ".
But i'll try some more and see if it works again.

It also seems that a combination with html5 cache manifest and this loader would explode in speed. I'll try that soon.

[thehydroimpulse]

A little update on the module system here, though maybe this isn't the place for it?

I've settle down with a CommonJS type system. This simplifies everything, because you aren't changing anything from Node.js to the browser. All: require, exports, and module.exports work the same as in Node.js.

An example module:

// controller.js
var Controller, model;
model = require('model.js');

Controller = (function() {

    function Controller() {
        console.log("Controller constructor...");
        this.model = model;
    }

    return Controller;
})();

module.exports = Controller;

As you can see, the syntax is the exact same. I'm currently working on some aliasing and relative module lookup. I'll also add coffeescript support and multiple file extensions. Would it make sense to use the same ./ notion as in node.js for local modules? Maybe you could then define system module, for Tower, let's say.

require 'tower-application.coffee'
# or
require 'tower-application' 

Instead of

require './folder/to/module/tower-application'

I also made a pretty simple Cake task to build all the files. It will search within a specified path, grab each file and stringify the code into app.js along with the module loader. It also uses UglifyJS to concatenate and do some fancy shortening on part of the code (there are some issues with the mangle and squeeze with the system, not sure why.).

Here's what a resulting app.js looks like with two class modules:

(function(){"use strict";var e;e=function(){function e(){}return e.prototype.modules={},e.prototype.loaded_modules={},e.prototype.module=function(e,t){return this.modules[e]=this.appendSource(t,e)},e.prototype.appendSource=function(e,t){return e+" //@ sourceURL="+t},e.prototype.resolve=function(e){if(this.modules[e]!==null&&typeof this.modules[e]!="undefined")return this.loaded_modules[e]!==null&&typeof this.modules[e]!="undefined"?(this.loaded_modules[e]=this.parse(this.modules[e]),this.loaded_modules[e]):this.loaded_modules[e]},e.prototype.parse=function(e){try{var t;return window.exports={},window.module={exports:null},t=(new Function(e))(),window.module.exports!==null?window.module.exports:exports}catch(n){return new Error("Error while parsing module. Error: "+n)}},e}(),window.container=new e,container.module("controller.js",'var Controller,model;model=container.resolve("model.js");Controller=function(){function Controller(){console.log("Controller constructor...");this.model=model}return Controller}();module.exports=Controller'),container.module("model.js",'var Model;Model=function(){function Model(){console.log("Model constructor...")}return Model}();module.exports=Model')})()

Pretty nifty heh? I could probably shorten the module loader by 30% or so, because I'm using CoffeeScript classes written in Javascript (I love writing Javascript after knowing how to properly write out classes and stuff).

I want to unit test the system, but then I'll be able to start and experiment merging this and Tower on my own branch. Though a proper build system is properly needed, maybe I'll integrate GruntJS, along with Cake(just for a shorter command). Cause it's kinda hacky atm.

Repository: https://github.com/TheHydroImpulse/resolve.js

Edit: I still need to think of a way to auto-load specific modules. Sorta like a starting module so you can execute some code. Right now, I'm just using the console in chrome, but I'll need to think about it more.

Thoughts?

[lancejpollard]

This looks pretty awesome @TheHydroImpulse. I don't 100% have my head wrapped around it yet, need to spend some more time looking at the source (not sure how you're handling require('model') yet in the example just briefly looking).

If this works - and with that little amount of code - this will be extremely useful for any project.

Nice name too, resolve.js! Don't forget to register it to npm.

[thehydroimpulse]

Thanks! I'm just cleaning up the files atm, then I'll push to npm :)

require('model') is simply an alias to container.resolve('module'). It's just a simple regular expression replacement for simplicity.

I'm currently adding a watcher ability (using chokidar :)) that would recompile the files. I'm also testing Ember.js into the system, and it's currently going well so far.

I'm also redoing the documentation, and having full specs in the README. Possibly make it easier for people to understand the concept a little more.

[lancejpollard]

Awesome! Really excited to test it out.

[thehydroimpulse]

Just a little fyi here...

I've added a whole bunch of tests and it seems to work in most browsers (including IE 7), which is pretty awesome. I've also added a full build step that is super quick using grunt. It's simply a bunch of grunt tasks and helpers, along with some config options that I'll most likely be moving into a Resolve.json file to simplify things. The build script will minify and concatenate, along with the usual UglifyJS optimizations (squeeze, mangle, etc...). So far there is no lag in building, and it's not optimized at all.

I also added the ability to require modules in three different, simple ways.

• Fully Qualified Namespace
• Semi Qualified Namespace
• Non Qualified Namespace

Fully Qualified Namespace is fairly simple. It's an absolute path with the module extension (js). e.g:

require("/app/controllers/client/postsController.js");

Semi Qualified Namespace is an absolute naming but without the file extension e.g:

require("/app/controllers/client/postsController");

Lastly, Non Qualified Namespace is a relative pathing that has an optional file extension (either way works :)) e.g: (This uses the same syntax as CommonJS (node))

// app/controllers/client/postsController.js
require("./../workingController");
require("./../workingController.js");

I've matched the syntax to the CommonJS format to make it easier to use it along side node. Doesn't matter which style you require modules in, the loader will convert it to a readable format (Fully Qualified Namespace).

I've also been working on a "require call cache" feature. Basically recording every require call and creating a map. This makes relative paths work because at any time you'll have a module.parent object in each module that will output the callee module. You could use this feature to draw an actual map and see all the dependencies live or something.

I've been reading a ton about AMD, cons and pros, and peoples opinions about it. I'm not a fan of it, though, I do agree on some of it's goals, mainly, having a single module format that's universal. But some people might want to utilize Tower in an AMD format, which is why I might implement a "compiler" or "transpiler" to convert the CommonJS to AMD using the "hybrid string" method. This would allow people to decide which method they would like to use in development or production.

With that said, I found out SoundCloud (the upcoming, not launched yet, version) is using the "hybrid string" (the one I'm building) and a CommonJS to AMD conversion. http://backstage.soundcloud.com/2012/06/building-the-next-soundcloud/ and specifically one of the dev's comment: http://backstage.soundcloud.com/2012/06/building-the-next-soundcloud/#comment-557577776

Interesting what people are using in production in large scale systems.

[lancejpollard]

New link. https://github.com/TradeMe/bootup.js

[thehydroimpulse]

Got a new proposition for client side (and server side) module organization.

CommonJS:

// controller.js

function Controller(){
     console.log(this.name);
}

Controller.prototype.name = "John";

module.exports = Controller; // exporting.

I'm a pretty big fan of CommonJS and would rather it's techniques than AMD's. Right now, the client side is going toward AMD support with Yeoman, RequireJS, and a ton of libraries are giving support right now. I'm not too keen on AMD, especially with the new ECMAScript Harmony modules coming out, or, is already out for specific browsers.

SoundCloud is going to the ECMAScript modules and CommonJS. They have a "transpiler" (source-to-source compiler) to convert between module systems. Throughout development, they would use either AMD (asynchronous loading, and the ability to loose the build tool), or ECMAScript modules. When they move into "production" or testing, they would run their build tools, which would convert either systems into the hybrid-string end system. This, in turn, provides huge capabilities for development, such as loosing build tools, that would increase productivity and the speed of testing.

The hybrid-strings solution is pretty good, and I'm in the process of creating a Yeoman generator and tool for it, to where, you can, within a new directory, run the yeoman init command, yeoman init resolvejs, and it would create some configuration files that would hold settings such as, the source folder, the lib, or temp folder, and the output / dist folder. You could then run the yeoman build command, and your files would be watched, built, and outputted. Because you have the temp folder, the first time the build tool would run, it would compile (minify, concatenate) each module into lib, but would keep it's directory structure. When a file is changed, only the single file is re-minified and concatenated, then, every file inside the lib folder would be built in the following way:

modules['currentmodulepath'] = 'encodedcode'; // uses encodeURIComponent to prevent special character messing and escaping the quotes used to surround the code. 

The build task would also place the loader and the needed code with it. Right now, even without Yeoman, it compiles near real-time. Though it's sorta complicated on setting it up because of the statically typed directories, etc...

Because the source is going to straight Javascript, we could loose the bulky build system for development (but have the options to change the module system) and provide an AMD or ECMAScript module system.

We would still need some kind of transpiler to convert to, and from module format. The solution could be within: http://sweetjs.org/

It's backed by Mozilla and kinda by Brendan Eich (creator of Javascript), so it's only going to get better. This would allow one to write in any of the module formats, and the end result could be a different module system.

To use a macro within sweet.js (and this might be in ECMAScript 7... long time away though) you need to define the following:

macro module {
  case $module:ident {
    $(export $x;) ... } => {

        var cmodule = {};
        cmodule.exports = {};
        (function(){
            $(cmodule.exports = $x; ) ...
        })();
  }
}

I'll admit the syntax is a little weird, but it gets easier when you spend time tinkering with it. This macro converts a ECMAScript module to a CommonJS module.

ECMAScript to CommonJS

ECMAScript:

module A {
    export 1;
    export 2;
}

CommonJS:

var cmodule = {};
cmodule.exports = {};
(function () {
    cmodule.exports = 1;
    cmodule.exports = 2;
}());

With the macro, you'll need to have cmodule instead of module. This isn't that much of a problem and will be fixed when I figure it out (only spent an hour or so on sweet.js).

A work in progress, but this could provide a huge boost in productivity. CommonJS in the browser currently requires build tools (well the hybrid-strings solution does), but AMD is pretty nifty in asynchronous fetching of modules.

CommonJS to AMD

var helper = require("some/path/to/helper");

function Controller () {
   this.name = "Nope";
   console.log(this.name);
}

Controller.prototype.name = "John";

module.exports = Controller;

define(['some/path/to/helper'], function(helper){
   function Controller () {
       this.name = "Nope";
       console.log(this.name);
   }

   Controller.prototype.name = "John";

   return Controller;
});

ECMAScript to AMD

module ControllerModule {
    function Controller () {
       this.name = "Nope";
       console.log(this.name);
    }

    Controller.prototype.name = "John";

    export Controller;
};

define([], function(){
   function Controller () {
       this.name = "Nope";
       console.log(this.name);
   }

   Controller.prototype.name = "John";

   return Controller;
});

Some pretty nifty stuff. Sweet.js now has browser support, which would simplify things and it's extremely fast and only getting faster.

Thoughts?