Packages, Plugins, Bundles

[thehydroimpulse]

I've started to create a package system similar to meteor and other frameworks. There extra functionality that people would create that doesn't really belong inside the core framework. Things like extending the authentication system, routing, "hot code push" - client side maybe, supporting other custom templating languages that Tower natively doesn't.

These packages would use a specific API to communicate and modify the core and flow of the application. I think meteor got it's simplicity right, though completely avoiding npm, which I don't really agree with, their smart packages are awesome. Adding coffee-script is a one liner, adding an authentication platform is other single command. I think this could be somewhat extremely powerful to users.

Right now I'm placing packages within the packages folder in an example app. Each package is separated inside it's own folder. Each package needs to have a package.js file that bootstraps and makes the folder a package.

Tower.Package.register(function(){

    // Add information to the package.
    this.info({
        name: "Auth",
        description: "Hello, World",
        version: "0.0.1",
        author: "Daniel Fagnan"
    });

    this.dependencies(['routing', 'sockets', 'models', 'controllers', 'views', 'templates', 'core']);

    // Register files the package is going to use:
    this.addFile('client.js', 'client');
    this.addFile('server.js', 'server');
    this.addFile('shared.js', '*');
    this.addFiles('base/', [
        'login.js',
        'logout.js',
        'features.js',
        'security.js'
    ], '*');

    this.init(['server.js']);
});

Very simple, intuitive API, though this is still the first version and many improvements could be made. There are still things that need to be thought through, like handling client-side code. But I'm also trying to fix that issue with a brand new "module system". I think in development, we shouldn't concatenate, nor minify any JavaScript or assets. It should all be "raw". This would speed up development quite a bit. When going into production you could package everything up in a bundle using Resolve.js, Require.js, or Harmony Modules if you which (that's the goal anyways). In production, you'd have a single JavaScript file, CSS file, etc...

Back to packages,

Dependencies: I thought about when packages should be included; before everything starts, after sockets are initialized, before router, after router, etc.. But none are really customizable because each package might have a different need.

I've implemented a "Ready" module. Tower.Ready. Each module, core piece would tell this class when it's ready: Tower.Ready.new('router'). The API needs some work, but it's pretty simple. Tower.Ready.is('router') would return true only if the router has called new on Tower.Ready.

The dependencies are just that. A package would define what system pieces it needs, and the package will only be included and executed until ALL dependencies are safely resolved.

Everything else inside the configuration file is fairly straight forward.

You can get all the current code at my fork of Tower within the packages-feature branch(default): https://github.com/TheHydroImpulse/tower

Let me know what you guys think!

[lancejpollard]

Damn @TheHydroImpulse, making serious progress, pretty sweet. Any chance you can post a list of the different events/states/phases/etc. that we would want to provide dependency load hooks for? That would help finalize it.

Random thoughts...

My first impression is that having this sort of API would definitely be powerful for:

• hooking your module/package into the load process
• lazy loading your package's internal dependencies (using that string/eval bit you posted if possible!)
• having a consistent/easy way for defining both client and server dependencies

Not sure exactly what you mean when talking about concatenating/production/etc., but I think all of that should be done with Grunt from now on (moving the tower-application/server/assets.coffee code to grunt for example). Development should load raw files, but compiled languages like coffeescript/stylus/etc. should be compiled to the target language first, even in development (until error handling is exact). But the compiled coffeescript should stay in individual files during development (direct 1-1 map), only concatenating in production.

I'm also thinking we don't want to create a package management framework that's separate from NPM. What advantages does Meteor's package management system have over NPM, or rather, what makes it awesome from your experience? (I haven't used it).

The main thing is, developers should be able to create a server-side only, client-side only, or server-and-client module/plugin using the default tools they're used to (npm on the server, nothing, or jQuery, on the client). Then essentially this package API would make integration more powerful, something like that.

I like the way you've described it:

a specific API to communicate and modify the core and flow of the application

NPM can't really do that. So I'm wondering what your thoughts are of how this would play alongside with NPM.

The last thing is I'm starting to move to simpler/shorter APIs wherever possible. Specifically I'm thinking:

• Minimize the use of methods on namespaces. Only use a method on a namespace if the method is very specific and only used on that class/object, such as Tower.Model.field (which I'm even thinking should be simplified to just Tower.field, or at least aliased).
• Make the methods chainable wherever possible. So you could do this.addFile('client.js', 'client').addFile('server.js', 'server')...

So instead of Tower.Ready.x, just Tower.ready. I'm doing this with validators, operators, modifiers, serializers, and a few other things right now in getting this better model api going. I'll be jotting notes down for the new API here (though it's just notes, only some of which has been implemented: https://gist.github.com/b6c811c7a27ee944317d). The goal is to merge express and ember api's where possible.

Also might be possible, where the Tower.Ready.new and Tower.Ready.is both run the callback Tower.ready based on the state of the application. I think jQuery does this - so if you lazy load a jQuery plugin, and it has $(document).ready, it will still execute.

Very excited to see this coming along.

Thoughts?

[thehydroimpulse]

Awesome, I love where this is going....

So first, right now, grunt, even in development, within Tower's source is concatenating and sometime minifying all of the client-side code. If a single file changes, grunt re-calculates everything, not just that one file. I solved this problem by having a "temp" folder, or a lib folder for client-side code. When grunt starts up, it builds and compiles everything. Building would minify all the client-side scripts and clone the filesystem. Meaning if we have files like: packages/tower-application/client we would get, as a result, tmp/packages/tower-application/client/. This allows the use of require and not having to worry about whether the path changes when building. <-- 1:1 mapping as you said.

Compiling would be concatenation. But first, when a file changes, only that file is re-minified. Concatenation or compiling would be different. All the minified files, even if a single file changes, would be concatenated if were in production or something. My ResolveJS build system has maybe a 1/3 a second of lag, but that's not even noticeable.

Not sure if it's just me, but having only the copy:js task has a noticeable lag of 5-7 seconds. The task isn't laggy, but the watch trigger is. When you edit a file, grunt's watch task doesn't trigger until 5 seconds later. But that's an aside point.

I think having coffee-script support as a Tower package, like meteor does it (again, another example from meteor. I think they do a lot right, and we could learn from them). But meteor doesn't compile the coffee-script (not a 100% sure, but that's what it seems) and so they don't have some of the complications we do. Though the Tower's source is a bit different.

I personally have never used meteor's package management system, but from what I've seen it has a few advantages. NPM is strictly for server-side code, that's it, though you can have client-side code, and assets, someone using your package wouldn't know how to leverage those components. I think one of the node.js/npm developers wrote a blog post on how NPM should leverage these components, but nothing has been set.

I think Tower could leverage NPM and abstract it to provide a powerful package management but uses NPM for the actual fetching of packages. We could have tower package install auth, as an example, the authentication would most likely come with Tower. The result would be to create a new tower-packages.lock or something, and provide a json list of packages installed. We would then simply call npm and install the package. We could, however, have aliased names for packages. That means, one could type: tower package install righton, but that package might be taken in NPM, so it could be called righton for tower and righton-tower-package in NPM and git. Just some thoughts, but it could work. The package system would then look for a tower-packages.lock and see all the npm installed tower packages. Though one could create a vendor/packages folder and create an application specific package.

We would still use NPM, because it's an extremely powerful system, but we would alias it behind our own command.

The goal is just like you said, a single API to provide a way to create a server-side package, or client-side package, or both. With the ability to have assets and other components.

NPM can't really do that. So I'm wondering what your thoughts are of how this would play alongside with NPM. I think the mix of our own solution in combination of NPM could solve this. Though there could be a better way down the line and we could change things if it's not going to break anything.

I also think this could make contributing a lot easier. They wouldn't have to spend hours studying the codebase and figuring out how everything works. We could provide a nice, powerful API and heavy documentation that goes along with it. One could provide html5 boilerplate instead of bootstrap, or in combination.

I think Tower.ready is better, much better. To set a component as ready, you could call Tower.ready('routing', true) and when checking if a component is ready you could call:

Tower.ready('routing', function(){
  // Stuff here....
});

Basically, each package would "essentially" be surrounded by this code for each dependencies they have. package.js

this.dependencies(['routing', 'routes', 'sockets:connected']);

app.js

(function(){

   var Class = (function(){
         ///....
   })();

}();

would theoretically become:

Tower.ready('routing', function(){
   Tower.ready('routes', function(){
      Tower.ready('sockets:connected', function(){
           (function(){
                 var Class = (function(){

                 })();
           })();
      });
   });
});

Again, theoretically. The package system would simply wrap callback.apply({}); with all these Tower.ready statements. A simple, but elegant solution.

One could still call Tower.ready synchronously, and it'll all work.

Hooks:

I'm still trying to think of all the hooks, events, ready states. So if you have any suggestions just let me know. I've made a wiki page, so that we can keep a list of hooks, ready states, etc... https://github.com/viatropos/tower/wiki/Packages

Also, the API inside package.js would be chainable. Just for sake of simplicity.

Also, I'm writing everything in JavaScript, not sure why, but I just really like making APIs in JavaScript compared to CoffeeScript.

[lancejpollard]

Compiling Tower's client-side source code

I get that rebuilding the entire dist/tower.js for the client is slow, but kind of confused from the first part about the tmp file system. If the goal is to speed up development, one other approach would be to make it so the tower-<module-name>/client.coffee file only has a bunch of require './x.coffee', and then use that to create a set of <script> tags using grunt. Then we just load all of the tower files individually on the client during development; there won't be any require calls b/c grunt will convert them to <script> for the browser. That would be simple and would speed things up. Maybe a short term solution.

The copy:js task should only be run once, when you first start running the watcher for developing (or contributing to) the tower source (in coffeescript). But once the watcher is running (make watch), the copy:js command shouldn't run again. So that 5s lag shouldn't be an issue if I'm understanding correctly, but if you can speed it up that'd be cool too.

Also not sure why the grunt watch task takes 5 seconds to run every time you save a file, that definitely shouldn't be happening. It should, however, be the case if you are talking "recompiling the dist/tower.js file for the client every time you change any file in the tower src", because it's concatenating all the files from scratch and then compiling the coffeescript to javascript...

In the end, at least for the simple solution to making development of the tower client-side src faster, I think the following should work:

• make tower-<module>/client.coffee only contain require statements
• compile those require statements into an array of <script> tags
• load those script tags into the browser.

So right now, the tower-controller/client.coffee looks like this:

require './shared'
require './client/controller'
require './client/actions'
require './client/elements'
require './client/events'
require './client/instrumentation'
require './client/states'
require './client/flash'

Tower.Controller.include Tower.ControllerActions
Tower.Controller.include Tower.ControllerElements
Tower.Controller.include Tower.ControllerEvents
Tower.Controller.include Tower.ControllerInstrumentation
Tower.Controller.include Tower.ControllerStates
Tower.Controller.include Tower.ControllerFlash

But if you put all that Tower.Controller.include code into another file, then you would end up with this:

require './shared'
require './client/controller'
require './client/actions'
require './client/elements'
require './client/events'
require './client/instrumentation'
require './client/states'
require './client/flash'
require './client/includes'

Then grunt will watch for any changes to tower-*/client.coffee, and compile the script tags:

<!-- the contents from `shared` ... -->
<script src="/tower/lib/tower-controller/client/controller.js"></script>
<script src="/tower/lib/tower-controller/client/actions.js"></script>
<script src="/tower/lib/tower-controller/client/elements.js"></script>
<script src="/tower/lib/tower-controller/client/events.js"></script>
<script src="/tower/lib/tower-controller/client/instrumentation.js"></script>
<script src="/tower/lib/tower-controller/client/states.js"></script>
<script src="/tower/lib/tower-controller/client/flash.js"></script>
<script src="/tower/lib/tower-controller/client/includes.js"></script>

And if there is just some single hook in the server-side templates, then you can put into coffeekup a helper such as javascripts towerSrc, and it would expand to all of those script tags.

That would mean this:

• individual files are compiled one at a time for the client (just as they are on the server now)
• it's fast, there's no recompiling all of dist/tower.js at once, which is super slow like you're saying
• we don't need another package management system

At least for the short term.

Package management

I love the idea of unifying the client/server package management system. I am not a fan of creating another package management system specifically for Tower. I am a fan of creating a generic client/server package management system for JavaScript, which definitely seems like the direction you're going. When it comes to package management, Tower should use what the community uses. Right now there is a nice/standard node package manager, but no nice/standard browser package manager. A lot of people have tried, some include:

• https://github.com/twitter/bower
• https://github.com/bpm/bpm (I think this is what Ember uses, or used to use)
• https://github.com/bpm/spade (bpm may have evolved to this)

I started integrating bower (that's the component.json in the tower repo), but it's not perfect yet.

My thinking is, you're digging really deep into this package management stuff, and if you find a solid solution like you're getting too, I'd create another project like your resolve.js and make that a generic client/server package manager.

But I also think that it would be easier and more adoptable to develop on top of NPM if at all possible. I'm tending toward not having tower package x mainly because it starts creating some sort of "vendor lockin".

Potentials for integrating client-side stuff with NPM are to either:

1. add custom config variables to package.json
2. add custom run-script hooks that tower looks for
3. add a file such as tower.init.js, or your tower-package.lock

Personally I am most a fan of the first one, because you could basically just define browserDependencies, and maybe an order to them, and then w/ grunt (watching a generated tower app), it would check each package in node_modules for browserDependencies, and compile them to vendor/javascripts or public/javascripts or whatever. This way, people can build libraries for tower without any additional knowledge.

Then to hook into customizing the app startup or whatever (the hooks), if there was a tower.init.js (similar to how you add the railtie.rb file), tower could load that.

I'm also a fan of what you're saying, that this thing would wrap your modules with the callback stuff. That's clean.

[thehydroimpulse]

I guess we have a few options here... Some might work, others might not....

So I guess for development we decided that we should serve raw files. We would just have a ton of script tags, and that's perfectly fine as we don't need to compile anything (except coffee-script).

Now it's deciding or finding the perfect way for require statements. I don't think we need any "module systems", we just need a way to communicate between modules.

Solution A:

We could have Tower become a bunch of packages in the way they are already. We'd have to modify them a tad bit, to follow a packages API, but this could have increasing flexibility. This means, the ready states would become each package. Each package, though, could have sub-ready states like: sockets:initialized and sockets:connected, purely as an example.

Independent: Packages would have to be independent from each other. One would need to have dependencies, that's where the package.js comes in. You would specify other packages you need. controllers would need models and models would need sockets or something.

Client Side: Because we don't have any require statements anywhere, we could follow the package.js file, read all the client-side files in each module, and concatenate them. Concatenating isn't slow, minifying is. Concatenating smaller files also isn't slow and would be 50x faster then concatenating a single tower.js file. That means, each package would return a single file for the client-side.

packages/controllers/client.js
packages/controllers/shared.js
packages/controllers/client/pattern.js
packages/controllers/client/events.js
packages/controllers/client/notifications.js

Would return a single:

lib/controllers.js

Or:

lib/controllers.client.js

Depending on the naming.

We could also have packages decide how it's concatenated:

Tower.Package.register(function(){

    // Add information to the package.
    this.info({
        name: "Controllers",
        description: "",
        version: "0.5.0",
        author: ""
    });

    this.dependencies(["models", "sockets", "application", "http"]);

    // Register files the package is going to use:
    this.addFile('client.js', 'client')
         .addFile('server.js', 'server')
         .addFile('shared.js', '*')
         .addFile('client/events.js', 'client');
         .addFile('client/notifications.js', 'client');

   this.build('client', function(){
       this.file(this.default, [
             'client.js',
             'shared.js'
       ]);

       this.file('controller.events.js', [
             'client/events.js',
             'client/notifications.js'
       ]);
   });

});

This API is completely off the top of my head, but might work. With a few modifications, I think this could work. This would eliminate the need for require calls completely (client-side) and we wouldn't need any module system!

Server Side: The server side could use the require statement, so there's no change there, and the build API would only be for the client-side.

The second thing would be for custom packages. I think we could drop the tower package install ___ and replace it with raw npm. Right now, the package system will look in the packages/ folder and will look in each sub-folder. It only view it as a package if theres a package.js file. We could have app specific packages put inside vendor/packages and npm install packages in node_modules. We wouldn't need to have a tower-packages.lock or anything, though we could, and that might speed up the process a little bit, depending on how slow filtering through each node module.

The build process would be the exact same for custom packages. They could have the same this.build() method, if they want to customize it. Assets would also need to be registered as a file: this.addFile('images/*, 'client');. The "builder" would copy the assets invendor/imagesfor images,vendor/stylesheets/for css files, etc... It would also concatenate the client javascript and place the resulting package's javascript invendor/javascripts`.

We could have a packages.lock json file that would, when building each package, write the package, and it's outputting JavaScript files' location.

So for controllers package with a default build, you would get this result packages.lock

{
  [
     "controllers": [
         {"type":"js", "src": "/vendor/javascripts/controllers.client.js"}
     ]
  ]
}

The coffeecup helper would look at this file and make the script tags for all of them, in order of occurrence.

From I've explained here, I think this could work, but not having used this system I can't say if it's effective or not. Having Tower's source a bunch of packages isn't necessary, but it would simplify things and make things consistent.

Thoughts?

[thehydroimpulse]

The only noticeable downside would be order of dependencies and to decouple everything as much as possible. Or make sure that order is right.

[thehydroimpulse]

Another solution, instead of concatenating, it would order the script tags in order of file inclusion:

this.addFile('client.js', 'client');
this.addFile('shared.js', '*');
this.addFile('client/controller.js', 'client');
this.addFile('client/events.js', 'client');

Would result in the following script tags:

<script type="text/javascript" src="vendor/javascripts/controllers/client.js"></script>
<script type="text/javascript" src="vendor/javascripts/controllers/shared.js"></script>
<script type="text/javascript" src="vendor/javascripts/controllers/client/controller.js"></script>
<script type="text/javascript" src="vendor/javascripts/controllers/events.js"></script>

I think this solution would be MUCH better then the previous. This would eliminate require statements completely, now relying on the order of file inclusion.

Thoughts?

[thehydroimpulse]

So right now I made the packages system look in these three default locations. You can customize it by changing package.json's tower.packages.lookup array.

[
  "vendor/packages",
  "node_modules/",
  "packages/"
]

With the current amount of node.js modules within node_modules it's not slow whatsoever. We could also check the global node_modules but that would get a little tricky and OS specific, unless NPM has a helper for that. But for now it's all local. We could have a tower search or something to check if a module is tower specific, or you could list the currently available modules either in the command line or on the website/wiki.

I think leveraging NPM is the best solution, because each package could have node module dependencies that they could bring in, or bower dependencies.

[lancejpollard]

Looking good.

I like the idea that ready states are the names of each package, and you can have sub-events like sockets:initialized, that's solid.

Packages would have to be independent from each other. One would need to have dependencies, that's where the package.js comes in. You would specify other packages you need. controllers would need models and models would need sockets or something.

Is there a way to do this with just package.json? The "other packages" could be tower- or non-tower-specific modules, installable from NPM, so instead of a package.js specifying the tower packages, could we just figure that out by inspecting the packages in dependencies and devDependencies, etc? That would be cool.

You could also add a boolean to the package.json, such as "tower": true, to aid in the traversal. This seems like it would be equivalent to adding a package.js to each sub-package.

For the package API, what if it was just a separate client/server git repo, maybe packager.js, so the api was this:

packager.register(function(){
  // same stuff you posted above in Tower.Package.register
});

This would make it usable outside Tower. I would love using something like this, that worked the same on the client and server. Maybe for both node.js and browser support:

// so it's global both in node and the browser
var root = typeof window === 'undefined' ? global : window;

root.packager = {
  register: function() {}
};

Also starting to wonder if we can somehow make this packaging stuff a grunt-contrib package. This way if your project uses grunt to do all its build stuff, we can just look for some grunt function such as grunt package:client and run that, which would output a package.client.js and package.server.js with the packager.register code in it. Just throwing out ideas.

I'm onboard with the things you've described in the last few posts. Let me know when I can mess around with it. Going to do a bunch of API stuff today. Cheers.

[thehydroimpulse]

Is there a way to do this with just package.json? The "other packages" could be tower- or non-tower-specific modules, installable from NPM, so instead of a package.js specifying the tower packages, could we just figure that out by inspecting the packages in dependencies and devDependencies, etc? That would be cool.

For the dependencies you could, but you'd also need to have a package.js somewhere for each package. Especially for the client-side, it'll take care of file inclusion, building, etc... I personally like the programmable approach to a "settings" file, but for the basic settings, such as name, description, even dependencies, we could place that inside the package.json.

We could have that tower: true in the package.json for each dependencies or devDependencies. The package,js is to provide a more advanced configuration file for each package. They can include NPM modules, with a this.use(); statement, file inclusion (mainly for the client-side, but since were going for consistency, we can do the same for server-side), etc...

package.js is purely the first idea that came up, we could use packager.js or anything else. I think having the Packager or packager variables their own global. That way people could just include the packager system, and they'll be good to go. I'll make it as customizable as possible, that way people could use it with Tower, or their own projects and they'll be able to customize hooks, ready states, etc...

Yes, packages will be for both client/server. They could either wrap each file within a package with: packager.ready(function(){}); and this would fetch the dependencies in the package file, or they could add extra dependencies as a parameter. In Tower, though, they could leave this little bit of ceremonial code and leave it to the bundler. This would fix some of the dependency issues with having a ton of script tags. Just an idea.

I'll start experimenting with all these ideas, and possibly get a demo out.

[lancejpollard]

Sweet, this sounds great. Yeah calling it package.js makes sense.

[thehydroimpulse]

Just wondering, when you place javascript files inside vendor/javascripts do they automatically copy themselves in public/javascripts/vendor/? Or is it just when you start the server, does it copy?

Right now, the bundler system works really well. It places all javascript files for each package inside: 'vendor/javascripts/packages/{name}`. It also adds to a json file for each file we add in those folders. You could search through each file recursively to find all the files and create script tags for them, but that wouldn't be extremely efficient.

[lancejpollard]

When you start the cake watch process, the vendor/javascripts are copied.

[thehydroimpulse]

Ok that's what I thought.

So far the packages are in good progress. I even successfully made a coffee-script package that compiles coffeescript files in any other package and serves them appropriately. Most notable in client-side code, right now, every client-side javascript file in packages are served as is (raw), so compiling each file independently, when it changes is extremely fast. But it was more of a test then anything else.

I was also thinking of aligning the package system as similar as meteor's packages. Though the API in the actual packages will be different, that we cannot change, the boilerplate stuff could be identical. Might open some doors to package transferring between both frameworks.

I'll have a working demo sometime soon, not sure how I'll merge my stuff as I'm writing it all in JavaScript, and Tower's source is staying in coffee-script, so you can just clone my fork of Tower until I figure it out.

But we'll see were it goes.

[thehydroimpulse]

Example coffee-script package: https://github.com/TheHydroImpulse/tower-coffee-script/ . Just a demo of a fully working compiler for packages. In any package (when using the coffee-script package) you're able to write coffee-script files. When a file changes (within a package) the file will go through the bundler. The coffee-script package simply extends the bundler by registering a new extension and processing each file when it changes, added, etc...

[edubkendo]

@TheHydroImpulse have you checked out https://github.com/component/component btw? might be useful