Modules

[voidcontext]

Hi,

Is there any plan, where sails will support modules? Under a module I mean a collection of controllers, models and views.

As I see from the code, it can be easily extend the current hooks to load controllers and models from more directories. Maybe there are more complications, as I'm not familiar with the whole source (yet)...

[tijhaart]

+1

[terox]

+1

[voidcontext]

Just found that #594 is related with this issue.

[devel-pa]

1208 satisfy you?

[voidcontext]

Yes, that looks nice. We'll see how it fits into the current concepts. Thanks!

[rachaelshaw]

@voidcontext tagged this as a feature request

[danielbaak]

+1

[devel-pa]

I stopped to change inside the core of sails because I have to keep up with the changes for v0.10 and is very difficult. I'm working right now on an extension that overrides some of the hooks to achieve that, until now only for controllers and models. As soon the views will be available I will let you know. Paul

[devel-pa]

Forgot to specify that I'm trying to do this node module for v0.10 but the code is changing in sails faster (and radical) than I can adapt the module. Also the views associated with controllers are broken in v0.10:

[devel-pa]

Hyper experimental: https://github.com/devel-pa/sails-apps

[kmcgrath]

Very interested in this feature. I've used this pattern in both Mojolicous and Rails (with engines). In both you can actually mount sub applications, that can run independently or be mounted into a master application.

This results in small apps (modules) that can be developed, tested and run on their own, but also have the ability to load any number of them into a application shell to expose them as a "single app."

I'm just getting into Node.js and I'm working on converting a Rails project with this pattern. It would be nice to not only be able to look at a directory of modules but also use a npm module, kind of like the adapter framework for orm.

Possibily something like:

var app1 = require('my-small-app');

// In some config
config.mountapp = {
    '/app1': app1
}

Where app1 is a Sails module or maybe even small Sails application. If an application, the app would have to know to use the http connection, etc... from the parent app. Like express middleware?

[kmcgrath]

I figured out how to load a sails app within a sails app through config/express.js for v0.10.

https://gist.github.com/kmcgrath/8510253

This is kind of brute force, but does allow you to develop an independent set of models, views and controllers that can run independently or be incorporated into a parent application.

The globals are turned off for the included application, so you need to use require to use models. The parent globals are available to the included app.

Maybe a next step could be to register each app so it is available through the parent sails global? Or maybe this is a terrible way to go?

I created a repo for a ShellApp that automatically includes and ExampleSailsMountApp from a different repository: https://github.com/kmcgrath/sails_shell_app

Just checkout, npm install and raise the sails to take a look.

Comments and criticisms welcome, just messing around with the idea.

[devel-pa]

I'm not 100% sure if you can load two, as in what I'm remembering from 0.9.7, you can use only once customMiddleware

[kmcgrath]

Couldn't you just use one customMiddleware to load multiple apps?

customMiddleware: function (app) {
  var Sails = require('sails/lib/app');

  var app1 = new Sails();
  var app2 = new Sails();

  app1.load({
    appPath: path.dirname(require.resolve('ExampleSailsMountApp')),
    globals: false
  },function(){});

  app2.load({
    appPath: path.dirname(require.resolve('ExampleSailsMountApp2')),
    globals: false
  },function(){});

  app.use('/app1', function(req,res){
    app1.hooks.http.app(req,res);
  });

  app.use('/app2', function(req,res){
    app2.hooks.http.app(req,res);
  });

}

[devel-pa]

Sorry, I agree, I misunderstood something at first, you're right. Anyway, I think that the solution is very good, I have some concerns, but nothing big at the moment. As for the models, I liked the easy access but not the pollution of the global scope. sails.models.foo it's equally good. In the test I've made I got a conclusion: to have global scope for subapps it would be interesting to have in config/globals.js a new parameter domain: 'sails', defaults to sails but it can be renamed to app1 for subapps. In lib/app/exposeGlobals.js instead of global['sails'] = sails; to have global[sails.config.globals.domain] = sails;. And, taa daa, we have now sails and app1 in global scope. The only change from your stuff is the loading:

async.series([

  function (cb) {
    app1.load({
      appPath: path.dirname(require.resolve('ExampleSailsMountApp'))
        }, cb);
      }
  ,
  function (cb) {
    exposeGlobals(app1)();
  }
]);

[kmcgrath]

The config you mention, would that be in the sub app? I don't think the sub app should be able to set it's name/domain or create a global without the parent application allowing it to.

I think the parent application should have full control over how the sub app is initialized and mounted.

I also wanted to try and get policies to work with the mounted application so that the sub app can be protected and/or pre-processed further by the parent, the "sails way".

I'm started tinkering with a couple of other solutions.

1. A core mounts hook
2. A mounts generator

I have a first pass at a core mounts hook here: https://github.com/kmcgrath/sails/tree/feature/mounts-core

You mount another sails application by adding it to the package.json and config/mounts.js in the parent application.

Basic config:

module.exports.mounts = {
    '/app1': {
        app: 'ExampleSailsMountApp'
    }
}

Where the key is the mount path and the value is a object for initializing and mounting the application.

This integrates application mounts right into the core, if they any are defined. Policies could easily be updated to process for mounts as well as controllers.

I did run into some issues with how Sails loads connections directly targeting a application's node_modules directory. I had to make that a recursive lookup through the parent. For instance if both the parent and sub app use sails-disk on a npm install the module will only be installed in the parent's node_modules directory. The sub app was failing to load because sails-disk did not also exist in it's own node_modules directory.

parent_app
--  node_modules
--  --  sails-disk
--  --  ExampleSailsMountApp
--  --  -- node_modules
--  --  --  --  [[empty]]

I didn't know if that same pattern could cause other quirks, so I've also started down a mounts-generator path. The idea here is that you could use a generator to checkout a sails app, build it in full and place it in a mounts directory. Now when the apps are loaded they will have all the dependencies they need within their own node_modules.

I'm hoping to have some good examples of each sometime this week, see which (if any) is a viable solution.

[devel-pa]

As any node package, the sub app should be able to define it's defaults, and yes, the namespace too, overridable from the parent application. I admit that a hook is better, it's what I wanted to make for v0.10, and a hook that is loaded from a node package, not in the core. A subapp should define only part of the settings, that will be overridable by the parent app. sails-disk is an adapter for the database, it should not exist for the subapp and not needed as a dependency. And, node_modules shouldn't be empty, it should contain the local dependencies. Anyway, I think that this feature, if not implemented in core, is big enough to bring a lot of discussions and variations, and possible optimization, until sails rich version 99. Let's find the best way, the simplest way, to be extended later.

[kmcgrath]

I have updated ShellApp using my mounts-core fork of sails.

Sub-Applications are configured in config/mounts.js and protected by policies in config/policies.js (just as you would with a controller).

I'm really digging the pattern, but would like to make sure I'm in line with sails style and implementation. Here is the current diff, any insight is welcome.

[artyomtrityak]

I will start using sails.js when it will support HMVC structure.

[mikermcneil]

Hey guys, would you mind explaining a use case where this would need to come into play? I'm happy to help, just don't really understand the point at the moment. My thoughts are that, if you want to have a modular sub-app, you should keep its code separate so that it can be deployed in a different server infrastructure/cloud and scaled on its own. Then two sub-apps can talk to each other via HTTP (or ws:// I suppose).

But I'm probably missing something- so looking forward to hearing y'all's thoughts.

[devel-pa]

Yes, they can talk, but I need integration. I want Authentication separated from Authorization separated from Contact management (etc...) that can be added or combined at any time in the further as needed. And I'm talking about the UI too, with it's own JS widgets, CSSs etc. Once developed, a module evolves independently. The idea is to obtain modularity at UI/action level, not at a middleware level. As you can see everybody talks first about MVC (as files, not as architecture) and not about services.

Sorry if the answer is slope, I've received an ugly job offer.

[artyomtrityak]

HMVC allows to structure your code not in controllers/, views/ and models/ but modular like users/controllers ./views ./models . If application grows this pattern allows extend it without any complexity issues.

Please check Django python framework.

[RWOverdijk]

@mikermcneil It's like my card @ the trello board.

[albertosouza]

@devel-pa , @voidcontext and @artyomtrityak cant use sails Hook api for this?

I made this ugly example for theme selection: https://github.com/wejs/we/blob/master/api/hooks/theme_system/index.js

In my example i get https://github.com/wejs/sails-we-theme-default ( theme, npm package ) and add as npm package enabled with config in: https://github.com/wejs/we/blob/master/config/views.js and the file.

With that logic I can create multiple templates with different structures, save for npm packages and then download and activate what I want

He uses Sails 0.9 Hooks to load one custom views url and layouts to work as Themes. But sails0.10 had a lot of improvments on Hook system.

[devel-pa]

A modular example, not HMVC, using hooks is done by kmcgrath . OK, I'm pro modularity and anti HMVC, but that's only my choice. Your approach description looks like an ideal one, for me. Then, maybe, the discussion could be: How SailsJS can support node modules that adds new hooks.

[mikermcneil]

@artyomtrityak Thanks, but can you provide a concrete example? I.e. a web app for finding public restrooms that needs HMVC to ___. In general, the feature seems pretty simple to add-- I just don't want to roll out something new, only to find out we need to refactor it later and introduce a bunch of breaking changes.

@devel-pa ok cool- so then one use case is being able to install/build plugins - I have some ideas there- as @albertosouza, hooks are a good option for true low-level access to core. In many cases, you might want a hook plus a generator. Let's keep brainstorming here.

@RWOverdijk Here's the Trello link for posterity: https://trello.com/c/2BOdgmlp Thanks for that-- it helps a lot. I get the benefits for the plugin system- but is there any reason someone would do this besides sharing modules between multiple projects? Trying to wrap my head around all the different potential usages.

[danielbaak]

@mikermcneil To me HMVC (or whatever we want to call it) has the following benefits:

1. Allows me to write plugins (e.g. User mgmt.) once and just pull them into new projects and adapt if necessary
2. Makes it easier for team members to not get in each others way while developing different parts of the application (I know, git provides merge functionality, but isn't it way better if the merges just pass automatically and no one needs to go through the code pecking two parts of a changed file together).

I guess there's other ways of achieving these benefits, but having something as obvious as a child directory that is easy to understand (at least for me). It also allows you to tell new team members things like "the Task Manager module is all yours, take care of it" instead of explaining "you have files x, y, z, and for routing alpha, and if you need to check user rights, modify beta ..."

Hope I'm making sense.

EDIT: my background is cakephp which calls this concept "plugins" http://book.cakephp.org/2.0/en/plugins.html

[mikermcneil]

@danielbaak I also spent some time in Cake, have flashbacks sometimes (mainly about the recursive query syntax)- but all in all, it wasn't so bad, yeah?

In that scenario (number 2), you would oftentimes still share models though, right?

[danielbaak]

@mikermcneil Yeah, it was ok, sails is way better though (in my oppinion).

I guess it's unavoidable some times. I tend to try to extract the models that are used in several places into a "CommonModule" of sorts. I've seen this approach used on the client side in angularjs projects.

[RWOverdijk]

Literal copy-paste of advantages:

Advantages

• Cleaner directory structure
• Grouped files (you know where to find the files for which part of your application)
• Separated concerns (every module can be swapped out)
• Reusability (you can share modules)
• Bundled assets (load grunt tasks for modules to bundle assets with them, allowing you to change, and override default behavior)

For me the biggest wins are the ability to share modules (user management, blog, cms or components to a community driven, pluggable application) and the Cleaner directory structure.

The idea, that everyone can be responsible for part of the application is nice. You can then work on each module on its own, test them on their own etc. This means, that if you, as a community, set up some best practices / guidelines and start building some cool things with shared effort.

[mikermcneil]

I tend to try to extract the models that are used in several places into a "CommonModule" of sorts

@danielbaak which === plugin, imo. So I think I'm beginning to understand. As far as implementation, I think the cleanest way would be to make sails load idempotent (@sgress454 already dealt with a require cache issue today that should make that much easier). Then we could call something like:

sails.load({
  appPath: path.join(parentAppPath, 'api/subApp1'),
  paths: {
    controllers: path.join(parentAppPath, 'api/subApp1/controllers/'), 
    // ...
})

Currently, there are special hooks, adapters, and blueprints directories you can create under API, which allow you to quickly drop in your own private plugins (or to develop plugins for sharing w/ others). Perhaps a Sails "plugin" is a suite of folders, as you guys are suggesting, that make up a fully functional app, minus some configuration. This plugin might contain hooks, adapters, generators, blueprints, controllers, services, or even views. In this scenario, plugins would be strongly encouraged not to include a config directory (unless the config was just providing defaults for custom config within the plugin, but even then, I believe it would be better if it kept its implicit defaults private). As far as including assets, I guess that's fine too as long as they don't depend on grunt. The trouble would start when we wanted to kick off multiple Grunt child processes-- doesn't seem wise. Also, keep in mind we currently expose global models and a sails object. These globals can be disabled, but it makes coding up your app a lot less enjoyable. So it'd be better if they shared a sails object, I think (assuming we're speaking about the private plugin situation)

[mikermcneil]

@RWOverdijk I'm all for smaller modules- I'm just kinda wondering why they all have to live in the same repo/project structure? It's like, what's the point, right?

I'm all for making it easy to build modules by being able to create a file in a folder and get started, but I think that's really just a convenience-- eventually, you'll want to be able to take truly reusable bits and break them off into a separate repo/module so that you can use them in other projects with the standard node facilities (npm install + require). Even if they're private, that stuff still works great (and you can even set up a private npm mirror)

I realize I'm not really offering up a solution here, apologies for that, but that's how I'm seeing it at the moment. I definitely agree that we should provide a really easy way to create plugins within your app structure for development, but I don't think we should encourage that as a "good" or "right" deployment model. Of course, people can always implement hooks to load modules from wherever they like-- I'm more addressing the stance that we take by default as a framework (particularly as a node framework).

So what do y'all think? Would require()ing plugins solve the problem in production? That is, assuming you can still use some sort of mechanism to develop plugins within your app structure (e.g. the suggested plugins directory).

[RWOverdijk]

@mikermcneil I've explained all of that in my trello card. It doesn't have to live in the same repo. It doesn't even have to live in the exact same structure. The point is that you can have it run, register its routes, its controllers, its views and have it do what it does. I'm willing to explain it in detail to you (I'm on skype, you're not I guess).

[kmcgrath]

I'm still working on a "mount" solution in my spare time. I've been trying to keep up with the latest master but it's been evolving faster than I can pace with at the moment. I just pulled the latest into my feature/mounts-core branch and my example ShellApp. There is a issue with sails-disk I need to work out.

My philosophy has been to try and develop this as a standalone hook. But due to the loading of adapters (connections) I've had to touch a very small part of the sails core. Maybe with some help I could get rid of that code.

My use case comes from large enterprise environments. Commonly there are internal apps, external apps, or client specific apps that all access the same core backend APIs, datasources, and systems. The design pattern I follow is to modularize as much as possible. The main app is basically a shell for all of the modules (mounted apps) that is consumes. The main app will set specific configurations and possibly add some small very specific use cases that don't exist in the modules. The modules themselves can also be developed and run independently. They follow their own release cycles and have their own mini-project plans.

At the end of the day I have tiny apps that can hook up routes, models and collections for ticketing systems, monitoring environments, messaging, etc... that I can pick and choose from to implement for the current project.

I'm not married to the path I've started down, but I would like to help where I can with getting modular development rolling with sails. Coming from Mojolicous (Perl) and Rails Engines I like the idea of small mountable apps. It makes sense for what I do.

[edy]

We develope a shop system based on sails. an use case for modules is shown in this diagram:

   +---------+
   |customer1|+----------+
   +---------+           |
                         +
   +---------+         +------------+            +-------+
   |customer2|+------->| shop-system|+---------->|sailsjs|
   +---------+         +------------+            +-------+
                         +
   +---------+           |
   |customer3|+----------+
   +---------+

each customer has its own assets. if they pay enough, they get their own controllers and models. every customer also has its own git repository (contains only assets and their controllers/models). our shop-system is just a node_module (dependency in package.json)

it would be cool to have a native /api-loader where you throw some paths to multiple api-folders at sails and sails would then magically mount them.

[mikermcneil]

Thanks everybody! Let's carry out the rest of this discussion to https://github.com/balderdashy/sails/pull/1522 to avoid duplication