Proposal with working code: separate core from boilerplate. Adopt contrib module system.

[MichaelJCole]

Hi,

This is a proposal to add a core and plugin system to MEAN.JS

My motivation for this is:

1) To get an awesome clean boilerplate for new projects 2) Get updates to core code easily through npm 3) Get new features through modules from an open and thriving community

The TLDR;

Here is a MEAN.JS boilerplate app using a new core and contrib plugin system implemented via npm.

To see the code in action:

git clone git@github.com:MichaelJCole/mean.git
cd mean
git checkout meanjs-core
npm install
grunt build
grunt

Notice package.json has two modules: meanjs-core and meanjs-module

In action:

1. Signup a new user and you should get an email.
2. Click a confirmation link and check your server console.

   Proposal

The proposal is to adopt a framework for boilerplate, core, and contrib code.

To move forward, I propose we:

1) start to refactor core code out of the boilerplate and into meanjs-core 2) add new features (email verification and lost password) to meanjs-core 3) refine the boilerplate for the best end-developer experience 4) experiment with contrib modules and configuration, propose some best practices 5) adjust core best practices as we go

Please send any suggestions or comments! What do you think?

The details

I wanted to add three features to my MEAN.JS app:

1. email verification
2. lost passwords
3. subscription to an email list at MailChimp on signup.

1) & 2) seemed like 'core' functionality. Something MEAN.JS would probably want to maintain

3) is very custom. Probably not maintained by MEAN.JS, but something I could share with other developers.

I created a plugin system in the MEAN.JS boilerplate forked here that uses it's package.json to import modules meanjs-core and meanjs-module.

In action: Signup email

1) Sign up as a new user. An email should be sent to the address you used.

This email came from Mandrill email service account (I'll change the keys shortly). You can config your own email transport in config/env/development.js 'emailTransactional'

The core function sendConfirmEmail is called from the boilerplate in app/controller/users.server.controller.js:

/**
 * Signup
 */
exports.signup = function(req, res) {
  // ...  

  // Then save the user 
  user.save(function(err) {
    if (err) {
      // ...
    } else {
      // ...
      req.login(user, function(err) {
        // ...
      });

      var templateData = {
        user: user
      };
      // Send confirmation email
      req.core.users.sendConfirmEmail(req, user, 'confirmation.email', templateData);
    }
  });
};

In the console you should see:

meanjs-core - users.server.controller.js - sendConfirmEmail...
---> about to send to: test23@michaelcole.com
---> email sent
POST /auth/signup 200 46ms - 254b
Message sent: 250 2.0.0 Ok: queued as 0801A20024B

In action: confirmation link and custom contrib module

2) Click a confirmation link:

This has two requirements:

1. Mark the user's email as confirmed in meanjs-core
2. Sign user up for MailChimp email list in contrib meanjs-module

The route is configured in the boilerplate users.server.routes.js like this:

  // Confirm user's email
  app.route('/auth/confirm/:confirmationCode')
    .get(app.core.users.checkEmailVerification)
    .get(app.core.users.processEmailVerification)
    .get(app.mean.myModule.subscribeMarketingEmails)
    .get(app.core.users.completeEmailVerification);

There are 4 controllers here:

1. Core: checkEmailVerification checks the link. Not ok => go to error page. Ok => call next()
2. Core: processEmailVerification. Update db. Call next()
3. Contrib: subscribeMarketingEmails. Use MailChimp API. Call next()
4. Core: completeEmailVerification. Return 'ok' to client

This let's the end-developer:

• add/remove/remix these controllers to get the functionality they want.
• Get bug fixes through npm.
• Have an ecosystem for contributed modules.

  To add your own custom module

Add a contributed module in your boilerplate's server.js:

// after mongoose initializes db
var meanModules = {
  'myModule': require('meanjs-module')(db, config) // edit config files if needed.
  // add your module here.
};

How it works

meanModules, core, and config are automatically attached to app and req in /config/express.js so they can be used like this:

// Config
app.config
req.config
// Core functionality
app.core.[submodule].[export]
req.core.[submodule].[export]
// Contributed modules
app.mean.[export]
req.mean.[export]

For technical details, the meanjs-core/README.md was my coding plan and notes.

meanjs-core and meanjs-module are pretty simplistic right now. I implemented just enough to demonstrate how core and contrib modules would interact with the boilerplate.

Results of clicking the link

Clicking the link should reply with 'ok'. So clearly I didn't fully implement this yet. But check your server console to see what happened.

[nodemon] starting `node --debug server.js`
debugger listening on port 5858

 NODE_ENV is not defined! Using default development environment

Failed to load c++ bson extension, using pure JS version
meanjs-module - requiring...
meanjs-module - configuring...
Failed to load c++ bson extension, using pure JS version
meanjs-core - requiring...
meanjs-core - configuring...
meanjs-core - users.server.controller.js - requiring...
setting up email
MEAN.JS application started on port 3000
meanjs-core - users.server.controller.js - checkEmailVerification...
checking.  Success!
meanjs-core - users.server.controller.js - processEmailVerification...
updating user object
meanjs-module - subscribeMarketingEmails controller...
meanjs-core - users.server.controller.js - getCurrentUser...
meanjs-module - User subscribed to marketing emails
meanjs-core - users.server.controller.js - completeEmailVerification...
GET /auth/confirm/adsfasdfasdf 304 15ms

You can see console messages of when the different functions are being called.

Comments

So, it's not fully baked, but it's enough to move in a direction.

Todo:

• Core Client-side AngularJS code. My background is server side, but I'll need to find a way to get the email list functionality to work. Symfony2 in php used symlinks to pull in client code from modules.
• Mongoose data models. It would be nice to factor this out of boilerplate and into core.

You read this far, thanks! Click reply and tell me what you think?

[rschwabco]

@MichaelJCole - I fully agree that the core and the various modules that we have should be separated, this looks like the right direction. I want to test this out myself a bit more, and like you suggested explore a "full stack" level modules, so that implementing is easy on all fronts - both server and client side. (Aside: I'd even go as far as suggesting fully "full stack" modules, that allow you to maybe use different DBs for each module, etc.)

[MichaelJCole]

@roieki Ok, great! I'll keep moving forward then. As a way to refine the idea, I'm working on fleshing out those three features: password reset, email verification, and MailChimp

Re: using different db's - would each module use a different MongoDB? or would MEAN support other DB's like CouchDB or MySQL?

It would help separating app and core code if modules could modify the app's Mongoose models at startup. Separating the Mongoose init out of express.js would allow this - module init's could lookup app models and add a sub-document.

Anyways, it's definitely something that needs a bunch of eyes and brains. I'm holding off making more changes like that till we're moving forward together.

[amoshaviv]

@MichaelJCole Excellent work!

Love your ideas and what you've done so far. I have a few remarks:

1. Mean-core - did you take a look at the mean-core module I implemented in our vertical-test branch? it is published to npm not on github yet. The idea is to decouple core functionality from the boilerplate without hurting its flexibility. This is a delicate process since we don't want to obfuscate a code developers will need to change. A nice example is the express.js file, should we include it in core? If we do so developers won't be able to support middleware implementation. For instance adding Socket.io would not be possible since you need to change the express.js file for that. So we need to create a list of files that can go into the core maybe offer some way for overriding default implementation.
2. Mean-users - I think users functionality should be separated to into its own module, the users module is still lacking reset password and email confirmation features, but we can add those in to the next version using nodemailer. This will basically seal the users module feature-wise. However we have the same problem like the core module. what if I want to change it? remove and add stuff? How can we maintain flexibility.
3. Mean-module - love that idea since contributed modules can have a huge impact on development time. However we still need to figure out how to implement it in a way that is not hurting the agileness of the core. A lot of work should be put here.
4. Structure - We have to decide on structure for this move to a framework. Once it is sealed we don't want to shake the framework fundamentals any more. Structure should be our major concern in this transition.

After a dreadful week of being sick, I'm finally back working on the 0.3.1.Trying to implement the mailing features. Once we release this version I think we should arrange a proper discussion with the community about the move into a more structured framework.

Again thank you for your involvement, plz let me know what do you think. Amos

[lonormaly]

How about arranging a meet-up (physical / virtual)?

Shai

On 19 במאי 2014, at 16:12, Amos Haviv notifications@github.com wrote:

@MichaelJCole Excellent work!

Love your ideas and what you've done so far. I have a few remarks:

Mean-core - did you take a look at the mean-core module I implemented in our vertical-test branch? it is published to npm not on github yet. The idea is to decouple core functionality from the boilerplate without hurting its flexibility. This is a delicate process since we don't want to obfuscate a code developers will need to change. A nice example is the express.js file, should we include it in core? If we do so developers won't be able to support middleware implementation. For instance adding Socket.io would not be possible since you need to change the express.js file for that. So we need to create a list of files that can go into the core maybe offer some way for overriding default implementation.

Mean-users - I think users functionality should be separated to into its own module, the users module is still lacking reset password and email confirmation features, but we can add those in to the next version using nodemailer. This will basically seal the users module feature-wise. However we have the same problem like the core module. what if I want to change it? remove and add stuff? How can we maintain flexibility.

Mean-module - love that idea since contributed modules can have a huge impact on development time. However we still need to figure out how to implement it in a way that is not hurting the agileness of the core. A lot of work should be put here.

Structure - We have to decide on structure for this move to a framework. Once it is sealed we don't want to shake the framework fundamentals any more. Structure should be our major concern in this transition.

After a dreadful week of being sick, I'm finally back working on the 0.3.1.Trying to implement the mailing features. Once we release this version I think we should arrange a proper discussion with the community about the move into a more structured framework.

Again thank you for your involvement, plz let me know what do you think. Amos

— Reply to this email directly or view it on GitHub.

[dutu]

with new features being released it would be important for existing implementations to have the possibility to update with new releases...

[MichaelJCole]

Hey Amos, no problem and happy to help :-)

(OMG This post is way too long. Let's meet up to discuss!)

If I was the target end-developer for MEAN.JS, here's my user profile:

1. I build web apps. I'm a good programmer, but I'm more interested in building functionality for people, and less interested in choosing a template system.
2. I'm new to Node.js. Node.js is an awesome opportunity to build things not possible elsewhere.
3. I want to build an app quickly. That means the most default technology choices possible. I don't need the best template engine. I need the standard one. I'll fix it when it's not working for me.
4. I want access to a plug-in ecosystem. Re-using other peoples code builds my apps faster. Sharing code gets me work/builds my business. E.g. Wordpress

So, that's me as possible end-user, and what led me to MEAN.JS. MEAN.JS seems best-of-breed for 3). I'm writing blog posts for 2), and this proposal was for 4).

If we're together on that, then here's a possible Developer Experience to shoot for:

1) Super easy to get started with an app that does something:

• Yoeman generator awesome!
• Example app with users already in it. Amazing!

2) Start customizing by adding in an existing module

• This would be adding routes, modules, and controllers to the boilerplate.

3) Make some tweaks here and there.

• Rewrite an existing controller to do some new work.
• Add a field to the user profile

3) Start building my own modules

• Yoeman generator

4) Update minor releases for bug fixes

• in theory the boilerplate is so lightweight, it doesn't need to change. It's mostly configuration.
• npm update for core and contrib modules
• Easy

5) Update major releases for ???

• Hard. Should have a good reason to update. MEAN.JS isn't Wordpress.

There's alot of features there.

So, with that in mind, your bullet points are a crucial start.

1) I wasn't aware of the core module in the vertical branch. Overall I'm a huge fan of the vertical module approach.

• IMHO express.js is definitely part of the boilerplate. Eventually, every developer will want control of this, no reason to make them fight for it.
• Adding high-level core calls in express.js will help separate and update the code.
• Adding app middleware through the Module API would help with this. The mean-seo module and adding socket.io are great test cases.
• Pulling some things out of express.js might make this easier - e.g. Mongoose initialization.

2) mean-users as it's own module

• Yes, good idea. I can see a developer not wanting users in their app.
• Having users in the boilerplate seems like a quick win for getting started quickly.
• User is also likely to be highly customized for different apps so having the Model in the boilerplate is a good idea.
• An idea was to have mean-core decorate the boilerplate Mongoose user model.
  • mean-core offers controllers and services
  • end-developer picks and chooses what they want.
  • end-developer customizes their own user model.
• My motivation for making a "mean-core" module was to define:
  • 'core' as something the meanjs team maintains
  • 'contrib' as something the larger community maintains.
  • However it's packaged works great.
• Yes, the boilerplate, and all the core modules would have to version together.
• There's also an argument for drop-in replacements to core modules. What if I want to use emails as the login for my users?

3) mean-module and module API.

• Keeping the Contrib module API simple is key.
• Getting contrib modules to update to a new API version is really painful.
• E.g. for Drupal, it's so painful, new releases come every 4-5 years.
• There seem to be a list of services a module can provide. Controlers, Models, Middleware, etc. What's that list for the next version?

4) Structure.

• The only way to figure this out is experimenting. Planning only goes so far. And then everyone wants to do it differently.
• Yes, upgrading is very painful.
• I think it's about managing expectations at this point. Moving to a framework will take a couple releases.
• Going through a beta period would help with expectations.

These are big goals, but very doable - Symfony, Drupal, Joomla, Wordpress all did this. MEAN.JS is a great start, and has alot going for it. We'll just have to work on it a bit.

I see two ways to accomplish this:

1) Solo / Business sponsorship

• What MEAN.IO was doing. There's alot of value in owning the next Wordpress right?

2) Building a team

• meetups, shared list of goals, roadmaps, personal battle plans, irc, and standups for progress.

The second option is alot of work, requires trust and making mistakes, but c'est la vie :-)

[MichaelJCole]

Hey, Amos, you said you were working on verification emails. Looks like were duplicating effort.

I just did an update of mean-core. It added a much improved send function with node-mailer. One cool feature is depending on the function was called it would use default templates in the module or templates in the app.

Here's the sending function:

https://github.com/MichaelJCole/mean-core/blob/master/app/services/email.service.js

[amoshaviv]

Ok, I agree, decisions needs to be made.

I will discuss ownership first, since it kind of impact everything else. Full ownership of the project is not something we desire, I left the mean.io project after an aggressive feud, which surrounded around ownership of the project and its assets, trademarking included. This happened after 4 months of ongoing discussion about the implications of such a move. since I started this project for educational purposes, I didn't want to block other companies ability to contribute to the project. This means that I prefer the educational and social value of the project more than its financial value. This doesn't mean I don't believe in making money, I just think that in this case(and IMHO other projects like node.js) ownership is not the best way to go.

I might mistake saying that in such a determinative manner, but I believe this is a straight answer that should be given to contributors.

Derived from this, is our intention of forming a team, which includes developers willing to devote their time to the project, developers like you :). The operational side of this is not well established yet, but I think the further we get, the more structured it will become. We try our best using different tools for sharing this information with the community. for instance, the trello board is currently the best way of knowing where mean.js is going, but other measures can and should be taken.

To conclude this, I think it would best to set up a skype conversation to fully discuss the roadmap, we've taken some steps in direction in the vertical-test branch, which I believe will become v.0.4.0. But I do think we can still develop v.0.3.x further, which will make a nice foundation for next versions.

I'd love if you could add your email related methods using a PR, and we'll take it from there instead of working on the same thing :).

Let me know what you think, Amos

[MichaelJCole]

Amos, that sounds great! I think that's a huge win for everyone involved, and will naturally attract more collaboration and enthusiasm. It's what attracted me to MEAN.JS over of mean.io

I don't think it's a mistake to say it, I think it's a core value of the project. Saying it will naturally attract energy to make it happen.

Ok, great. Yes, let's definitely connect up on Skype. I'll email you to schedule.

Happily! I'll finish up the email verification feature, and package it for a 3.0 pull request this week. I'm excited to develop the vertical-test branch further.

Hurray!

Mike

[netusco]

I fully agree on that it's also what attracted me to use mean.js over mean.io

[rschwabco]

Hello all!

I set up a Doodle for our first MeanJS virtual meetup. If there are more than 10 people who are interested in participating, we'll probably have to think about a more robust communication solution. I hope to see you all there!

Roie

On Tue, May 20, 2014 at 9:08 AM, netusco notifications@github.com wrote:

I fully agree on that it's also what attracted me to use mean.js over mean.io

— Reply to this email directly or view it on GitHubhttps://github.com/meanjs/mean/issues/68#issuecomment-43646709 .

[MichaelJCole]

Hey Roei, cool! Do you have a link to the doodle?

Mike

On Wed, May 21, 2014 at 11:56 AM, Roie Cohen notifications@github.comwrote:

Hello all!

I set up a Doodle for our first MeanJS virtual meetup. If there are more than 10 people who are interested in participating, we'll probably have to think about a more robust communication solution. I hope to see you all there!

Roie

On Tue, May 20, 2014 at 9:08 AM, netusco notifications@github.com wrote:

I fully agree on that it's also what attracted me to use mean.js over mean.io

— Reply to this email directly or view it on GitHub< https://github.com/meanjs/mean/issues/68#issuecomment-43646709> .

— Reply to this email directly or view it on GitHubhttps://github.com/meanjs/mean/issues/68#issuecomment-43784741 .

Michael Cole http://Powma.com (512) 333-4372

[srinivasarajui]

I am new to open source but want to get into MEAN seriously. Can I Join

[amobrem]

I would like to join the meetup. Thanks

[ilanbiala]

+1 for a better way of meetups, how about Google Hangouts, chat, and video available.

[romainkoenig]

+1 with everything on this post. I'm new to node and to meanJS. I'll continue to watch this project and build my projects until I have enough experience to really join and dive in the project so I can actually bring something usefull

[lonormaly]

+1!!

[useralive003]

i am getting error while cloning i have added

git clone git@github.com:MichaelJCole/mean.git Cloning into 'mean'... Permission denied (publickey). fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.