how to make the Poseidon core hookable and extendible

[vesparny]

I've red the amazing series of Minty by @robconery, as well as his issue on the Ghost repo https://github.com/TryGhost/Ghost/issues/1459

Wondering how to create a real extendible and hookable application, I ended up with the great Scatter module. It's basically a dependency container, like the Spring java framework one.

At this moment Poseidon is built on top of it.

I know @robconery is a big fun of event-emitter. But my question is: How can we extend an application with events? Especially if we need to make developers able to do async stuff?

[robconery]

The way I've always thought about it is that (in Node) an application is a collection of modules brought together to do a certain thing. For instance an eCommerce app would have Sales, Accounting, Fulfillment and Reporting modules (for instance). These individual bits stand alone - but you put them together and they make a Store that sells stuff and makes money.

The way I've been (trying) to make this work is by having the main modules (Sales/Accouting etc) emit their main events and the Store just wires them together.

Given that - the extensibility is already in place from the core module set itself. A plugin can simply leverage the core components and respond to a set of events in a custom way. Poseidon is simply the main wiring. To shift this to our app here - we can split the app out into a module set that makes sense for publishing:

• Authoring (text replacement, storage, suggestions, whatnot)
• Review (any workflow related to content review)
• Publication (versioning, formatting, etc)
• Syndication (api surface like metaweblog, pingers, integration etc)
• Reporting (stats of all kinds)
• Membership (basic login/oauth action - which I have in the queue)

This is off the top of my head, of course. The idea here is that each of these modules emits events, maybe like "copySubmitted" for the Review module or "editionReceived" for Publication. This makes Poseidon a general "wrapper" for these modules that stitches them all together.

[vesparny]

@robconery that sounds great, I'm a big fan of separation of concerns too. The only thing I can't figure out is how to use EventEmitters for dealing with asynchronous code.

Imagine that Poseidon emits an event during the blog post retrieving, I'd like to create a plugin which, listening on this event, could get the post content, query the database, and use the result from the query to decorate the initial content, after that the regular workflow should re-take place.

I'm wondering something like WordPress hook system.

Do you think we might achieve this whit events?

Could you please take a look at Scatter and Scatter wiki ? It's been used developing Hadron, a cool blog engine.

If you look at the code is pretty clean, and looks like a very well structured application.

I tried to shape the first commits around the Hadron and Scatter ideas.

Ciao

[robconery]

@vesparny Scatter looks interesting but you don't need IoC with Node or JS - it's evented so you can hook in and extend as you need. IoC is the opposite kind of thing - you ask the container to give you what you need. I have a rather strong opinion about using IoC with Node :).

Anyway - this is an interesting problem set that I think can be solved with a library like async (or the like) - setting up a hook system like Wordpress's. I don't know if (in your scenario) straight-up eventing would work in the automatic way that Mitch is going for. In other words - if we want to do something "in-line", say transform the post content, then the event chain would need to be tweaked.

Have you played with the async library much?

[vesparny]

@robconery I know that asking the container what I need is an antipattern, but in this case Scatter is taking care of instantiating modules and inject them as needed.

If you look at the code I wrote you can see that each module il decoupled from each other. I know that this sounds weird in nodeland, just because we are much more used to the module pattern.

I played with the async library, but i felt much more comfortable using a promises library like bluebird which is far more fast than q, when or whatever.

You can easily do what you can do with async, avoiding callback-hell as well as leveraging a more consistent way to handle errors.

So your idea is to have a global poseidon object which emits events? and developers would be able to attach functionalities to these events in order to do async stuff right? But how can the system know that all the registered plugins have done their jobs, and it can finally proceed to the next step?

I think we would implement something like Scatter Services

This discussion is getting really interesting :)

What do you think about the code I wrote? At this stage it would be easy to define the right way for proceeding since we still have few lines of code already written, and we are in time for re-thinking everything as needed.

I think the real goal here is to create a simple poseidon-core module, which gives the ability to developers to extend it. The more hook we expose, the more plugins we will potentially have.

So if you have time to fork what I did, and rebuilt it in the rob-way, I'd love this.

@spacemonkey what's you take here?

[robconery]

@vesparny The downside to IoC is that it propels you into writing a Big Ball of Mud. In looking over the code for the project so far, we have 1 storage directory, 1 config, 1 test directory - these files/directories will grow based on all the functionality we have coming in and soon we'll have a BBM.

That's the joy of Node, to me. It's what I tried to point out to the Ghost team - "Think in Modules" because that's NPM and that's Node - going against that (with IoC, for instance) suggests that Node might not be what we want - maybe another platform like Java or .NET. That's not a slam - these platforms/languages work well with that kind of setup.

I'd love to create some separate repos here with my general idea - each of the main modules I mentioned above would have their own - and you could wire them as necessary. They don't share storage, each one is completely isolated. They don't share config - you pass that in as needed when instantiating.

@spacemonkey I'm happy to take a swing if you like - nothing speaks better than code :)

[spacemonkey]

Ok you got my interest :-)

Show me! Actually that’s the whole reason I reached out to you, I understand where you are coming from and would like to see how we can leverage node for all it is really worth, and not turn it into the next mod_php shudder

So you’re essentially saying that the Poseidon Core is essentially the main controller and a really big package.json? One of my biggest beefs with the big monolithic stacks was lack of configuration information between plugins. Does the Core become this getter/setter for environment info, or do all of the separate modules need individual configuration that needs to be passed back to the main app for consolidation?

Just wondering how the 3rd party plugin market would look in this scenario - we want all the add-ons to be using the same database, cache, and so on.

Also thinking about g11n, how’s that going to look with a smattering of modules, as opposed to a single core?

[robconery]

Thanks Mitch - happy to get it rolling. The first thing is to (probably) get away from the notion of "Poseidon Core" - that's what we want to avoid (in my mind). An application in Node is defined by the modules it uses - so Poseidon can (and should) be just that - a collection of modules.

We then have a web app that pulls these in and wires them together - but that's entire configurable based on need.

And yes - I'm wondering a bit how we can do plugins without getting crazy. Theory is fun and all but if you want a WP-style plugin system you have to get meta :).

RE g11n - that could be the responsibility of each module maybe?

Finally, I might suggest allowing 3rd-party folks to (basically) do what they want RE storage. Why, for instance, should we have our membership system in the same storage container as our docs and reports? We could pop stats/logs into Redis, articles/posts into Mongo or Elasticsearch, and leave membership up to User.app or something.

I know - arm waving. I do think it's important to focus on how plugins can be managed in this case and it comes down to an interesting question: can we make an "installer app" that reaches back to a plugin repo and uses NPM to pull down modules? Honestly - I don't know. But I think that would be a fun thing to try this weekend ;).

[vesparny]

@robconery can't wait to see what you have in mind :+1:

[scottwolpow]

Not being as proficient as the rest. I want to see if I understand this so far. Poseidon will be a router of sorts. Grabbing the needed modules for an application's (site) purpose. It will not impose restrictions on storage, languages, front end etc. Do we then have a check list or some other means of what to grab? Or do we just load these into a project'?