Package system

[olivierlambert]

Hi everyone,

This issue will centralize efforts for creating a plugin system for Shout. It means to be OK on architecture choices, API, how to install them, basic documentation etc.

The main goal is to allow Shout to grow faster and easier, by letting community add features without rewriting the core of the application themselves.

Before getting there, we need first:

• [ ] think about a global architecture to fulfill the plugin capable system (what is the core of Shout? what can be a plugin?)
• [ ] develop a trivial demo plugin: it will help a lot for global design and act as a draft documentation. How do you want to develop your plugin to be plugged into Shout? This will help to drive the architecture.

IMHO, a plugin should be easy to:

• [ ] install (npm i shout-plugin-myplugin)
• [ ] load (reload shout will load all the installed plugins)
• [ ] set (a view on all installed plugin, with the possibility to change a setting of each plugin)
• [ ] develop (trivial demo plugin in a repo in the README file of Shout)

Also, to avoid over engineering, a basic installation of Shout without plugin must work.

My first (candid) question is: is it possible to achieve this goal without rewrite Shout from scratch? Let's think about the cost/benefit of this option :)

[spencerthayer]

“is it possible to achieve this goal without rewrite Shout from scratch?” @erming said a plugin infrastructure would require “overhauling the architecture of both the backend and the frontend.”

It seems like @floogulinc is considering a full rebuild of Shuo. If their team has been thinking about doing a rebuild and if the Shout team is now contemplating a rebuild perhaps right now is the time to combine the totality of forces on a fresh build for Shout 2.0. (Maybe even a new account with a URL like https://github.com/shoutirc .)

[olivierlambert]

I can support the "load" of creating the repo, creating issues label and create/copy various basic files (README, contributing etc.).

@erming and @astorije what's your opinion about this? Is the core of Shout would be very long to re-write with a plugin system in mind?

If yes (I do not answer for you! ), that would be also an opportunity to be the most standard as possible (flow, standard coding style etc.)

If no, I think we'll have to make some heavy changes anyway. So it's only a question of load estimation on you, core devs :)

[astorije]

It seems like @floogulinc is considering a full rebuild of Shuo. If their team has been thinking about doing a rebuild and if the Shout team is now contemplating a rebuild perhaps right now is the time to combine the totality of forces on a fresh build for Shout 2.0. (Maybe even a new account with a URL like https://github.com/shoutirc .)

It appears that the rebuild you are mentioning is far more ambitious than I was hoping it would be. Not only there will be a shift in programming language (when discussing with them, I heard about Go, Haskell, Lua, another one I had never heard of, ... not sure where they're headed now) and in technology (it won't be IRC anymore but Matrix), but also in spirit (their project seems to be aimed at network providers). I might be wrong about many points, but if any of the Shuo team would like to answer, please do not do it here: this thread should be focusing on the plugin system only. Thank you :-)

Regarding namespacing Shout in its own organization, that's my wish since day 1. Let me have a word with @erming if he is willing to change his mind on this (already reserved https://github.com/shout-irc).

As much as I like rewrites and refactors, it appeared to me that they are a unicorn in the open source world. People tend to spend too much time on defining that new best thing and never get to start it. In practice, very few projects have complete refactors. What I think would be best here is to come up with a fairly decent plugin system that can handle most of what we want to achieve. Then, we can start "exporting" features from the existing core to different plugins. We'll quickly reach the limit of our plugin system, but that's fine, we can fix it/extend it later, and embrace the idea of plugins needing rework to comply (which is why versioning of plugins should be considered). By the time we reach these limits, there won't be too many plugins around anyway.

@olivierlambert, I pretty much agree to everything you've said in your first post. One thing though:

Also, to avoid over engineering, a basic installation of Shout without plugin must work.

I am very much in favor or "pluginizing" as much as we can, and shipping Shout with mandatory plugins. This will help compartmentalizing features of Shout, and will be the best possible dogfooding. I know I'm rambling, but I was pretty amazed at how Atom dealt with this: pretty much everything is a plugin, like search/replace, but it's transparent for the user. If we can have something remotely similar, modulo the much much more modest team behind the project, that'd be awesome. There were discussions on the plugin system in https://github.com/erming/shout-roadmap, would be nice to add them here.

Thanks for your involvement, @olivierlambert!

[spencerthayer]

@astorije “What I think would be best here is to come up with a fairly decent plugin system that can handle most of what we want to achieve.”

Yup. That's ideal. However it would be nice if we could build an extensible system that doesn't have limits.

[astorije]

No limits is always better, but going from 99% freedom to 100% costs you always more than the first 80% :-) Let's start to list what are the required features, options, ...

[spencerthayer]

OT'ish: This is likely just ignorance on my part but I've never actually seen a successful enterprise (or other) implementation of Matrix. Are their any websites that are actually using Matrix much less WebRTC?

[lucyllewy]

Some immediate high-level ideas that come to mind for plugin-related functionality:

• plugins need to be able to hook any message type from IRC
• they need to be able to hook any /command
  • there needs to be thought about the potential for command-name clashes with two different plugins - e.g. two /me commands implemented in separate plugins with different implementations.
• coming from WordPress, I like their filtering mechanism where each plugin can register to receive certain callbacks to manipulate data on-the-fly before handing off to other plugins. (Such a filter in WP is for plugins to all have the chance to manipulate a post's content before it finally gets sent to the browser.)

[floogulinc]

@spencerthayer btw, Matrix does not need to be all caps. Matrix is a pretty new protocol, you can read about it at matrix.org.

Another note on plugins: In the rewrite of Shuo we were planning (probably not going to happen now, we are focused more on Matrix it seems) we were going to seriously seperate the client from the server. This would mean having separate client-side and server-side plugins. I was looking at using Polymer as the base for the client and as such, web components would have worked really well for plugins.

But as far as Shout is currently, a plugin system would probably just be one system that works both on the frontend and the backend, seeing as they are pretty intertwined at the moment.

[olivierlambert]

@astorije rewriting won't mean being too ambitious. For me, conditions to start a rewrite are:

• keeping the same language (current core devs are used to it)
• start small but deliver something actually works

For this, we could consider to have a small run to deliver a very basic (but working) core. Then add stuff progressively.

To avoid a full start from scratch, you can just keep the very basic existing code (eg: connecting to a IRC server) and just improve it a little, and start from this point.

[astorije]

Thanks @diddledan, I think that's the sort of lists we need to go somewhere :-)

@floogulinc, even in the current state, I am convinced we still need to have plugins for the server and the client separate. Even though the client and server are intertwined, the code architecture doesn't really allow for that. But I don't know, that's the sort of things that need to be discussed. For example, it seems pretty obvious that server plugins should be installed by the server administrator. However, what about client plugins? As a user, if I want emojis to appear on client (see #572), I should not have to request my administrator to install it! On the other hand, as an administrator, I don't want users to manage client plugins freely if the code is untrusted and lives on the server, potentially in multiple versions, ... Discuss! :-)

@olivierlambert, I have seen way too many propositions for these so-called rewrites and refactors in the open source world, including on this very project, to know that they rarely actually happen. Even though I call for discussion, I'd love to see this started as soon as possible, otherwise too much discussion will lead nowhere. My take on this is that we should add a plugin support to the existing Shout, so that people can start writing plugins ASAP. Then, we can start moving existing features from Shout to plugins, which will somewhat correspond to the rewrite you are mentioning.

I see 2 major topics/questions to answer:

• Management: Do we have to split server-side and client-side plugins? Who installs the client-side plugins, users or administrators? How does one installs plugins? How and where are plugins stored once installed (for example, client-side plugins on the server or a local storage on the browser somehow)?
• API: What is required from Shout to let plugins interact with it? Does it work with an event-mechanism only? What is the contract essentially?

---

Regarding the management topic, I suggest using npm as the plugin registry, but that depends how plugins get installed for the client mostly. Plugins can have a tag or set of tags for discoverability (shout-plugin, shout-server-plugin, shout-client-plugin, ...) and that saves us some trouble. This is ideal if plugins are handled by the administrator, much less if users can freely install plugins... Lots of tools are using or switching to this mechanism as npm becomes more and more powerful everyday, like Chai for example (which is doing this based on other tools as well :-) ).

Plugins must expose the minimum and/or maximum version of Shout that they are compatible with. If version X of Shout offers a new event for plugins to use and my plugin uses it, then Shout in version < X should not be able to use that plugin. A way to store that could be a shout.json files (or shout.yml, or equivalent) required by all plugins that expose these sorts of metadata.

[astorije]

On top of @diddledan's bullet points, plugin should be able to:

• Trigger notifications
• Read and write the configuration file(s).
  • Keys should be namespaced (such as pluginname.key = value to avoid collisions.
  • As much as possible, prevent plugins to mess with configuration items from other plugins
  • (Proper configuration should be defined: at the moment it's user-wide in a file and per-browser in a cookie, probably need to make this better)
• Display items in the settings page(s) to let users edit the plugin configurations
  • Either a settings page per plugin, or entries in the global settings page?
• Access (read/write) the settings cookie?
• Themes could/should be seen and integrated to Shout the same way packages are (maybe using a different npm tag such as shout-theme)
• ... (Will add here)

@olivierlambert, whenever you feel like it's the right time to do so, feel free to create a wiki page here or to PR on the shout-roadmap to consolidate all this and start defining a plan :-) I'm very excited about this and can't wait to see/help work on a first working version!!

[astorije]

@olivierlambert, I renamed this issue "Package system". I noticed "plugin" does not convey the meaning we are trying to give it.

"Plugin" means extra, optional, non-essential features that some people might want to add to Shout. Therefore, when people come up with a feature that we believe should be a plugin, they feel like we are shutting them down because it's not important enough. "Package" merely means a feature of Shout. Whether it's a fundamental atomic feature or something very specific to your installation, it should be a package. Saying it should be a feature or a plugin makes a strong assumption on what we think administrators will need on their instance, which is wrong.

A Shout with 0 packages should work in the dumbest possible way and everything else should be a feature. I made a strong case (I believe ^^) on #153, if you want more details.

Let me know if we can work together to get this party started :-)

[AlMcKinlay]

I think, based on @astorije 's explanation on #153, I'm all up for the package system, and think we need to get it done as soon as we can, because it'll allow us to get more done quickly.

However, I think that something that is a prerequisite of that is modularising the client code, to make it more simple and organised.

Is there a ticket for this? I couldn't find one, but it'd be good to get discussion going on that.

[astorije]

[...] and think we need to get it done as soon as we can, because it'll allow us to get more done quickly.

Agreed. Anything to bring to this discussion that can help with this is more than welcome. I'm thinking starting a wiki page about this might be a good idea. What do you think?

However, I think that something that is a prerequisite of that is modularising the client code, to make it more simple and organised.

Can you detail this please? I'm not sure to understand what you mean here. But if I do, I'd say that I'm not sure this is a prerequisite. I think that anything that is not a deal-breaker is not a prerequisite. I'm not saying that we should rule this out of course, but I want to avoid setting up obstacles if they are not mandatory (so this could be in parallel or afterwards, but non-blocking). Again, let me know with the details (what prevents the package system?) and I'll adjust :-)

[AlMcKinlay]

Sorry, I didn't mean that it is stopping it, but more that it will make it easier. Currently we have 1 large file for all the client code, and if we are doing something as large as a package system, it would be easier to do it in the first place if we split it into smaller module files.

For example, one that handles the cookie and settings. Then we know exactly where to go add the code for allowing a package to tie into the options. Much easier, anyway.

[AlMcKinlay]

Also, yeah. A wiki page sounds good.

[astorije]

Hey @YaManicKill, you're right essentially, but I'm thinking this would come naturally as this project evolves. Maybe I'm wrong :-)

[astorije]

Alright, I've started a wiki page at https://github.com/erming/shout/wiki/Package-system-design-and-plan. It doesn't contain every details that was given here, but it's a start and I'll make another pass.

[AlMcKinlay]

I was having a think about how the install process would go for a package. Here's my thoughts:

• npm install -g shout-package-name
• add "shout-package-name" to an array in the config.js file

(optional idea)

• Have user-specific settings for turning each package on and off

This would obviously be pretty easy to install, and then in the code we can iterate over the list of packages, require them, and run the specific functions (onMessage, etc). Obviously, this would probably need #612 to be done, which would make it really easy to require the modules on client side as well, and this also fulfills the requirement that they be client/server as well, we just have specific commands for server, specific commands for client, and bob's yer uncle.

Thoughts?

[dgw]

I'd like to see Shout eventually support some sort of "Global settings" page that the admin(s) can access to enable/disable modules, in addition to users being able to set their own module preferences. Maybe the globals are defaults?

But that presumes that Shout will distinguish between admins and "normal" users, which is probably waaaaay out of scope for this particular issue.

Which is to say, SGTM the way you stated it. :+1: Bonus points if the installed modules can be listed somehow in a GUI and enabled/disabled (maybe after the next restart) from there. But, again: admin users. :pensive:

[AlMcKinlay]

Well, having "admin users" is actually really easy. We can just add a setting to the user.json

admin: true

But, the problem is having that mean something :-P

[AlMcKinlay]

My initial thoughts on how the package system will work. I want to get started on the initial version of the package system that will only have 1 thing: modifying an incoming message. Let me know if anyone has any thoughts.

https://github.com/erming/shout/wiki/Initial-Package-System-Interface

[MaxLeiter]

Should each plugin have it's own config file or modify the current 'shout config' command? Maybe 'shout config'?