docpad / docpad

Empower your website frontends with layouts, meta-data, pre-processors (markdown, jade, coffeescript, etc.), partials, skeletons, file watching, querying, and an amazing plugin system. DocPad will streamline your web development process allowing you to craft powerful static sites quicker than ever before.
https://docpad.bevry.me
Other
3.06k stars 243 forks source link

Abstract out Plugin Loading #391

Closed balupton closed 6 years ago

balupton commented 11 years ago

Would be cool to abstract out our plugin loading, so other systems can use our plugin architecture, and perhaps even our plugins, without having to use all of docpad.

--- Want to back this issue? **[Place a bounty on it!](https://www.bountysource.com/issues/431951-abstract-out-plugin-loading?utm_campaign=plugin&utm_content=tracker%2F32879&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F32879&utm_medium=issues&utm_source=github).
supersym commented 11 years ago

I have been thinking about the plugin architecture for a while now. There are several angles of approach one could take, but for now, let's stick with what you have proclaimed here as a feature/milestone/goal: being able to use docpad without the core of docpad.

There are a few ways I could envision such a thing, in the end it boils down to, what I believe, a uniform and standardized approach on which other people e.g. third parties would be able to depend upon. Not changing frequently, steady - pre-announced changes / expansion at a incremental rate - much like... publishing an API I'd say. v1 -> 2 -> 3 (very mature) without too much in between. You'd have to make it near perfect/the way we like on basically the first version release straight away. Else, and you'll have enough problems with that as it is, it'll be just too hard for people to keep up with stuff, change their code to hook it up again (fix it) and they'll get annoyed and just ditch it before you can say "n00b".

This does bring me to the facet of what third parties would be. If I take your statement very literal, I'd assume you mean you'd like to see people say hook-up their Wordpress site from their side on with certain docpad (modular) plugin features. This would involve their manual effort primarily and is just as complex as you make your interface. For us to expect the "core team" behind Wordpress, or any other for that matter, to facility a hook to DocPad, you'd need to have made a big name for yourself. Them hooking into Flikr, or Google stuff sure, but I don't think we are at that scale yet and you'd eventually in the long run something like this.

So I think for it to succeed, you'd need to 'lure' the users of those (web) frameworks into hooking into docpad, and you need to make it as easy as 1,2,3 I'm afraid, else you'll loose on the less tech-savy users of those solutions (there is a reason Joomla, MediaWiki, WordPress, whatnot keep making their installation process easier and easier as people grow dumber and dumber I fear).

One way you'd be able to more or less facilitate a hookup, though you'll need a solid backend - might even have a business model upon it make a buck or two - is to provide a web SaaS like architecture where you'd have core components run server-side on easy web-form based selection of plugins, features and what not. You could even discriminate as in free as free beer, or some reasonable fee for the more exclusive parts. I don't really see why people wouldn't instead just get docpad with the core and render stuff client-side themselves, but eventually it just might be a bit too hard for some to use the console and you'd be able to use a more 'friendly' web based UI as a paid service to access it and a central rendering server which spits out desired stuff. I don't see anything wrong with that, though like I said, I think docpad already is quite user-friendly - but maybe not for everyone? Anyway this might be a bit off-topic though but just one of potentially a dozen models you'd be able to pick from and differentiate your time upon, assuming you (balupton) eventually would like to make at least a living out of docpad. If the plan is to generate income some other way, nothing said - but I know balupton spends a large amount of time upon docpad so one could envision a separation between community and official support /service/feature expanded one (enterprise version or membership whatever u want to call it)

Now back to the plugin topic: the point I haven't addressed yet is that of the role of the core system in all of this. A.t.m. the plugins are somewhat tightly bound to the core of docpad as in most require some part of the rendering process to take place, system events to fire and so on. How are you going to emit events from a non-existent system if plugins rely on it? And even if, how are you going to do this without breaking any and all plugins already in use? This requires some serious thinking ahead! (much like you'd need to do when deciding to use a formal API I guess).

These are just my first thoughts. I'll soon try and post some more, I've been forking/depending on a few nice solutions in different areas which should spawn some more brain-farts... I guess I should also write a bit on how I've viewed plugins, the construction of them, pragmatism, feature expansion and what not. In the end, that might lead to a discussion on the desired 'leanness' of the core system, what to integrate (i.e. become part of the core functionality ~ kernel) and when to wrap something up in a plugin (regardless the method to connect/interface the external and internal we decide to use to hook those two up). These in my book would be the kind of decisions the... what do they call it,... supreme ruler or dictator .. Google moment BDFL would make.

(to be continued...)

balupton commented 11 years ago

Thanks mate,

The idea here is so much to make other systems dependent on DocPad, or the DocPad eco system. But rather, for them to be able to load in their own plugins in the same way we do in DocPad.

For instance, project x could use our plugin system to project x plugins, rather than having to code their own plugin loader.

While it would be nice for us to be able to share plugins between these different projects/systems. E.g. share a DocPad plugin with Project X users, and vice versa. I don't think that is a requirement as of yet - but when it does become one, what you say is very onpar with things we must consider.

Delapouite commented 11 years ago

Good points.

I don't know if this should be the focus already. Before clicking on this thread and discovering what it was all about, I wrongly expected the main topic would be something like "How do we split the main (and big) lib/docpad.coffee in smaller autonomous plugins ?"

We could describe docpad as a rendering pipeline, mainly reading files on one hand, processing them and then writing the transformations on the disc. For now there is a succession of actions during the generation process like "parsing", "contextualizing" etc... with corresponding events we can hook with.

So what about making these steps independent, like docpad-plugin-parsing, docpad-plugin-contextualizing so in the end the official docpad release would be some kind of metapackage involving those official plugins?

Just an idea, sorry to hijack the conversation.

balupton commented 11 years ago

Yeah, the issue with parsing, contextualizing, rendering, writing, is they must happen in order. I feel this is the real crux of docpad, as they are all dependent on each other. However, anything that is not those things, should be abstracted out into generic node.js packages. E.g. the plugin loading should just be plugin-engine and have no association with DocPad, besides DocPad uses it. Nother thing to be abstracted out is the configuration loading and handling, which could just be called config-engine among the same lines.

supersym commented 11 years ago

Ah ok. That provides a bit better understanding of your desired approach. Indeed, following that path would ultimately make sense, if only due to the limited capacity of people being able to cope with large amounts of plugin spam :P

Now if you'd just be able to define the requirements in a really loosely bound fashion... Today I was again faced with some thoughts on this subject when working on an idea I had on how I wanted to craft a CLI program in CS. This led to the discovery of some (abandoned I think) solution(s) that currently sits tucked away in a very raw shaped repository outlining the first traces of a more general envisioned 'visually sugar coted' stdout message processing. Ideally I'd see that in a proper english-semantic wrapper to form natural sentences (or close enough) to aid us more in our desire to see/measure what's under the hood of docpad. But also to gain statistics, share information, highlight key (rendering, contextual, instance) related events. Furthermore, I was also writing the readme.md file a bit more along that way, in the end would boil down to a way of listening out on the whole chain of processes triggered by 'docpad run' or 'generate', over multiple occasions (trend), multiple instances (distributed), in parallel running asynchronous code. Phew, that was a mouthful of potential scrabble winners... but anyway, you may have already guessed or known this, the solution to solve such a challenge, would be in the employment of promises and deferred objects to regulate the whole flow and gather data for use in between. Earlier I learned of IcedCoffeeScrpt and today I found Dojo to be using the exact same concept in its own API which was described as being a a consumer-only promise object represents a read-only view. I had a great time reading some parts of the wikipedia article on abstraction while I was pondering the subject a bit and playing with the mediawiki API ^^

I guess your answer, balupton, on stuff eventually being designated as 'single purpose' non docpad affiliated tools also answers the (philosophical) questions I had on how to deal with apps like ender and grunt as both of these could be seen as complete or very mature and stable considering how young node is and how much stuff also gets disbanded. Ultimately, I do feel this holds a strong relation with the npm architecture and nature of open-source though: if there is a need from the community to continuously work on a open-source project, a distro or say jquery would be good examples, in the end the community will deliver the people to maintain and develop this software.

Still I must say, I have already seen a fair share of some very nice nodejs ideas to either be partially implemented or barely reaching a state of 0.1b worthy. Perhaps I should less view it from the perspective of the potential I think it holds, and more as what orginally spawned them - a hacking weekend or simple demonstration after which its intellectual father goes on with real life. Docco is a very good example I think and more of the same counts for its related cousins, all lacking in more then one area to my flavour but some come close.

But, and maybe thats just me, I feel this to be somewhat of a loss...a silent death in the corners of GitHub, never to be developed again, and so I took it upon myself to - atleast for now - to hand-pick these solutions and (as a learning project I guess) try to mold that towards a docpad plugin. Luckily the open-source world (and github) shrive on the principle of forking and more then a few times I found people to be explicitly keeping a repo commit-wise alive out of respect for the maker/tool!

And while I learn more and more, on the CS language, asynchronous functional programming and the node.js landscape (available solutions, which ones I can absorb/fuse with :) I get some more thoughts and try to structure/order them to my best extent possible in here and in gist/github. Having worked on a few draft plugin projects, I think I have a ok feel now on how (cheap ;) some 'plugins' were essentially done - not being a bad word because I feel, as a Arch Linux fanboy, elegance in simplicity to be the ultimate achievement.

My final point, I guess, is this: the opposite also holds true when you consider the amount of effort and time it takes to implement a plugin, to more tightly it's probably coupled with the core and the less portable it will likely be, unless you'd want to take a 'double edged sword' approach both coupling more tightly while still keeping the 'standalone' features/options in tact. A good example would be how we implemented most renderers. You could have just simply 'spawned' process with a single command and run it essentially just run as the CLI command tool using the available argv at hand, but instead choose to expose some of the additional logic and features as configurable options for the end-user to tingle with.

But if you'd really believe in this, and I doubt any of use are ready to go that far, would also break away any remaining argumentation left to justify keeping, say a 'docpad-plugin-stylus', in a own plugin. That as opposed to just leveraging the existing 'stylus' app without that component 'knowing' about any docpad existence (remains dumb/ignorant or unknowing).

Either way, I think it's a safe bet to, at least for now, further organize the core and external components the way you've been doing while, something I'm fairly active in, continuously reviewing its purpose, features, possible improvements, mashing up with other problem-domain related/thematic cousins and, who knows, may even end up keeping several near-death projects alive. Individually weak but together stronger than even and with a punch because it can lift upon the docpad architecture of the document model and it's meta-data, layout, partial interop. In that sense, it would be a real waste not to do this and just merely stick to occasionally publishing a stand-alone engine. And, like I said playing safe, if you are able to consume some smaller plugins into larger abstractions or even core entities, the amount of external plugins and 'unregulated growth' of the plugin base in terms of quantity and not quality, will not be so bad.

supersym commented 11 years ago

Oh and forgot to mention... the cool thing of course about upstream is that you can merge the changes with your own easily if the code base is under active development and you don't want to miss out on free bug fixes and feature implementations. We also all know the trade-off: reliance upon external factors not changing too much so that it breaks your own pieces of software that were built/depending on it. This is why, when choosing dependencies for my plugins, the maturity of the software, if and how it's currently being developed and it's reputation as being a 'respected player in the field'. Personally I can very much live also with the previously mentioned 'great idea never properly implemented due to time/loss of interest' and picking up that ball where they left up. /Super's second-hand car/app refurbishment and sales talk :)

balupton commented 11 years ago

Work is underway at: https://github.com/bevry/plugin-engine

Perhaps we can adopt some things from: https://github.com/flatiron/broadway

balupton commented 6 years ago

See https://github.com/docpad/docpad/issues/658 for more details.