Add support for plugins

[jonbullock]

Add support for plugins for getting external content such as Twitter/Facebook and outputting it via the rendering pipeline.

[jonbullock]

Consider hook point for output as well so you could write a plugin to process output, e.g. CSS/JS minify.

[jonbullock]

Should allow tools like this to be used.... https://code.google.com/p/wro4j/

[jonbullock]

Consider JBoss Modules or Furnace from JBoss Forge?

[vietj]

imho you should keep it simple, at least for the moment.

In my various project I use java.util.ServiceLoader for discovering plugins, @PostConstruct for life cycle.

It is dead simple, comes with the standard JDK and works like a charm.

my 2 cents

[vietj]

btw, I am considering embedding JBake in one of my projects (http://juzuweb.org) to generate content at compilation time (using annotation processing) and would really appreciate if it remains simple to embed with just the jar it needs (specially if it uses plugins for decoupling the core from the additional features).

[jonbullock]

Thanks for the suggestion, I'll definitely look into that.

Excellent, I'd love to see JBake used in that way. If you need anything just let me know. I'll make sure this is a consideration when the requirements for plugin support are defined.

[rajmahendra]

I am working on this enhancement. For latest status please find https://github.com/rajmahendra/JBakeProject

[RainerW]

For an plugin architecture OSGi would be nice. Supports versioning, and with bndtools live reloading comes out of the box. But not sure if this suites the idea of an library anymore.

[jonbullock]

@RainerW Thanks for suggestion, plugin support is definitely on the roadmap but as you rightly mention what JBake is will have an affect on how support is implemented.

[mwanji]

+1 for @vietj 's ServiceLoader suggestion. It's a nice, simple way of doing things.

I was wondering how plugins would be integrated whrn JBake is used as an application. Would JARs be put in a plugins folder, and the contents of the folder be included in the classpath? How would plugins manage their dependencies?

[vietj]

ServiceLoader uses a ClassLoader for loading services, so it is a matter of the runtime that setup the ClassLoader to chose either a ClassLoader that load from the classpath or from another URLs with an URLClassLoader or another one. There are several ways to manage dependencies, in the case of CRaSH at the moment it is quite simple with a lookup in a shared plugin context, that has been working well until now (1.3 currently) because I don't have complex service dependencies, I expect this to become limited at some time and at this moment I plan to rework this with graph dependency management and somehow injection (both are trivial software), eventually I can spend time on this and contribute if it's needed.

[aalmiray]

+1 on @vietj's suggestion for ServiceLoader. I can suggest jipsy for keeping those service metafiles up to date.

@vietj: as an aside to your first comment, do you manually find and trigger @PostConstruct annotated methods or is there already a JDK utility that does that for you?

[vietj]

@aalmiray in case of Guice : the abstraction for JCR-330 does it manually indeed as you suspected

[jonbullock]

Thanks I have got a note to look at jipsy when we come to sorting out plugin support.

[RainerW]

Do you really need a plugin architecture inside JBake-core ?

At the moment JBake consists of two things:

• jbake-core * The actuall business logic to generate pages.
• jbake-cli * The commandline interface to use jbake-core directly from the commandline

I would argue, that jbake-core should be a plain library: As classic Library jbake-core should beextensible, but just plain Interfaces as Extenstion points are enough. Having some (DI/Plugin) magic happening inside core will not allow me to use two JBake-core instances with differen configurations etc.

    new JBake()
        .addMarkupProcessor( new PlainTextProcessor(".txt" ) ) 
        .addPostParser( new SitemapCreator()  ) 
        .processFolder( "site" ) ;

On the other hand, the CLI could use DI/Plugins, but this is just an sample application that happens to be shipped. This would not impace my own application that uses n-instances of JBake-core to generate n-different blogs on my server

To sum up: I vote against DI or Plugin stuff inside jbake-core, use plain java interfaces instead.

[lefou]

@RainerW You expressed my feeling about this ticket and the DI/CDI (#72) one very well. A plugin is more than just an API to allow extensions. No need to invent some plugin architecture to simply add some extensions via API. If your extensions need some services, well, simply inject via constructor or setter (if optional), but without any container or framework. It's up to the application, e.g. the CLI, to lookup/detect/find extensions and provide it to core via API.

Just my 2ct, Tobias

[melix]

@RainerW @lefou Let's see what we are talking about today. It's about software that generates a static website. Do we really need CDI for that? I honestly don't think so. But to be honest, reading OSGI frightens me as much. Seriously, who needs hot reloading of plugins in such context?

Let's keep it as simple as possible.

[rajmahendra]

I feel CDI Extension with SPI will be good. annotate the event of parsing,templating, etc something like that (spi and cdi annotations) Else we need to create a implementation interface and pass jbake objects into it (plutin) OSGI i feel its too big to develop on a small scope of jbake.

[vietj]

We are debating about framework approach versus completely integrated solution here.

JBake should remain a plain Java library (you know as framework that you use with Java code and no magic) and then people should create their own wrapper around it (cdi,osgi,spring,guice,whatever) the way they like it in a module of the project or a project of the github organization.

This way everyone will be happy to make its own integrated solution that will not conflict with other, while some people can use it as a simple library.

[aalmiray]

Ye gods! How did OSGi find its way here? no, no, please make it stop :wink: I'm with @vietj and @melix here. Keep it simple, keep it cooking. Loding plugins via ServiceLoader and supplying them a PluginContext or Project or similar should be enough to get this feature off the ground. If further refinements are needed (such as dependency injection) then those will come out after we can kick the tires of the first impl, wouldn't you agree? peace.

[lefou]

@aalmiray I hope you mean the jbake cmdline app, when you say "Loading plugins via ServiceLoader". I think, the core of jbake (e.g. the jbake-core library) should not look for extensions in any way but should provide API to register extensions with it. Btw. this already is dependency injection.

[jonbullock]

Keeping it simple to start off makes most sense, which is why I'd like to look at utilising ServiceLoader to start off with.

[metlos]

my 2c - ServiceLoader is great but sooner or later you'll run into problems of 2 plugins requiring different versions of the same support library and with the flat classpath you're done.

In 1 of my projects, I actually combine the "normal" service loader mechanism - that a "core" library uses to locate the plugins with Forge Furnace used at the CLI level that takes care of (down)loading the plugins and classloader isolation (using JBoss Modules underneath).

https://github.com/revapi/revapi/blob/master/revapi-standalone/src/main/java/org/revapi/standalone/Main.java#L272 - here is the method in my CLI that uses Furnace to load the plugins and then uses their individual classloaders with ServiceLoader (line 298 boils down to a ServiceLoader call).

[mwanji]

@metlos you could mitigate that by requesting that plugins inline their dependencies. That might even be necessary for the command line app.

Jboss Modules could be a good solution, depending on how well it applies to both command line and embedded uses.

That said, this ticket is two years old, I stopped holding my breath for a pluggable architecture a long time ago.

[jonbullock]

Thanks for the advice @metlos definitely look into this.

@mwanji Unfortunately I've not had the time I'd hoped to dedicate to this but I'll get there in the end :)

[metlos]

@mwanji true - if you have the requirement that the plugins inline all their deps into a single jar, you could do even without JBoss Modules. Just use a URLClassloader load each plugin separately and then use ServiceLoader.load(Plugin.class, pluginClassloader).

I used Furnace because it comes with a number of bells and whistles I liked - automatic downloading of deps based on the maven metadata in the jar, actually NOT requiring the deps be inlined and therefore not potentially loading the same classes several times in different classloaders, etc. But as a first iteration, the simple way of doing things with URLClassloader and inlined deps might be enough.

[ieugen]

Hi,

Quite a debate around the plugin feature. I think Jbake core should remain simple and do just what it is supposed to do ( @vietj, @melix, @aalmiray ). There are many things required to do when building a static website but jbake-core should stick to baking pages.

From my experience, building sites involves some of the tasks bellow (depending on the size of the website and how you work ) :

• managing dependencies - HTML, CSS, JavaScript or even other types of static files you put on the web (pdfs, images)
• some sort of compilation or asset processing: minification, uglification, image processing (many versions of the same image)
• generating the static web pages
• moving files around
• packaging the website
• deploying it

Now, if we look at the above list, jbake should do just the generating the static web pages part. I don't believe it should take on the other tasks as there are better and more establihsed tools out there that do that.

Having said that, I'll describe bellow my setup:

Of course, there are a lot of way to solve the above problems. I had success with using Gradle. My setup involves using Gradle to drive the build process and handle all the above tasks and I've used it to empower a less technical person to build and deploy websites.

Both setups use bower to manage front-end dependencies. The first assumes bower is available, the second installs a specific version of node, npm and bower.

The second gist also shows how you can:

• build a tar.gz distribution of your website for delivery
• version that distribution based on the git hash and commits (version releases with git)
• deploy the website to production environment via ssh

The power of this setup is that it is very flexible since I can change almost anything. One added bonus is that, having the gradle wrapper script makes it super easy for people to get started. They just need the JVM installed and the wrapper will download groovy and all required dependencies.

I've tested it out with my brother who does graphics and design and he's working independently right now. All he has to know are a few task names.

Being a build tool, gradle can handle the rest of the list of tasks I mentioned. It can do it either directly or by calling the native tools.

Hope it helps clear some things up.

https://gist.github.com/ieugen/95832f0e5aabea3bda92 https://gist.github.com/ieugen/95832f0e5aabea3bda92

[jonbullock]

What you suggest is certainly interesting and food for thought. The original and still current goal of JBake is to provide an out of the box solution, not just part of a solution requiring additional tools.

Granted the JBake API is not mature enough right now to enable you to pick and choose the functionality you want JBake to provide or not provide you with. But that's something that can be addressed without other user groups, such as the CLI users who may have no idea what Maven or Gradle is, losing out on extra functionality they require.

It is a balancing act for me at the moment when it comes to suggestions of new functionality for JBake, as certain functionality may overlap with other tools such as Gradle and Maven or various plugins. However the plugin/extension system offers a way for JBake to be optional in the functionality it provides. Allowing lines to be drawn in terms of scope of functionality, to define what JBake does out of the box as a solution, to define what extra functionality JBake can provide you with the right plugin in place.

[lefou]

As it looks like, we are not making any significant progress with this plug-in thing. Before any further debate about what plugin- or DI-technology should be used, we should determine which extension points we actually need. Different renderers and readers come to mind. But maybe also some kind of post-processors and the like. Each wanted extension point should be clear about its prerequisites and it's outcome. With such a list, we can try to arrange an API for these extension points. Until this point, we should be fine with pure plain old Java code, no framework, no ServiceLoader and no required annotations. And only after that, we should debate whether we want to provide addtional DI features.

After all, if we stick to plain Java, we can of course evolve our extension system. No need to get it complete and final in the first version. Our main concern should not be extension point version compatibility (at least not yet) but how we get certain features plug-able.

I can see two feature requests linked to this ticket: #19 and #78. Also there was a mention of Twitter/Facebook integration and content post-processing.

@jonbullock Could you list other feature requests, that are in your opinion good plug-in candidates? Are there currently built-in features, that should be moved into an plug-in?

[ieugen]

@lefou : I agree we should define how things interact inside and the interfaces for them. This is the most important part.

Regarding technologies, ServiceLoader is part of the JDK and quite simple to use so I would consider it a first option.

I have started working on some docs to support collaboration

https://github.com/jbake-org/jbake/wiki/DomainModel

[jonbullock]

These are the features I've thought could be addressed via a plugin/extension system:

78

216

223

243

https://groups.google.com/d/topic/jbake-user/64MiZRTl_3I/discussion https://groups.google.com/d/topic/jbake-user/7Ep7vPG6t-8/discussion https://groups.google.com/d/topic/jbake-user/lIF7ExtJtlE/discussion

There is also this page that covers the proposed system: https://github.com/jbake-org/jbake/wiki/Plugin-System

This is a my vision for the system (taken from https://github.com/jbake-org/jbake/pull/78#issuecomment-101652595):

Here's my original vision for plugins/extensions:

There will be a number of hook or extension points in the JBake API, each at various points along the processing pipeline. Such as content crawling & parsing (to include more content than just static files), pre-processing (allowing modification of content before format conversion say from Markdown to HTML), post-processing (allowing modification of the HTML to be rendered), rendering (allowing template engine related alterations), and asset processing (allowing minification of CSS & JS or conversion from LESS to CSS). Each of these points will have their own plugin interface to allow multiple types of plugins, so there can be multiple plugins for each extension point. What each plugin can do depends on the interface for that extension point in the processing pipeline, so at the content crawling and parsing stage the plugin can interact with the content store. At the asset processing stage the plugin will be passed each asset file and it's up to the plugin to decide whether to perform additional processing on the file or not.

I realise this is a proof of concept and slightly different to what I've just outlined however I can see having an init method potentially beneficial. I also like the idea of having a separate plugins folder.

Anyway I just wanted to get my vision out into the public so it can be discussed, changed, fixed or updated. All comments welcome!

I'll also knock up some diagram and post this to the dev mailing list too.

[shalugin]

@jonbullock Is there any news on this feature? All discussions are very old.

[jonbullock]

Not yet, but it's still on my to do list.

[LightGuard]

Reviving the zombies here....

Any push for this, or a planned version for it?

[jonbullock]

An organisation has recently reached out about this too.

My aim was to get some sort of simple plugin support up and running quickly to kick start the discussion and help move it forward. To this end I did start looking at this recently but it's dropped down the priority list again. I'm going to tag it for 2.8.0 so it gets reviewed again soon.