Provide a mechanism to register plugins after boot

[danielo515]

Hello @Jermolene ,

Since you have expressed your will to implement a system to register plugins after the initial boot I'm opening this issue as a reminder and a formal petition.

Here is the original thread:

https://groups.google.com/forum/#!topic/tiddlywikidev/pmYJYEwKruc

This feature would make possible some awesome configurations, specially for sync adaptors.

[danielo515]

I have been thinking about this and it could be a bit hacky. Because plugins may need to execute some code at startup, and relies on the runbefore and runafter ordering methods, a second round of plugin registering may not be possible. From my point of view there are several options:

• A plugin could expose a new module type: plugin-loader which only responsibility is to provide an array of plugin tiddlers. This modules should be idempotent and without side effects. They will run before any startup or registration process.
• Boot JS could check for plugin tiddlers stored on local-storage. This will require a reboot, because at first boot any plugin or method will leave plugin tiddlers under certain namespace and after reboot the boot module will load them.
• Use a workflow similar to the previous one but using a web-worker and a cache. Maybe the boot could install a worker and allow other processes to communicate with it to save plugins and maybe other stuff.

I would love to hear some input about this, @pmario @Jermolene @matabele @twMat and why not, @felixhayashi

[tobibeer]

@danielo515, this being a very fundamental decision, before we discuss any solutions and investigate strategies as to how to implement a mechanism that allows to load plugins after boot / startup, lets perhaps look at this from more of a birds-eye view first:

Why

• What are the goals to be achieved with this functionality (conceptually)?
• The answer is not: because I want to load plugins after boot / startup ;-) ...but rather why you want to load them later, and perhaps why you want whatever your reasons are for that.

What

• What applications, processes, workflows are then possible (that are not now)?
• What capabilities would this deliver, "good" and "bad".

Need / Problem

• What are the limitations of the current plugin loading mechanism and what do they prevent you from doing and how?
• How is this mechanism more beneficial than the core plugin loading mechanism on startup?

Risks / Threats

• What are pitfalls that can come with such an implementation?
• What (higher) security vulnerabilities might this introduce in a server environment?
• What technical issues are to be expected (and how to prevent)?

[danielo515]

Why

The answer is not: because I want to load plugins after boot / startup

because I want to load plugins after boot / startup 😛

What

What applications, processes, workflows are then possible (that are not now)?

You will be able to:

• Install and try plugins without reloading
• Load plugins from the server dinamically
• Load plugins from sources that are not the html file, for example local-storage

Need / Problem

What are the limitations of the current plugin loading mechanism and what do they prevent you from doing and how? Currently you have to download a new copy of your wiki. So it is impossible to install plugins to online wikis.

How is this mechanism more beneficial than the core plugin loading mechanism on startup?

Is not more beneficial, it's just another extra way that you can take advantage of.

Risks / Threats

The same as the regular one, I think.

What are pitfalls that can come with such an implementation?

If a plugin expects to be execute before some core modules (bya, run-before) it could lead to unexpected results if those core modules have been already executed.

What (higher) security vulnerabilities might this introduce in a server environment? I don't see any possible vulnerability on the server

What technical issues are to be expected (and how to prevent)?

Because plugins can have startup modules, and the order of the startup modules is important, all this stuff have to happen before any startup module have been executed. This makes virtually impossible to plug any plugin that has a startup modules after the boot process has been completed. Any other plugin should work fine.

We may provide a mechanism that requires a page reload, but not a file download.

Regards

[Jermolene]

Hi @danielo515 there is a mechanism to preload tiddlers before the boot kernel executes.

In $:/boot/bootprefix.js we non-destructively initialise the global $tw.preloadTiddlers:

https://github.com/Jermolene/TiddlyWiki5/blob/master/boot/bootprefix.js#L93

Then in $:/boot/boot.js we load any tiddlers present in the $tw.preloadTiddlers:

https://github.com/Jermolene/TiddlyWiki5/blob/master/boot/boot.js#L1890-L1892

So, you can create a raw markup tiddler containing a script tag that runs just enough code to load the tiddlers from local storage, and push them into $tw.preloadTiddlers. Of course, you'll be running before the boot kernel, and so won't be able to require other modules; you'll have to make the code entirely self sufficient.

[danielo515]

Hello @Jermolene

That's exactly what I guesed 😅 https://groups.google.com/d/msg/tiddlywikidev/Z7gPMd3HckQ/AcCEZt2UBwAJ

I consider that a feasible workaround, but not a solution for all the scenarios. It is hopefully enough for mine, but what about other possible plugins/flows? What if it takes too to load? Why not allow hot-plug of plugins? Probably it will be restricted at first to plugins that does not deeply integrate in the boot process.

Not to mention that we have to fix #2159 for a better user experience, but that does not bother me so much.

[Jermolene]

Hi @danielo515 I don't see any references to $tw.preloadTiddlers in the mailing list thread.

but not a solution for all the scenarios

Which scenarios that we've discussed couldn't be addressed with the technique?

Why not allow hot-plug of plugins?

Because it's incredibly complicated, as we are discussing elsewhere. And definitely out of scope for the proposal in this ticket.

[danielo515]

Hello Jeremy,

Hi @danielo515 I don't see any references to $tw.preloadTiddlers in the mailing list thread. Sorry, I didn't wrote the implementation details, but that was my intention.

I suppose that something like this is more than enough:

window.$tw = window.$tw || Object.create(null);
$tw. preloadTiddlers = $tw. preloadTiddlers || [];
// [ ... ]
// rest of the code

Which scenarios that we've discussed couldn't be addressed with the technique?

Technically this allows any adaptor to fetch plugins from the server, but if it takes long the boot process could get very boring. Probably something like #2254 will help.

And definitely out of scope for the proposal in this ticket.

Well, my initial proposal was exactly that, provide a mechanism to hot-plug plugins. Maybe we can lower the requirements and find something in between: Not hot-plug but not as raw as tagging it as rawmarkup. Something with a small bootstrapped TW environment, where the modules designed to load other plugins are executed before anything else.

[Jermolene]

if it takes long the boot process could get very boring

That sounds like an optimisation concern. As discussed elsewhere, I generally find it's better to make decisions about optimisations based on hard, measurable performance data.

[Jermolene]

A new PR #4259 addresses this problem.

[danielo515]

Thanks @Jermolene . My original problem was for plugins that include javascript, but this is definitely one step further. If you don't plan to ever implement this for JS modules you can close this issue.

[Jermolene]

Hey @danielo515 good to see you! In the future we may be able to offer dynamic loading for plugins that include certain types of module (e.g. JS macros) while still precluding others (e.g. startup modules, because it's not possible to offer the same semantics for a dynamically loaded plugin as one loaded during startup).