Idea: Nimib plugin system (both dynlib & ordinary import)

[HugoGranstrom]

This is an idea inspired by @pietroppeter's work on nblog using JSON as an intermediate form for a static site generator and Ark (former Ivy). To summarize, the idea is to generate the NbDoc as usual, but instead of rendering it at the end, it is converted and save to a JSON file. Then a separate program reads these JSON files and renders them.

This got me thinking, how does the second program have access to all the partials, templates and renderProcs needed to render all of these files? One answer to this question is that we start to structure nimib themes and libraries as plugins. More specifically, we could say that each plugin must export a initNimibPlugin proc that accepts a var NbDoc. In it, the plugin would populate the contexts and renderProcs. This is basically how themes work in nimib today:

nbInit(theme = revealTheme)

Now instead, imagine that we could supply multiple of these procs, one for each plugin/theme:

nbInit(plugins = @[revealTheme, plugin1Init, plugin2Init])

The sweet part about this is that these initNimibPlugin procs could come from anywhere! They could come from a module we have imported or (and this is the important bit) it could come from a shared library. In other words; this would allow the partials, templates and renderProcs to be accessed by the rendering executable by loading them using for example std/dynlib.

This might not be enough for implementing a static site generator in nimib, but it is a step towards it. And it is not that big of a change relative to what we currently have, it's mainly just that we start to recommend a certain structure of nimib libraries.

You have probably thought about a lot of these aspects already, but I haven't found a good issue to discuss it, so I'm creating this one. Is there anything I'm getting wrong or that you have different opinions on?

[HugoGranstrom]

I haven't tried playing around with shared libraries myself yet, but the way I understand it is that we would be able to load the initNimibPlugin proc from a shared library plugin.so something like this:

import std/dynlib

type
  NimibInitProc* = proc (doc: var NbDoc)

let lib = loadLib("plugin.so")
let initProc = cast[NimibInitProc](lib.symAddr("initNimibPlugin"))

[pietroppeter]

yep, that's pretty much the idea I had in mind, thanks for writing it down!

in general the idea is that:

• the content of a nimib document is what is stored in the chain of blocks (possibly nested once we have container blocks. which contains the type/name/command of block (is it a nbCode? a nbText? a nbWhateverIComeUpWith?) and the data of the block (code, output, custom fields, ...). This is the part that we serialise as json.
• how the nimib document is rendered (theme, but in principle also backend) should be information kept separate. it can be loaded during initialization in a single document or it could be loaded in a separate binary to "re-hydrate" the deserialised json with rendering information.

I like the idea of multiple plugins (also in principle it is an api we can introduce without breaking anything), it would help if we split down our current defaultTheme to multiple plugins since it does a lot more stuff than just a theme (in particular it does set up all the rendering).

As a side note, I am realising now that none of that we load during init actually needs to be loaded during init, it actually needs to be loaded before rendering (and then saving) the final document.

[HugoGranstrom]

Good to know that we are on the same page :+1:

it would help if we split down our current defaultTheme to multiple plugins since it does a lot more stuff than just a theme (in particular it does set up all the rendering).

Very good point :rocket:

As a side note, I am realising now that none of that we load during init actually needs to be loaded during init, it actually needs to be loaded before rendering (and then saving) the final document.

Haha, very true actually :o But I don't see any reason to move them anywhere else thannbInit. If we want to speed up the runtime, we could just inject the list of plugins as a variable that is then accessed by the rendering proc. This way, we only initialize the plugins when/if we render the document. To me, it makes the most sense to specify the plugins when initializing the document at least.

[HugoGranstrom]

Something else that I've noticed would be nice is to be able to run some functions when nbSave is run. For example here where I will have to use mustache-nim's Value type to represent a seq[Table[string, string] instead of the simpler Table[string, string]. If I had the possibility to add a proc that would be run at nbSave, I would be able to use a Table[string, string] and convert it to the mustache Value just for the rendering.

This would basically allow us to use arbitrary ordinary Nim types and not have to fight against the non-ergonomic Value (or in the future JsonNode) all the time.

[HugoGranstrom]

I'm actually tempted to start implementing parts of this. Especially the procs that should run on nbSave. And I'm thinking that we can introduce a plugin: seq[proc (doc: var NbDoc)] to nbInit but still keep theme and just run both. That way we don't break any existing code. And if we run the plugins in nbSave I will get the behavior that I'm looking for. @pietroppeter Do you have any objections to this? Or improvement?

[pietroppeter]

Hi, I have just re read this. I am not sure I have completely understood the nbSave and the example in reveal js, could you expand a little bit on that and a brief plan on what is the part you plan to start implementing? I do not think I will have particular objections, but I am rather slow these days (lots of things distracting me) and a bit more of explanations would help me understand and see if there are improvements I can make. Thanks for thinking about working on this!

[HugoGranstrom]

So, the objective in nimiSlides is to allow users to specify fields in a configuration object sent to Reveal.js. E.g:

Reveal.initialize({
    mouseWheel: true,
    slideNumber: 'c/t',
    center: false
});

So we dynamically want to create these key-value pairs (e.g.center & false). There are simply too many to wrap them all manually. The idea is to have a proc that adds these, nb.setConfig("center", "false") to allow maximal flexibility for the user. Now they can use whatever new options new version of Reveal.js adds without us having to update anything.

The natural type to store this information in would be a Table[string, string] where we simply do table["center"] = "false" internally. But, there are two problems:

1. I don't have any way of transforming this Table to the appropriate mustache value. I need to run it just before nbSave which isn't possible currently without forcing the user to remember to call a insertConfigTableIntoContext() calling nbSave.
2. So what if I instead just use mustache's context instead, it supports Tables. The problem here is that mustache doesn't support looping over all key-value pairs of a table. It can only loop over lists. So now I have to deal with something like seq[(key: string, value: string)] inside a Value instead. And this is horrible to modify because Values are not user-friendly basically. What was before a simple assignment is now a search and replace in a list.

So if I could use a Table in my code and before nbSave is run, convert it to the nasty seq[(string, string)], I would make my life a lot easier. I can use nice native Nim types in my entire program and then write a proc that converts it to a Value and inserts it into the context when it is finished. That's fine now because I won't change the value anymore (because we have reached nbSave).

Do you understand my problem here? I can't use a nice type because mustache wants to use a nasty type. And I can't convert the nice type to the nasty type without forcing the user to call a proc before calling nbSave.

a brief plan on what is the part you plan to start implementing?

It will be very simple actually:

1. Add the optional plugin parameter
2. Either add a field to NbDoc or inject a variable with the list of functions to call.
3. Call all plugin functions in nbSave
4. While I'm at it I could as well start dividing the defaultTheme into smaller plugins (that defaultTheme calls)

I do not think I will have particular objections, but I am rather slow these days (lots of things distracting me) and a bit more of explanations would help me understand and see if there are improvements I can make. Thanks for thinking about working on this!

No worries, it's summer, so I don't expect 24/7 lightning fast responses :smile:

The one thing that I'm not entirely sure about is if we need to make distinctions between "render-plugins" and "functionality-plugins" and possibly more kinds of plugins. I wouldn't want to run this proc in both stages in the context of an SSG. I would want to run it when generating the JSON, but not when loading the JSON back again because then we would have an empty Table (because we have started a new program) that would override the value in the context. So we would need a way to specify that it shouldn't run specific plugins. Also we could want "functionality-plugins" that run either at the start (nbInit) to initialize some variables, or at the end (nbSave) to store variables in the context.

Ok, this turned out to be more complicated now that I think about it. My proposed solution would work for the current nimib, but making this work well with the future SSG makes it all a bit harder. If and where each plugin is called will matter very much.

[HugoGranstrom]

The last two paragraphs are mostly brain-storming so don't be afraid to ask if there is parts that are not making sense (there definetly is!). The summary of it is that different kinds of plugins will have to be run either at the start or end (and others doesn't matter) and then we have some plugins that the SSG should only run in the Nim->JSON stage and not in the JSON -> HTML stage.

[pietroppeter]

Hey, thanks for the extended summary, I understand it better now. How much of the ugliness you experience is due to use of mustache Value type and would using JsonNode for data be helpful here? Especially considering that with jsony we could add interface to feel like we are dealing with actual types. It is something I definitely want to do at some point, and it should probably happen sooner rather than later.

I was thinking about having plug-ins added in nbInit and called in nbSave and while I like the idea (having stuff called in nbSave means we do not have extraneous information in the blocks that we need to skip during serialization), I think it might break something at the moment: say you override some parameters in default theme, if then the plug-ins of default theme is executed at the end, it will override the override. Probably this can be fixed by initializing a new context with plug-ins first and then overriding with the context available from nbInit. I have a feeling though that changing to have a data: JsonNode might make this a bit better (the idea was that context is initialized from JsonNode and this could be done after running plug-ins).

[HugoGranstrom]

How much of the ugliness you experience is due to use of mustache Value type and would using JsonNode for data be helpful here?

I'd say that it wouldn't solve the core problem: even if I used a JsonNode that was a Table, it would still be converted to a Table in the context, which can't be rendered the way I want it to. It would make working with the seq[(string, string)] less cumbersome, though. So it doesn't solve my problem, it just elevates it.

say you override some parameters in default theme, if then the plug-ins of default theme is executed at the end, it will override the override. Probably this can be fixed by initializing a new context with plug-ins first and then overriding with the context available from nbInit.

That's very true :thinking: And yes, running the plugins first would make it work in this case.

I have a feeling though that changing to have a data: JsonNode might make this a bit better

Yes, it would help a lot to be able to save and reproduce a context from an object in the case of an SSG.

[HugoGranstrom]

So basically we have these three (four) kinds of actions we want plugins to be able to do:

1. Populate templates, renderPlans and renderProcs
2. Initialize stuff (initialize variables, populate the context)
3. Wrap up stuff at the end (convert variables to context)
4. (Run a plugin every time we create a block?)

So I'm thinking, could we represent a plugin as an object instead? Something like:

type 
  NimibPlugin = ref object
    f: proc (doc: var NbDoc)
    kind: NimibPluginKind
    of Render: discard
    of Functionality:
      runAt: NimibRun

  NimibRun = enum
    Init, Block, Save

Then we would be able to separate render-plugins from the functionality-plugins and only run the ones we need. The problem would of course be how we would create nested plugins like defaultTheme if it calls both kinds of plugins. And an idea would be to add a field of type seq[NimibPlugin] so that we get a tree structure of plugins where each plugin can specify other plugins. We would then flatten this into a list of NimibPlugins. The problem with this approach though is that it's harder to comprehend for a future plugin author what actually make a plugin a render-plugin or a functionality plugin. And the consequences of choosing the wrong kind would only show up when using the SSG.

[HugoGranstrom]

Okay, have thought a bit more about it now and a render-plugin only really needs the partials, so it would only need that Table[string, string] as its input and not the entire NbDoc. This could lower the confusion as the proc signature now is different for the two types of plugin. So the type would be something like:

type 
  NimibPlugin = ref object
    kind: NimibPluginKind
    of Render:
      fRender: proc (partials: var Table[string, string])
    of Functionality:
      fFunc: proc (doc: var NbDoc)
      runAt: NimibRun

With this, it would be more obvious what makes a render-plugin different from a functionality-plugin.

[HugoGranstrom]

I'll keep brainstorming here. I realized all of these (render, init, block, save) are just different hooks the plugin can run at! They are all just different points in the program that they can be executed. The render-hooks are simply run right before rendering the document. So we only need 1 enum instead of 2:

type 
  NimibPlugin = ref object
    kind: NimibPluginHook
    of Render:
      fRender: proc (partials: var Table[string, string])
    of Init, Block, Save:
      fFunc: proc (doc: var NbDoc)

  NimibPluginHook = enum
    Render, Init, Block, Save

With this, it's much easier for a user to reason about 1 kind instead of the 2 kinds I used previously. It's also easier to extend it with more kinds of hooks in the future.

[pietroppeter]

(relevant forum post: https://forum.nim-lang.org/t/10423)