refactor rendering

[pietroppeter]

a big refactor in rendering is ongoing (branch refactor-render).

I will add render objects and separte the rendering part from the block part. blocks will be generic (no block kind anymore) to simplify creation of custom blocks. also it will simplify how to add or customize a rendering backend.

Basically every block will have only a renderPlan (just a seq of strings). and according to backend it will execute the different steps. block renderPlan can be customized and rendering backend also will be customizable.

[pietroppeter]

other expected enhancements from this:

• possibility to customize escaping (current default is escaping the code block and not escaping the output block; the output block will then escape by default when there will be the possibility to easily to revert this for a specific block).
• a better nbImage
• other nbBlocks such as nbSkip (do not execute code, but show it), nbBase (no capture and no rendering, a base code block from which customization can be made; e.g. nbImage will be built from nbBase)
• better handling of rendering both html and markdown. In particular it will be possibile to change content based on output using specific "skipHtml" or "skipMd" rendering steps.

[pietroppeter]

recent development notes on the topic (closing obsolete PR #35, will be redone from scratch):

current rendering process

• nbSave calls NbDoc.write on nb object
• write calls nb.render proc on itself (uses nb.filename as filename)
• nb.render is by default initialized as renderHtml
• renderHtml:
  • calls renderHtmlBlocks and puts its result in doc.context["blocks"]
  • renders partial "document" given doc.context
• renderHtmlBlocks is just an append of multiple blk.renderHtmlBlock
• renderHtmlBlock is a case over block kind (code, text, image)
  • text: calls renderHtmlTextOutput which is renderMarkdown after stripping
  • code: call renderHtmlCodeBody (which highlights nim code and wraps it in pre-code) and optionally calls renderHtmlCodeOutput if there is output (escape stuff and wraps around pre-samp)
  • image: wraps in an image template

process to be

• at the beginning I should change only renderBlocks mechanism (leaving nbSave and usage of filename as is)
• so I will change content of renderHtml only
• and actually what changes is only the rendering of the single block

rendering of a single block:

• every block has a render plan (seq[string])
• default render plan is set for each named command (nbCode, nbText, nbImage, ...) (a new renderPlans Table in NbDoc object)
• render plan is backend indepenent, but:
• execution of render plan depends on backend (highlighting code is not done in markdown backend for example)
• a single render plan is usually something that:
  • populates blk context
  • renders a (single) partial using populated blk context (which is derived from doc context)
  • (optional) update doc context? no, not in this moment. stuff that could go on here: link headers, ...
• a parte: every block could have a series of step that are executed after normal execution and update doc context (e.g. add an element of Toc, an asset -image, file- being created, ...)
• again this step is something else, not for this part of refactoring

restricting scope:

• only block rendering (do not change doc rendering)
• not covering stuff that should be done during block execution (e.g. updating doc context)

example of API

show api for:

• nbText
• nbCode (nbCodeInBlock is trivial)
• nbImage
• nbAudio
• nbVideo
• nbFile (both versions)
• nbTextWithSource (from cheatsheet, remember not to escape stuff - at least in one case)
• nbCodeOutputUnescaped (or how I skip escaping output)
• templates from nimib reveal
• add a table/image/... output to a nbCode object with nbShow (or more than one)
• how would markdown backend work?
• how would another backend work?

template nbText(body: untyped) =
  nb.blk = newBlock("nbText"): body  # save code as string in nb.blk.code (we did not use to do this for nbText)
  nb.blk.output = body  # body must be an expression that evaluates as string
  nb.blk.renderPlan = nb.renderPlan["nbText"]
  nb.blk.partial = "nbText"

assert nb.renderPlan["nbText"] == @["mdTextToHtml"]  # adds "textAsHtml" to context with text converted to Html through markdown
# we could other steps to process headers for generating a Toc (or for collecting link that could be check for validity)
assert nb.partials["nbText"] == """
{{&textAsHtml}}"""

template nbCode(body: untyped) =
  nb.blk = newBlock("nbCode"): body
  captureStdout(nb.blk.output):
    body
  nb.blk.renderPlan = nb.renderPlan["nbCode"]
  nb.blk.partial = "nbCode"

assert nb.renderPlan["nbCode"] == @["highlightCode"]
assert nb.partials["nbCode"] == """
{{>nbCodeSource}}
{{>nbCodeOutput}}"""
assert nb.partials["nbCodeSource"] == """
<pre><code class="nim hljs">{{&codeHighlighted}}</code></pre>
""" # note new line here, note codeHighlighted can (and will) contain html
assert nb.partials["nbCodeOutput"] == """
<pre><samp>{{codeOutput}}</samp></pre>
""" # note new line here, note output is escaped

template nbImage(url, caption: string) =
  nb.blk = newBlock("NbImage"): body
  nb.blk.context["image"] = %* {"url": url, "caption": caption}
  nb.blk.partial = "nbImage"

assert nb.partials["nbImage"] == """
<figure>
<img src="{{image.url}}" alt="{{image.caption}}">
<figcaption>{caption}</figcaption>
</figure>
"""

# function that renders the single block
func render(nb: var NbDoc, blk: var NbBlock): string =
  for step in blk.renderPlan:
    if step in nb.renderProc:
      nb.renderProc[step](nb, blk)
    # else raise some warning?
  nb.blk.partial.render(blk.context)

# to use markdown backend? change partials, change renderProc, change also renderPlans?
# at the moment I should care the absolute minimum for the second backend if api for main backend is sound

[HugoGranstrom]

Looks great! I really like the separation and modularity of the population of the contexts (renderPlan) and what is actually rendered (blk.partial). This should suffice in making custom versions of the basic templates of nimib like nbCode by just using a different partial to render :D

A nitpick, but should the partials be nb.blk.partial = nb.partials["nbText"] or is there some magic that you just provide the name of the partial that is stored in nb.partials? :)

[HugoGranstrom]

@pietroppeter What's the time span we are talking about before something like this is released in a new version of nimib? I'm thinking about announcing nimib-reveal, but if it is in the next few weeks I'm thinking about postponing it until then. 😄

[pietroppeter]

It is really overdue so yes, I plan to get it done before the end of the year

[HugoGranstrom]

Great to hear! I'll wait 'til then, then :D

[pietroppeter]

Did not manage end of year but this my number one priority now

[HugoGranstrom]

No worries 😄 I haven't had time for much either this Christmas. It will be finished when it is finished 👌

[pietroppeter]

big milestone in new #78: refactor now is able to regenerate correctly hello.nim! 🥳

with respect to API published above a few changes in particular nbText and nbCode now are simpler:

template nbText*(body: untyped) =
  newNbBlock("nbText", nb, nb.blk, body):
    nb.blk.output = block:
      body

template nbCode*(body: untyped) =
  newNbBlock("nbCode", nb, nb.blk, body):
    captureStdout(nb.blk.output):
      body