A couple of settables for renderers

[otherjoel]

@mbutterick — this is another thought relating to your offer in https://github.com/mbutterick/pollen-users/issues/86#issuecomment-786851964. The thing I’m making is an alternate scheme for making templates and applying them to Pollen sources. As things stand right now, this means either duplicating the functionality of raco pollen render and Pollen’s project web server, or giving those things up.

It would be really nice (not crucial) to have a way to provide Pollen with:

• A function to use internally in place of get-template-for
• A function to use internally in place of render-markup-or-markdown-source

I think this would be enough to allow my (or any) alternative template scheme to be used in raco pollen render and the web server.

Supposing this were done using overrideable pollen/setup values, here is an example pollen.rkt

#lang racket/base

(module setup racket/base
  (require (prefix-in b: beeswax/render))
  (provide (rename-out [get-template-proc b:get-template-for]
                       [template-applicator-proc b:render-with-template])))

The existing get-template-for and render-markup-or-markdown-source functions would not change. Instead, they would become the effective* defaults for these settable values.

(* — If I were writing a PR, I'd probably have these two settables default to #f, and have pollen/render check them, rather than trying to pull the default functions into pollen/setup.)

[otherjoel]

• A function to use internally in place of get-template-for

On second thought it would probably be better for this to be: a function to use internally in place of get-default-template

[mbutterick]

Though I agree the setup module is the most obvious place for this, I’m also leery of making these rendering functions dependent on resolution through setup (because of the performance cost — in general I’ve been trying to keep things out of setup).

More broadly, how is beeswax ideally invoked? Per source file, per template, per directory, per project?

[otherjoel]

how is beeswax ideally invoked? Per source file, per template, per directory, per project?

Can you clarify “invoked”? Do you mean, “how would the user ideally cause documents to be rendered using beeswax templates”? Because what I’m proposing here is that (after some additional configuration step) they will do that by using raco pollen render (and the Pollen project web server), in the same way they do now: sometimes per-file, sometimes per batch of files (directory or pagetree). All beeswax (or another system) would supply is the #lang for the templates themselves and the two functions described above: one for finding a template and another for applying it to a particular source file (example of applying a beeswax template)

The alternative I envision would be for beeswax to be invoked via its own raco beeswax render that behaves in the same way as and does mostly the same things as raco pollen render does now. It would be kind of redundant but doable. I don’t think I am realistically going to duplicate the local project server functionality in any case though. I guess the workaround in that case would be to develop using normal Pollen templates and just add #lang beeswax/template to the top when you don’t need the Pollen web server any more.

[mbutterick]

I suppose I’m just trying to conceptualize what the narrowest / most general API between beeswax and pollen could be. For instance, is it fair to say that you’re rendering against a function (generated by a beeswax source file) rather than a snippet of plain text (generated by a pollen template file)?

[otherjoel]

is it fair to say that you’re rendering against a function (generated by a beeswax source file)

Yes, that is fair to say. And to be clear, in this framework, that function wouldn't have to be generated by a beeswax source file; it could be provided by any Racket module, as long as it is called render and looks like (any/c hash? pagenode? . -> . bytes?). (And indeed I would think beeswax template files would have a .rkt file extension.)

[mbutterick]

OK, so maybe the way for us to think about it is opening Pollen rendering to some function that returns bytes? (by whatever method) and then beeswax is a client.

BTW why do you need access to get-default-template?

[otherjoel]

OK, so maybe the way for us to think about it is opening Pollen rendering to some function that returns bytes? (by whatever method) and then beeswax is a client.

BTW why do you need access to get-default-template?

I’m thinking about what method Pollen could possibly use to know which render function to use (each template provides its own render function). It already has a heuristic for marrying up source files with templates, but that heuristic (specifically step 2) isn’t going to find templates that don’t follow those filename conventions (for example, templates with a .rkt file extension). Having a mechanism for substituting some other function for get-default-template seems like a relatively clean way of replacing just that second step in the template search process. Source files could still specify a template in the metas, but for those that don’t do so, there would then be some other default (but project-specific) way of finding the right template.

Essentially, Pollen could accommodate pretty much any auxiliary templating scheme, without caring at all about the details, if it can be given two things: 1) how to find the right template for a given source file and output extension, and 2) how to combine a given template with a given source file. A hook for replacing get-default-template was my idea for the first, and the hook for replacing render-markup-or-markdown-source was my idea for the second. It seemed like a smaller ask than proposing that Pollen make more-complicated internal changes.

[mbutterick]

I’m starting to think we’d be better off creating a separate hook within Pollen for this kind of thing, rather than shoehorning it into the current idea of templates. I foresee that there will be escalating difficulties if two unrelated things are allowed to be called templates, and if two essentially unrelated rendering routines are forced to coexist.

For instance, maybe instead of writing

◊(define-meta template "my-template.html")

We would create an idea of a renderer:

◊(define-meta renderer "path/to/beeswax/source.rkt")

And then when Pollen sees the renderer meta, it can go down the new rendering path, getting a render function out of that source file and applying it.

What do you think?

[mbutterick]

For instance, one problem with muddling the two ideas: if render-markup-or-markdown-source is given a certain path as a template file, it won’t know whether it a) is meant to be used as a good old string-based template, or b) it contains a render function. As you say, the distinction won’t necessarily be apparent from the file suffix. So I guess it would have to first dynamically evaluate the file (to see if it exports a render function) and if that fails, treat it as a string-based template. That is going to impose awful costs on a project that just uses string-based templates. Which is why I think the fork has to happen further upstream.

[mbutterick]

Following on that idea, renderer could be used with define-meta as illustrated above (to set the renderer per-source). We could also make it a setup value with a default value of #false or a path string. If it’s #false, then you get the current behavior of string-based templates. Otherwise, the value of the renderer is used for all the source files controlled by that setup module.

[otherjoel]

I foresee that there will be escalating difficulties if two unrelated things are allowed to be called templates, and if two essentially unrelated rendering routines are forced to coexist. … We would create an idea of a renderer

Yes, I concur about this! (Maybe I have already created some of this confusion here.) If this discussion does introduce anything new to Pollen itself, that thing really should be called a renderer and not “templates”. But also, I would not consider an individual #lang beeswax/template file (or the the render function such a file provides) to be, itself, a “renderer”! More on this below…

For instance, one problem with muddling the two ideas: if render-markup-or-markdown-source is given a certain path as a template file, it won’t know whether it a) is meant to be used as a good old string-based template, or b) it contains a render function.

To be clear, I was not proposing that render-markup-or-markdown-source be changed in the slightest. Rather, I’m suggesting that I be able to say to Pollen (via a setup value or some other way) “here’s a function that takes exactly the same arguments as render-markup-or-markdown-source, and returns the same kind of result; so, for this project, please use my function in place of that one.”. Note: this function I propose to supply is not the render function directly provided by the #lang beeswax/template file (which takes doc and metas and an output path), but a separate function that looks like this:

;; This function is meant to be used in place of render-markup-or-markdown-source. 
;; It takes the same arguments and returns the same result.
(define (beeswax-renderer source-file [maybe-template-path #f] [maybe-output-path #f])
  ;; [Determine output extension if not supplied]
  ;; [Determine template path if not supplied]
  (let ([render-func (dynamic-require beeswax-template-path 'render)] ;; ← get the render func out of the beeswax template
        [doc (get-doc source-file)]
        [metas (get-metas sourc-file)])
      (render-func doc metas (->pagenode output-path))))

The above is what I would call a renderer: not an individual template-thing, but the thing that finds the right template-thing for a source file and applies it. (And Pollen need not know anything about what goes on inside that function. It could simply be (lambda (x [y #f] [z #f]) #"same thing everywhere") as far as Pollen cares.)

Following on that idea, renderer could be used with define-meta as illustrated above (to set the renderer per-source). We could also make it a setup value with a default value of #false or a path string. If it’s #false, then you get the current behavior of string-based templates. Otherwise, the value of the renderer is used for all the source files controlled by that setup module.

Yes! This is the essential idea behind what I first proposed: a setup value with a default value of #false that can be used to hook in a different renderer on a per-project basis. I proposed this setup value would contain an actual function; I suppose it could just as well be a path string to a module that provides the function, but I still like the idea of using actual functions in there.

[mbutterick]

OK, realization is dawning. I am apparently converging on the right spot 😉

I was asking how you wanted to invoke these renderers because if you want to possibly do so per-file (say, via define-meta) then the value of the meta needs to be a string or other serializable value (which means not a function). (Though in a setup module, it could also be a function.)

Also, what data other than the doc and metas would you need to choose a template for a given source file? (Recalling that metas also contains here-path.) I ask because it seems like Pollen only needs to cooperate with one outside function (that is, taking your example, get-template-for and render-with-template can become a single function like get-template-and-render-with-it)

[otherjoel]

I was asking how you wanted to invoke these renderers because if you want to possibly do so per-file (say, via define-meta) then the value of the meta needs to be a string or other serializable value (which means not a function). (Though in a setup module, it could also be a function.)

Yes, let me paint the picture :

1. I decide to use beeswax in my Pollen project
2. I take my template.html.p and rename it to template.html.rkt and add #lang beeswax/template at the top. There, now my template is a beeswax template.
3. I add some setup value that (waves hand) in some way tells Pollen “use beeswax as the renderer in this project” — but regardless of the details, neither Pollen nor Beeswax knows anything about my template.html.rkt file at this point.
4. I run raco pollen render source.html.pm — or I preview the source file in the Pollen project server
5. Pollen sees that I want to use some other renderer (beeswax). So it tells my renderer “here's a source file and an output file extension; make it happen and give me the bytes.”
6. Pollen knows nothing about what happens from this point until step 9.
7. Beeswax could at this point do just like Pollen does and check the metas of the source file. If it finds a value for 'template then it would use whatever file that points to. (Or it could look for some other key like 'beeswax-template.) If it finds nothing there, it could search the filesystem for the .rkt file to use as the template.
8. At any rate, once the beeswax renderer has identified the .rkt file to use as its “template” for this particular source file, it dynamic-requires the render function from that file, and calls it, giving it doc metas and the output filename.
   • I do recall that here-path is in the metas, but that is the source filename path. I need the output filename to use as the value for the magic variable here in the template, which always points to the output filename.
9. The bytes from that function call are passed back to Pollen and Pollen writes them out to the file or uses them however it wishes.

[mbutterick]

I just pushed the simplest possible implementation of this:

1. establishes a setup value called external-renderer.

2. If this value is not #false, uses it in place of the usual render. You suggested using it in place of render-markup-or-markdown-source. It seems to me that an external renderer would want to be able to override any render operation. Though maybe there should be a mechanism for the external renderer to signal “nah Pollen, you can just handle this one normally”, e.g., by raising a certain exception.

3. The external-renderer must be a function that accepts three values: a source path, a template path (in the Pollen sense of that word), and an output path. You suggested passing doc and metas. But that incurs cost, and I can imagine external renderers that perform tasks that don’t need doc or metas. So it feels like this should be an opt-in. If an external renderer (like yours) wants doc and metas, you can pull them out of the source file in some convenient way (say with cached-require).

4. Nothing else changes, so AFAICT the raco commands, project server, etc. all work as usual.

I have not yet added support for a define-meta. But let’s sort that out in the next step. For now, I’m interested to hear whether this much is useful.

[mbutterick]

For instance, here’s a toy external-renderer that ignores doc and metas and just writes out the three paths it receives as input:

#lang racket/base

(module setup racket/base
  (provide external-renderer)
  (require racket/string)
  (define (external-renderer sp tp op)
    (string-join (for/list ([p (list sp tp op)]
                            [name '("source" "template" "output")])
                           (format "~a path = ~a" name p)) "\n")))

[otherjoel]

Thank you! This is very promising.

The external-renderer must be a function that accepts three values: a source path, a template path (in the Pollen sense of that word), and an output path.

This should work, although it’s a bit suboptimal for Pollen to spend cycles hunting for a template.html.p (or whatever) that is not going to be used; ideally the external renderer would be the one to find a template (if it needs one). Maybe some other external renderer would make use of it though.

You suggested passing doc and metas. But that incurs cost

Not true! I suggested this:

Pollen sees that I want to use some other renderer (beeswax). So it tells my renderer “here's a source file and an output file extension; make it happen and give me the bytes.”

Beeswax itself does indeed grab the doc and metas when it needs them; all it wants from Pollen is the two things above. And I can get those things from the three values you are passing me 👍

I have not yet added support for a define-meta

You shouldn’t have to do anything there. Again, the external renderer can check the metas if it wants to.

[mbutterick]

although it’s a bit suboptimal for Pollen to spend cycles hunting for a template.html.p (or whatever) that is not going to be used

I’m persuaded by what you said above that an external renderer would be a “function that takes exactly the same arguments“ as the internal renderer. Beyond that, the template path is used in keys for some of the internal caches. So in that way, the template lookup pays for itself.

Not true! I suggested this:

Apologies. You’re correct; I misread.

But like I say, this is the simplest version — I consider that a virtue because it keeps the interaction with the external renderer simple — see how far you get & we can improve based on your experience.

[otherjoel]

Preliminary results, it is working very well! Rendering via raco pollen render is working (including parallel rendering) and is much faster than when using string-based templates. Pollen project server works seamlessly also.

Here’s the external renderer I’m been testing with. For now it just has a single beeswax template file ("bw-template.html.rkt") hardcoded in:

(define (external-renderer sp tp op)
    (case (path-get-extension sp)
      [(#".pm" #".pmd")
       (let* ([doc ((dynamic-require 'pollen/core 'get-doc) sp)]
              [metas ((dynamic-require 'pollen/core 'get-metas) sp)]
              [render (dynamic-require "bw-template.html.rkt" 'render)])
         (render doc metas (string->symbol (path->string op))))]
      [else ((dynamic-require 'pollen/render 'render) sp)]))

One limitation of the current approach is that things go sideways if the external renderer tries to require any function from Pollen, because of the loading loop that happens with pollen/setup. So I now see the virtue of your original notion of supplying the renderer as a module path rather than directly as a function. I now think doing it that way would be a big improvement. For now, I can get around this by using dynamic-require to get the specific functions I need.

I like that I’m also able to use Pollen’s own render function as a fallback. This is another advantage of your implementation over what I originally proposed.

When I have time I want to do some more testing about how the Pollen cache behaves with an external renderer in the presence or absence of the normal Pollen template files (like template.html.p).

[mbutterick]

because of the loading loop

Hmm — maybe it will make more sense, ultimately, to make external-renderer a parameter rather than a setup value. Though that would probably mean you need to set it imperatively in a pollen.rkt, which is venturing close to the icky world of global variables.

[otherjoel]

Hmm — maybe it will make more sense, ultimately, to make external-renderer a parameter rather than a setup value.

Are you saying that would be preferable to passing external-renderer as a module-path (suitable for dynamic-require)?

It‘s a little fuzzy to me how setting a Pollen-define parameter in pollen.rkt (outside of the setup submodule) would have any effect during this stage of the rendering phase.

[otherjoel]

I want to do some more testing about how the Pollen cache behaves with an external renderer in the presence or absence of the normal Pollen template files

No surprises here of course. But being able to supply a template-finder in addition to external-renderer would be nice, so that touching things the external renderer uses for rendering a particular file will naturally invalidate the cache.