Add token preprocessor hook and URL query parameters

wrapperup commented 7 months ago

This adds a new lower-level hook called tokenPreprocessors to Environment that lets plugins transform a template's tokens into other tokens. It's like tags, except it isn't limited to a tag, and you output tokens.

Also adds support for URL query parameters. The sanitized path is passed to the loader, and the full path (with query strings) is passed to the token preprocessor, which unlocks a few capabilities.

Motivation

One use for this is to add template fragments for the htmx library (https://htmx.org/essays/template-fragments/). Basically, you can mark up portions of the template that can be rendered individually. For example:

Template Fragments Example Plugin

```js function fragmentPreprocessor( env: Environment, tokens: Token[], path?: string ): Token[] | undefined { if (!path) { return; } const splitPath = path.split("#"); if (splitPath.length !== 2) { return; } const [_, fragment] = splitPath; let indexStart = 0; while (indexStart < tokens.length) { const token = tokens[indexStart]; if (token[0] === "tag" && token[1] === "fragment " + fragment) { // Skip the fragment tag indexStart++; break; } indexStart++; } if (indexStart >= tokens.length) { throw new Error("Couldn't find fragment tag"); } let indexEnd = indexStart; while (indexEnd < tokens.length) { const token = tokens[indexEnd]; if (token[0] === "tag" && token[1] === "/fragment") { const fragment = tokens.slice(indexStart, indexEnd); return fragment; } indexEnd++; } throw new Error("Couldn't find end of fragment tag"); } function fragmentTag( env: Environment, code: string, output: string, tokens: Token[], ): string | undefined { const match = code?.match(/^fragment (.+)$/); if (!match) { return; } const compiled: string[] = []; const compiledFilters = env.compileFilters(tokens, output); compiled.push(...env.compileTokens(tokens, output, ["/fragment"])); compiled.push(`${output} = ${compiledFilters};`); if (tokens.length && (tokens[0][0] !== "tag" || tokens[0][1] !== "/fragment")) { throw new Error(`Missing closing tag for fragment: ${code}`); } tokens.shift(); return compiled.join("\n"); } export function fragmentPlugin() { return (env: Environment) => { env.tokenPreprocessors.push(fragmentPreprocessor); env.tags.push(fragmentTag); } } ```

Lets you write

<body>
  <div hx-target="this">
  {{ fragment button }}
    <button hx-get="/data">Refresh Data: {{ data }}</button>
  {{ /fragment }}
  </div>
</body>

and if I add this "#button" query parameter,

const frag = await env.load("my-template.vto#button");

my template now renders just the button:

<button hx-get="/data">Refresh Data: 10</button>

Design questions

[x] Does having the user-defined "context" object make sense here? Alternatively, passing the filepath down to plugins and supporting URL query parameters in loaders would work too (and trivializes caching).
[x] Any preferred name for "preprocessor" (or other types)? jinja2 calls their token processor hook "filter_tokens" for example.

Thanks.

TODO

[x] Implement with cache
[ ] Document

oscarotero commented 7 months ago

If I understand correctly, you want to run preprocessors to modify the tags before the compilation in order to transform, filter or generate more tags. I'm not sure about your use case (template fragments). It's easier with exports and imports tags.

And what you are proposing is the ability to load a template multiple times with different options

// Load with a fragment
const frag = await env.load("my-template.vto", undefined, { fragment: "button" });

// Load with another fragment
const frag = await env.load("my-template.vto", undefined, { fragment: "other" });

I don't think it's a good idea, it makes the cache invalidation more complicated, and it's confusing if you're importing the same file multiple times in your templates with different configurations.

The preprocessor idea could be useful for some use cases, but I don't like the CompileContext. I think if it's implemented, it should be applied only once, after the template tokenization and without dynamic options.

wrapperup commented 7 months ago

If I understand correctly, you want to run preprocessors to modify the tags before the compilation in order to transform, filter or generate more tags. I'm not sure about your use case (template fragments). It's easier with exports and imports tags.

Right, though this particular use case it's a lot more convenient to do it this way. Otherwise, you'd need to have some additional runtime conditions or split it off into a new file.

And what you are proposing is the ability to load a template multiple times with different options

I don't think it's a good idea, it makes the cache invalidation more complicated, and it's confusing if you're importing the same file multiple times in your templates with different configurations.

The preprocessor idea could be useful for some use cases, but I don't like the CompileContext. I think if it's implemented, it should be applied only once, after the template tokenization and without dynamic options.

Agreed, not really a fan of context either. It definitely complicates caching, so instead I think supporting query parameters in the path like so:

const frag = await env.load("my-template.vto#fragment");

Would be a better approach, and can it be cached easily since it's just part of the path. Older versions of Nunjucks supported something like this (but that got lost in translation a few years ago unfortunately). However, the file loader would need to support it and trim off the query params. If you like, we could split that off into another PR, since preprocessing by itself is already pretty useful.

oscarotero commented 7 months ago

Okay. I think this can be split in two different steps:

Step 1: Allow query params and fragments in the file path. /template.vto?name=value#foo. Vento should clean this part before passing the path to the file loader (for backward compatibility).
Step 2: Implement the preprocessor system, which would be a function accepting 3 variables. the environment instance, the array of tokens and the complete filename (with query params and fragments).

what do you think?

wrapperup commented 7 months ago

Okay. I think this can be split in two different steps:

Step 1: Allow query params and fragments in the file path. /template.vto?name=value#foo. Vento should clean this part before passing the path to the file loader (for backward compatibility).

Step 2: Implement the preprocessor system, which would be a function accepting 3 variables. the environment instance, the array of tokens and the complete filename (with query params and fragments).

what do you think?

Yep, perfect. I'll try that.

wrapperup commented 7 months ago

I also changed the name of the hook to "tokenPreprocessors" to make it a bit more specific, and to leave room for "preprocessors" in the future (ie. minify HTML directly from source, which isn't easy to do in a token preprocessor I'd imagine).

oscarotero commented 7 months ago

It looks great. Thank you!

ventojs / vento