Tera v2 wishlist/changes

[Keats]

On top of https://github.com/Keats/tera/issues?q=is%3Aopen+is%3Aissue+label%3A%22For+next+major+version%22

Parser

Operator precedence and expressions ✅

Parentheses should work everywhere and precedence should make sense. This is mostly already implemented in a new parser (private for now).

Should something like {{ (get_page(path=“some-page”)).title }} work? Right now it requires going through set which is probably fine imo.

Better error messages ✅

The new parser is hand-written so we can provide detailed error for each type of error. I’m also thinking of spanning all expressions in the AST for better rendering errors but not 100% convinced on that yet, will need to try.

Whitespace management

Do trim_blocks and lstrip_blocks (https://jinja.palletsprojects.com/en/3.0.x/templates/#whitespace-control) by default that can be switched on/off as one thing and not two.

Indexing changes ❔

Right now you can access array indices by doing my_value.0 or my_value[0]. This is in line with Jinja2/Django but it feels a bit weird.

How about allowing only my_value[0] so array index is the same as dict index and more like all programming languages (except tuple access)? The dot syntax doesn’t work when the index is a variable anyway which is probably the main usage of indexing. It would be a big departure from Django/Jinja2 though.

Features

call

https://ttl255.com/jinja2-tutorial-part-5-macros/#call-block is a good example of that feature. I do find the naming in Jinja2 very confusing though so it would probably be called something different

Investigating valuable ❔

https://tokio.rs/blog/2021-05-valuable This should avoid an awful lot of cloning and would improve performances a lot. Still not clear whether it will work though.

Improving filters/tests/functions

They should be able to get values from the context implicitely the same way expression do so we don’t have to repeat things like lang=lang in Zola for example. Maybe a first argument which is something like get_ctx: |key: &str| -> Option<Value>? It would be nice if there was a way to define the arguments in a better way than the current macros too. Related issue: https://github.com/Keats/tera/issues/543

Also remove some weird things added to match Jinja2 like https://github.com/Keats/tera/issues/650#issuecomment-1003735940

--

Any feedback / other items you want to change?

[azjezz]

global context: #765

---

another nice edition would be support for arrow functions: https://twig.symfony.com/doc/3.x/filters/filter.html

e.g:

{% set tasks_partitions = tasks|partition((task) => task.is_complete) %}

<p>Tasks:</p>
<ul>
{% for task in tasks_partitions[1] -%}
    <li>{{ task.description }}</li>
{% endfor %}
</ul>

<p>Complete Tasks:</p>
<ul>
{% for task in tasks_partitions[0] -%}
    <li>{{ task.description }}</li>
{% endfor %}
</ul>

[Keats]

We kinda have the filter/map function as well already but it's not going to be as an arrow syntax

[0xpr03]

Thanks for Tera! Not sure if this fits, but I'd like to have an easy way of displaying how long it took to render. I think this can currently be accomplished via workarounds of now(), but it would be nice if I could pass an Instant and tell tera to fill-in the duration since then (as late as possible obviously). That way I could display how long it took from starting to fetch data in the DB, to rendering this. It would already be fine if I'd have to insert that duration as the last element in my template, all critical loops would happen before and render-times are almost always displayed in the footer.

[Libbum]

I'd love to see something like floatformat.

[Bechma]

It would be nice also that we can check at compile time if there are errors, like in SQLx. Something like:

struct IndexContext {
    pub title: String,
    pub age: u16,
    pub name: String,
}

tera::template!("templates/index.j2", IndexContext{title: "hello world", age: 30, name: "John"})

[Keats]

It's not possible, you can have Tera functions creating new/updating context and we wouldn't know it at compile time.

[tuyen-at-work]

I would like to have cache buster in the static asset paths.
Something like:

/js/main.js?v=ab12cd34

where v value is the hash of the main.js file.
The reason for it is some static server/CDN support for caching, but if the file has new content, we don't have any method to tell browser to get it. Add this param should allow us set cache time in very far future but still serve new content when it is available.

In tera, it should be some thing like this:

{{ get_url(path='js/main.js', with_hash=true) | safe }}

[Keats]

Zola has that but I don't think it should be in Tera itself.

[tuyen-at-work]

I would like to have substring filter:

// Take first 10 chars
// If input length <= 10 return input
{{ input | substr(end=10) }}

// Skip first 5 chars and take the rest
// If input length <= 5 return empty
{{ input | substr(start=5) }}

// Take chars from 6th to 10th
// If input length <= 5 return empty
// If 5 < input length < 10 skip first 5 chars and take the rest
{{ input | substr(start=5, end=10) }}

[slatian]

First: Huge thank you for crating Tera, it has been a joy to use … that said, as someone who is guilty of overengeneering templates I've noticed a few things that I'd like to see in tera. I'm not expecting that any of these make it (I know when I'm writing things down that will create hours of work), but since this is titled "wishlist" :smiley: :

• When I {% extends "…" %} a template it feels a bit wired that I can't pass some kind of variables to it (other than blocks) to configure it. (I imagine that a template could have some {% configure <variable> %} statements which could accept arguments similar to how a macro works).
  Note: I could use macros, but then I'd only have a single block …
  I consider telling me that my templates are a bubbling hell of a mess a valid answer to this one.
• Structs/Dictionaries/Tables it feels a bit wired that I can create custom arrays, but not custom structs that I can access using the dot notation I can use for almost anything else.
• An extension to structs: Being a able to somehow return them from macros so that I only have to iterate over a list once to i.e. find the assets I care about.
• Global variables can't be used in blocks when extending a template. Example:

  {% set_global i_need_this_in_multiple_blocks = "Imagine something a bit more expensive here than a simple string" %}
  {% block pre_content %} … {{ i_need_this_in_multiple_blocks }} … {% endblock %}
  {% block after_content %} … {{ i_need_this_in_multiple_blocks }} … {% endblock %}

Usual idea disclaimer: Please take these as just Ideas, if you tell me they are bad that is okay.

[Keats]

1. It's an intentional design choice. This way we can make sure we have all the templates required at compile time rather than run time
2. Literal dict will be in v2
3. Maybe something different than macros then. Macros are fn(kwargs) -> text. I'd rather add something different if needed
4. That's normal. Rendering the content when extending ignores anything outside of the blocks. It's not clear semantically what should happen so we just ignore it.

[pickfire]

One thing I wish in tera is that since tera already have all templates needed at compile time, there is an option to allow for checking if context have everything needed for template at compile-time, otherwise there doesn't seemed much of a benefit over using jinja2/django instead, one of the pain points with jinja2/django is that we won't be able to know what if context is missing in template until we run it. Like what @Bechma mentioned

Looking at sqlx is a bit mind-blowing, I didn't know it was possible but sqlx really did do a query at compile-time to check if the query is valid, I used to have broken queries in the past sometimes when I missed testing it out, the other thing is the template which I am not aware of any interesting templating engine that allows checking it at compile-time. Probably not an easy task, but I don't think it is impossible either.

Maybe I can help to take a look at it since I am a bit interested too.

[Keats]

there is an option to allow for checking if context have everything needed for template at compile-time,

I don't think it's possible? You can load whatever data you want with Tera functions, think of Zola load_data that loads arbitrary json/yaml/csv/etc. Even then, this would mean you can only render templates if you're running Rust. Taking again Zola as an example, you don't really know the fields in the page/section extra until you parse the TOML files.

The kind of checking that SQLx does is neat, but not suitable for all usecases. For similar template engine that works that way you have https://github.com/djc/askama and https://github.com/lambda-fairy/maud that will actually generate Rust code but yeah, no dynamic templates there.

[pickfire]

I was thinking of doing something with the API:

render!("template", {
    item1
});

Then the generated code becomes something like by checking all the Expr beforehand

println!("template data {{ item1 }}");

But I think an issue is that for nested variable blocks like {{ map.field1 }}, I am not sure if there is a way to have the compiler do type elision around that, since during code generation time there is no type information available, so I think currently the way askama handle it is probably good enough, I wonder if there are something that allows println!("{{ item1.value }}");, I also haven't figure out how to do loops around the elided types.

I wonder if there is a way to cast the original type to a subtype by extracting only certain fields, like original type

struct OriginalData {
    a: String,
    b: String,
}

struct RenderedData {
    a: String,
}

[porglezomp]

One thing I would like is per-function safety, so I don't have to pepper my templates with | safe so much.

I would like if functions/filters/etc. could declare to Tera that their output is safe, so I can define a url_for which outputs properly pre-escaped text, and so that some builtins that have safe output can can work more nicely.

[Keats]

That's already the case? Eg see is_safe in https://docs.rs/tera/latest/tera/trait.Filter.html

[porglezomp]

Oh that's my bad, I completely missed that because of basically only ever registering closures as functions! So maybe what I'm asking for is a much simpler "built in wrapper type to declare closures that are safe", but now I can see how to make my own version of that as well, like:

tera.register_function("foo", tera::helpers::Safe(|args| etc.))

but now that's also not an incompatible change either, so :)

[sdedovic]

Re: Macros v2, copying my comment from #822 here

I believe macro definition should only be done:

• the top-level of a file (not in an if/else, for, or block)
• in any file (self, as in #822)
• in any location (after usage, i.e. forward reference).

See the next section to understand why.

---

With jinja2 templates you can do the following:

{% if some_val %}
  {% macro foo() %}42{% endmacro %}
{%  endif %}
{{ foo() }}

The macro may or may not be defined during rendering based on the truthiness of some_val. This is because Jinja takes the template AST and converts it into a Python module (using string concatenation) and the "rendering" is more or less evaling the Python at runtime.

This can't work easily with Rust, for obvious reasons. (Maybe if you turn the Tera template AST into rust code, compile it, and load it dynamically...) anyway,

This becomes a point of divergence for Tera macros. The current code will load all the macro definitions into one table as a first pass during rendering. Then they are looked up from this table when traversing the AST, and thus we cannot have dynamic, conditional macro definitions. This can be fixed by defining macros during AST traversal with Context in scope, or by extending the multi-pass approach to optimize the AST and generate macro definitions prior to rendering.

In any case, I personally believe this is likely an anti-pattern. Rarely is there a valid use case to define globals dynamically, and even less so when these act like pure functions, or "template snippets" as I see it.

[Keats]

We could make it work pretty easily inside blocks (if/for/block/filter sections) if we wanted but then we need to keep track of the context for each macro. Much simpler to disallow them.

[sdedovic]

I agree but but also happy to contribute some time into making macro defs inside blocks work too. Seems like a fun problem.

[Keats]

It's not hard, just that it doesn't make much sense imo to have things defined inside blocks in general.

[banocean]

Future flags, people using only Tera::one_off got a lot of useless dependencies

[fabianfreyer]

Maybe one could make Context a trait, and allow people to ship their own implementations to look up variables while rendering?

I'd e.g. love to be able to hook up tera to config-rs or figment as providers for variable values.

[pranabekka]

I'm not sure if this is the place for this, since striptags is a specific function implemented on top of the core, but I'd like it if you could specify a list of tags to strip out.

Something like striptags(tags="a, img").

My motivation is that when I create a preview of a page (from its first few lines), I want the preview to include formatting like code, sub and sup, but not links (especially internal ones), images, other media, etc.

Edit: Removing a tag along with its content would also be cool. Something like striptags(tags="a>") to remove the anchor tag and its children.

[banocean]

Functions as values in context

[Keats]

Something like striptags(tags="a, img").

That sounds like something that should be implemented outside of Tera itself

Functions as values in context

Can you expand?

[pranabekka]

That sounds like something that should be implemented outside of Tera itself

I suggested it here because striptags() is a Tera builtin, but I only know enough to use it with Zola and I won't complain. I guess the approach might be to override it or define a custom function.

Another thing I was thinking of was positional arguments, since I feel like some of the func(some-arg="") typing is a little unnecessary — instead, you could put in arguments in the order that they're declared.

The main blocker would be shortcodes, as far as I know, since it doesn't have a way of listing arguments, but that could be changed by using {% shortcode name() %} ... {% endshortcode %} blocks, and {% include %}-ing it, which would make it consistent with macros as well as other includes. You could then include shortcodes.html (or something like shortcodes/*) in the base template, to get essentially the same functionality.

[Keats]

striptags is essentially https://docs.djangoproject.com/en/4.2/ref/templates/builtins/#striptags

Positional arguments is not going to happen, I would use named arguments everywhere in Rust if I could as well.

Shortcodes are not part of Tera. They look like macros for similarity purpose but they are something implemented with a custom parser in Zola.

[Keats]

Anyone using {% call namespace.macro() %}...{% endcall %} and then caller() in the macro in Jinja2? It can be quite useful but it's a bit niche so I'm not sure whether to add support for it or not. In practice, I would probably make it work like Zola where you can call a macro in 2 ways:

// No body here
{{ macro::button(text="Submit") }}

// a magical body variable is injected as one of macro args
{% macro::submit(num=10) %}
  Hello {{world}}, I am rendered to string with the full template context 
  and passed to the arguments automatically. No {{ caller()}}
{% end %}

The main usecase (passing long content, branching etc) could be solved in a more general way with block assignment imo

[pranabekka]

Ah, thanks for the clarification

[Keats]

So the v2 is starting to look like something. I can render basic templates but haven't added support in the interpreter for inheritance/macros and filters/tests/functions. Inheritance/macros is all done on the parsing side, I just need to handle them. Filters/tests/functions I haven't decided yet their trait signature so that's going to happen later, parsing is done as well for them.

Perf wise it's currently slower than v1 but I haven't spent time on it, eg the teams benchmark is similar but the big table is about twice slower.

Only the happy path is handled, error messages will be worked once we are ok with the valid templates but everything is spanned already so it shouldn't be too hard to show good errors.

Are there some people interested in contributing at this stage? I'll make the repo public once it's in a better shape for more testing but for now mostly looking for help with the development. I'll invite anyone interested to the repo.

[Keats]

Macros and blocks are mostly implemented and I found some interesting tests copying them: https://github.com/Keats/tera/blob/master/src/renderer/tests/macros.rs#L274-L302

I am leaning toward removing that "feature" since the whole point of importing macros in Tera is to be explicit and this kind of goes against it. I think the fact that it was initially working was an accident more than anything.

[azjezz]

I am leaning toward removing that "feature"

I completely agree with this.

[eguiraud]

@Keats since you ask above: I for one would love to be able to pass a "body" to macros à la Zola shortcodes :)

It's handy when there is a multi-line variable. Sometimes it happens (while multiple multi-line variables are probably a much more niche use case).

[Keats]

I've added set blocks so you can do things like:

{% set body | i18n | safe %}
This is my body with some {{ content }}
{% endset %}
{{ macro::content(body=body) }}

which imo is nicer than the caller() syntax of j2. Slightly worse than jinja2 at the caller level but more usage than just macros.

[banocean]

Functions as values in context

Can you expand?

When you use one_off it would be useful to allow users to for example fetch resource needed. Passing functions in a context would enable this functionality without adding all useless stuff from the other now anavailble ways to do it. (I hope you understood everything, my English is terrible)

[banocean]

Is there any way to use parser and get data already parsed to analize it for potential optimalizations (for example not rendering raw strings) and rendering it like with Tera::one_off, but without string being parsed everytime? I I use it to allow users to set custom messages and this should have noticeable impact on the app performence.

[Keats]

Ok I've made the repo public: https://github.com/Keats/tera2 Still rough and no error handling but it's getting there. Let's create issues on that repo to keep things tidy

[eugenesvk]

The current parser library doesn't allow passing variables to the lexer and neither does the future lexer library :/ Would adding something [[, [%, [# as equivalent to the curly braces work with Latex?

I'd also love to have custom delimiters since besides the conflicting issues already mentioned I just don't like those ugly dupes

• {{}} and would prefer to use the 1-symbol and even for that symbol also visually smaller
• ‹›

Then the templating syntax is not "in your face" as much in the document

Similarly, all the other {% and {# delimiters could be replaced with much nicer unicode chars, thus freeing up these common symbols to use anywhere

(using custom delimiters in chezmoi config files, and it's a more pleasant experience)

Any chance for the future lexer to add this feature? Thank you

(not added the issue to the new repo to reference existing discussion)

[Keats]

It would be possible with the new lexer although I'm not sure if it will supported. If Latex etc work with Tera v2, it's probably not going to happen.

[jalil-salame]

One thing I've been missing with Zola is being able to run cli commands:

I would like to pipe the content of a block to something like dotviz to generate SVG graphs.

I don't know if this is a good idea as I can't seem to find any templating engine that does that.

Or maybe it's better to implement that in Zola itself.

[pranabekka]

@jalil-salame https://soupault.app is not really a templating engine, but it's built on using pipes to process data. You might even be able to use it after running Zola, to process code blocks with "lang-dotviz". There's an example here for generating SVGs from Graphviz.

[jalil-salame]

@jalil-salame https://soupault.app is not really a templating engine, but it's built on using pipes to process data. You might even be able to use it after running Zola, to process code blocks with "lang-dotviz". There's an example here for generating SVGs from Graphviz.

Looks like a good escape hatch, but Zola advertises being a "single binary", if you need a post processing step with a different binary it makes little sense, and you lose the ability to use zola serve to watch for changes and redeploy the website.

[Keats]

There's nothing preventing calling CLI tools in Tera, you just need to define a function that does it. Zola does not add a function like that because on purpose

[jalil-salame]

There's nothing preventing calling CLI tools in Tera, you just need to define a function that does it.

As far as I can tell, that is part of the Rust API, so you can't call a command from a Template, because that would be a bad idea, right?

Something like

{{ command "date" "+%I" }}

[Keats]

You would need to create your own function eg run_command and pass the args to so a call would look something like `{{ run_command(cmd='date %I') }}. Nothing like that in Tera by default but it would only be a few lines to add to your project if you want to (not in Zola though)

[eguiraud]

throwing it out here in case other people miss this: I would love to see maps become first-class citizens (e.g. having a syntax for map literals and the ability to add/remove keys from an existing map). Related issues: https://github.com/Keats/tera/issues/420 and https://github.com/Keats/tera/issues/673 .

[Keats]

Map literals are supported already in the tera2 repo linked above. I'm not too sure how to add/remove keys in a simple way from the syntax pov though, vs going through a filter

[eguiraud]

Ah, fantastic, sorry for the noise!

[Keats]

Anyone interested in doing performance improvement: https://github.com/Keats/tera2/issues/4 ? I've tried a few things but it's still slower than tera v1 for the big-table benchmark