Tera v2 wishlist/changes

[Keats]

On top of https://github.com/Keats/tera/issues?q=is%3Aopen+is%3Aissue+label%3A%22For+next+major+version%22

Parser

Operator precedence and expressions ✅

Parentheses should work everywhere and precedence should make sense. This is mostly already implemented in a new parser (private for now).

Should something like {{ (get_page(path=“some-page”)).title }} work? Right now it requires going through set which is probably fine imo.

Better error messages ✅

The new parser is hand-written so we can provide detailed error for each type of error. I’m also thinking of spanning all expressions in the AST for better rendering errors but not 100% convinced on that yet, will need to try.

Whitespace management

Do trim_blocks and lstrip_blocks (https://jinja.palletsprojects.com/en/3.0.x/templates/#whitespace-control) by default that can be switched on/off as one thing and not two.

Indexing changes ❔

Right now you can access array indices by doing my_value.0 or my_value[0]. This is in line with Jinja2/Django but it feels a bit weird.

How about allowing only my_value[0] so array index is the same as dict index and more like all programming languages (except tuple access)? The dot syntax doesn’t work when the index is a variable anyway which is probably the main usage of indexing. It would be a big departure from Django/Jinja2 though.

Features

call

https://ttl255.com/jinja2-tutorial-part-5-macros/#call-block is a good example of that feature. I do find the naming in Jinja2 very confusing though so it would probably be called something different

Investigating valuable ❔

https://tokio.rs/blog/2021-05-valuable This should avoid an awful lot of cloning and would improve performances a lot. Still not clear whether it will work though.

Improving filters/tests/functions

They should be able to get values from the context implicitely the same way expression do so we don’t have to repeat things like lang=lang in Zola for example. Maybe a first argument which is something like get_ctx: |key: &str| -> Option<Value>? It would be nice if there was a way to define the arguments in a better way than the current macros too. Related issue: https://github.com/Keats/tera/issues/543

Also remove some weird things added to match Jinja2 like https://github.com/Keats/tera/issues/650#issuecomment-1003735940

--

Any feedback / other items you want to change?

[KillTheMule]

Any feedback / other items you want to change?

I'm not yet a tera user, but I gave it a very serious look, because I want runtime templates (using askama for now, which uses compile-time templates). I'm templating latex files, which use brackets (in particular, the curly ones) all over the place, so what I'd need is different delimiters than {{ etc. Is that something you would consider? It would certainly be feasible to restrict this to 2-char delimiters (askama does this, I'm using ~< instead of {{ right now).

[Keats]

The current parser library doesn't allow passing variables to the lexer and neither does the future lexer library :/ Would adding something [[, [%, [# as equivalent to the curly braces work with Latex?

[KillTheMule]

Unfortunately, latex also uses [ / ] quite a bit as well, I can't imagine that working except for the most simple documents. I'm pondering trying addition pre/postprocessing steps, but that feels pretty brittle.

[Keats]

Does it use [[, [%, [# though? Single [/] would be ignored. Otherwise <<, <%, <# but it seems more common to me than squared brackets

[KillTheMule]

No, [[, [% and [# aren't really used and easily avoided if needed. Is that sufficient, though? Because one could also easily avoid {{ and the like, but other templates libraries if tried (e.g. jinja2) could not deal with the fact that there were single curly braces in the document.

[Keats]

Tera only cares about {{, {% and {# (and their closing equivalents), a single { or } will not be considered. See https://github.com/Keats/tera/issues/642 for a recent example.

[KillTheMule]

Huh, that would be awesome. You mention jinja2 in that issue, but I'm pretty sure it did trip over the brackets in the default settings (but jinja2 allows changing the delimiters, so I got it to work). Nevertheless, I'm taking tera for a test-drive tonight, thanks for your time, and thanks for tera!

[KillTheMule]

After using tera for some time (works nicely, thank you for it!), what I am missing is more fine-grained errors, in particular around rendering a template. When the template has a variable that is missing from the context, the error seems to be a generic Msg type, but I really want to give good feedback to my users (which are very non-technical people, and I cannot assume they speak english), which seems to mean that I have to manually parse the msg string and extract the info that I need. Moreover, if using array variables, it would be great to know which part of the chain was missing (i.e. if using var.field.subfield in a template, and var.field exists, it would be nice to have a possibility to not just show "var.field.subfield missing", but "subfield of var.field missing"... but that's just a shower thought :) A major improvement for me would be to get away from parsing msg strings manually.

[Keats]

So the next parser has spanned expression pointing to the exact span in the template. Getting which value didn't exists in an expression like var.field.subfield is a bit more annoying but might be doable. That's a good idea.

[MahaloBay]

Hi some ideas for v2 : I believe that there is currently no possibility of negation (or I misread the doc ^^) For example, i can {% if my_var %} but not {% if !my_var %}, often I found myself having to do {% if my_var %} {%else%} blablabla {% endif %}, same thing in "is containing", i think negation is a good thing to add

EDIT : i search in doc and i find this Maybe i was misread and should using not

[Keats]

Yep, you have to use not. So {% if not my_var %} and {% if x is not containing(..) %}

[jsha]

Hi! We're starting to use Tera in rustdoc. I like it a lot so far. One thing we're really interested in is performance. Rustdoc generates a lot of HTML in a single run, and there's often a developer waiting to look at the output, so we care a lot about speed. Also, the rustdoc team has put in a lot of effort speeding it up, so we want to make sure the move to templates doesn't slow things down again.

Right now I'm investigating a perf regression after adding a second template. Some themes that come up:

• malloc and free are responsible for a lot of the regression, suggesting number of allocations might be an issue
• regex is also fairly big, suggesting get_json_pointer could be sped up (#678).
  • BTreeMap::insert shows up high, suggesting that Context::from_serialize is costly.

The valuable crate looks very useful for reducing allocations, and possibly also for replacing the BTreeMap. My thinking is that when the input is a single struct, tera should be able to directly visit each field of the struct by name without building an intermediate BTreeMap of field names -> values.

[Keats]

We're starting to use Tera in rustdoc.

:o nice!

Performance is the main goal for v2 as well, I'm guessing a lot of allocations you see are from the JSON serialization, which is the bottleneck for Zola as well (minus syntax highlighting). I'm guessing rustdoc is the same (lots of text) that need to be essentially cloned when moving to Value::String.

I have some big hopes for valuable to reduce/remove those allocations but it seems the project stalled a bit.

[jsha]

Has it stalled? Looks like they're planning an initial release as of 8 days ago.

[Keats]

I am looking at https://github.com/tokio-rs/valuable/pull/59 which would be required for Tera

[KillTheMule]

One thing I'm wishing for right now was a way to get a template from tera to inspect it. In my case, I can probably just read the file from disc again, but depending on the structure of the program it might be worthwhile to provide something like that (never mind that it might just be faster than reading it again). Maybe a use case would be modifying those on the fly? Or check for certain content? The latter is what I want to do btw.

[Keats]

You can access a template AST, it's just not visible in the documentation since the AST is not stable.

[Keats]

It would be nice to have context local functions, it was pulled from v1 due to some breaking changes but it's on the table for v2

[ssendev]

I would like to have something like Components from Vue.js which in Tera would probably be macros 2.0. The difference to current macros being that they not only take arguments but also can have slots blocks.

{% macro user_list(users, user_icon="👤") %}
<ul>
  {% for user in users %}
  <li>
    {% block icon(user_icon, user) %}
      {% if user_icon %}<i class="icon">{{ user_icon }}</i>{% endif %}
    {% endblock %}
    {% block name(user) %}{{ user.name }}{% endblock %}
    {% block default(user) %}<a href="/users/edit/{{user.id}}">Edit</a>{% endblock %}
  </li>
  {% endfor %}
</ul>
{% endmacro %}

{# this uses the default content of all blocks #}
<user-list users="{{users}}"/>
{# rendered #}
<ul><li><i class="icon">👤</i>Aron<a href="/users/edit/1">Edit</a></li></ul>

{# children will be assigned to the default block #}
<user-list users="{{users}}" icon="🧍">
  <span class="disabled">Edit</span>
</user-list>
{# rendered #}
<ul><li><i class="icon">🧍</i>Aron<span class="disabled">Edit</span></li></ul>

{# blocks can be overridden and receive the arguments that are passed #}
<user-list users="{{users}}">
  {% block icon(user_icon, user) %}<i class="icon rounded {{user.color}}">{{user_icon}}</i>{% endblock %}
</user-list>
{# rendered #}
<ul><li><i class="icon rounded red">👤</i>Aron<a href="/users/edit/1">Edit</a></li></ul>

{# when the default block needs arguments it can be called via its name #}
<user-list users="{{users}}">
  {% block default(user) %}<a href="/users/delete/{{user.id}}">Delete</a>{% endblock %}
</user-list>
{# rendered #}
<ul><li><i class="icon">👤</i>Aron<a href="/users/delete/1">Delete</a></li></ul>

{# it would be possible to allow assigning the default block arguments directly but would cause confusion #}
<user-list users="{{users}}" {{ default(user) }}>
  {{ user.name }} {# this works #}
  {% block icon(user_icon) %} {# notice that only the first argument was retrieved #}
    {{ user.name }} {# this doesn't work since user is not defined #}
  {% endblock %}
</user-list>
{# and explicitly calling the default block would still be needed to empty it #}
<user-list users="{{users}}">
  {% block default() %}{% endblock %}
</user-list>
{# rendered #}
<ul><li><i class="icon">👤</i><a href="/users/edit/1">Edit</a></li></ul>

{# shorthand : to avoid {{ }} like in Vue.js and other templating languages might be nice #}
<user-list :users="users">

Using HTML like syntax to call macros might be controversial but I think the result is really neat and leaves the heavy syntax {% %} for control flow more distinct. Syntax wise a similar result could be achieved by leveraging web components but I think the point of server side rendering is to avoid JavaScript which this wouldn't. It's also possible to go all in on web components and make even the macro definitions use HTML syntax via <template> and <slot> but that might be a little too much.

I think this would be a very nice addition to Tera and would allow for seamless use of e.g. Tailwindcss

edit: naming them slot instead of block might still be desirable to avoid ambiguity

[Keats]

That's extremely unlikely to happen. Tera is not only used for rendering HTML so it can't just change its syntax for one usecase. Something like https://jinja.palletsprojects.com/en/3.0.x/templates/#call is likely to be added though.

[ssendev]

Ah of course then that syntax won't fly.

Call is already halfway there but would be improved with the ability to define multiple callers/yield points and to have default content if it's not provided at the call site.

{# the original definition syntax would work but here is an alternate proposal that's closer to call #}
{% macro user_list(users, user_icon="👤") %}
<ul>
  {% for user in users %}
  <li>
    {# a named caller can be defined like this #}
    {{ caller prepend(user) }}
    {# if caller is in {% %} it's a block with default content #}
    {% caller icon(user_icon, user) %}
      {% if user_icon %}<i class="icon">{{ user_icon }}</i>{% endif %}
    {% endcaller %}
    {% caller name(user) %}{{ user.name }}{% endcaller %}
    {% caller(user) %}<a href="/users/edit/{{user.id}}">Edit</a>{% endcaller %}
  </li>
  {% endfor %}
</ul>
{% endmacro %}

{% call(user) user_list(users) %}
  <a href="/users/delete/{{ user.id }}">Delete</a>
{% icon(user_icon, user) %}
  <i class="icon rounded {{ user.color }}">{{ user_icon }}</i>
{% endcall %}
{# rendered #}
<ul><li><i class="icon rounded red">👤</i>Aron<a href="/users/delete/1">Delete</a></li></ul>

To have a library of reusable components macros having these options is extremely useful as can be seen by looking at Vuetify where almost every macro component has multiple callers slots with the most egregious offender probably being v-data-table

[heroin-moose]

A variant of new() that creates empty Tera instance. For example, I'm writing an application that provides templating. The templates are stored in /usr/share/foobar/templates. However, a user can overwrite them by placing templates to /etc/foobar/templates. In order to keep {% include .. %} working the templates are referenced by relative paths. So basically I need two sets of identically named templates which I obtain by using WalkDir and stripping /etc/foobar/templates and /usr/share/foobar/templates.

[Keats]

Isn't that Tera::default()?

[heroin-moose]

Indeed. Thanks, I haven't thought of it.

[rinmyo]

it will be amazing if tera is able to access context. i'm working on KaTeX SSR integration for zola and I really need this feature!

[Keats]

Yep, accessing the context from filters/functions is definitely something I want to add, partly for Zola as well (so we don't do lang=lang etc)

[Piping]

is it possible to add a hex function or hex filter to put a number into hex form?

[nulltier]

A note comment about new hand-written parser isn't clear. @Keats are you going to kick off the pest grammar generated parser?

[Keats]

Yep, I have a version currently written with https://github.com/maciejhirsz/logos and started another with no dependencies at all. Note that I haven't touched it for a while, I'll resume after the next Zola release probably.

[nulltier]

Do you work on them in private? I see no dedicated branches.

[Keats]

It's on a private repo right now as it's just me playing around so far

[ghost]

Hello

Any feedback / other items you want to change?

Block assignments. I think assigning only expressions is too limited to use

For example, with block assignments:

{% set title %}
    {% block title %}{% endblock title %}
{% endset title %}

(#265 related?)

{% set important_text %}
    <strong>Don't forget this</strong>
{% endset title %}

I believe there's better examples..

[bemyak]

Just found this issue, so leaving my 2 cents here.

I'd like to have an automatic way of checking if no additional arguments were passed to a function / all arguments were handled. This is very useful to catch typos in optional arguments. Currently args are passed in a HashMap, so the function implementation needs to check that (and they usually don't).

It would be cool to have an ability to register some type for arguments deserialization, and have all this logic handled by tera.

[bemyak]

Also, unnamed function parameters (like in python) would be nice. It's really annoying to lookup e.g., get_url function every time to check what was the name of the first argument: was it value, path, link, page or something else?

[Keats]

I'd like to have an automatic way of checking if no additional arguments were passed to a function / all arguments were handled.

It's kind of hard some functions can have arguments validation hard to represent, eg a function that can take either a path or a url and one of them is required.

Also, unnamed function parameters (like in python) would be nice.

Very unlikely to happen. I'm a huge fan of keyword parameters as a way to document the code. Having to remember the order of the parameters feels worse to me, and you end up with stuff like get_something(url, true). If you look at it again in 6 months you probably forgot the arguments of get_something entirely so having the name there helps a lot. I can't have kwargs in Rust so at least I'll get them in Tera ;(

[bemyak]

I'm a huge fan of keyword parameters

Me too! Having to specify all of them is much better than having none.

However, there are many use-cases were they are just cumbersome e.g., val | default(value=1) has more noise compared to val | default(1).

I'm proposing to have a mix of both (like python has) e.g.,: resize_image('my_picture', width=100) is really clear to read. Unnamed params should go first ofc.

[Keats]

Yea I understood but adding positional arguments makes it more complex. Maybe I will add it but it's unlikely

[ssendev]

How about adding just one positional argument that way you avoid the possibility of get_something(url, true) and still make live easier in most cases. To make implementation easier it would be possible to add it under a known key like first_arg to the Hashmap.

[bemyak]

Stumbled upon another minor thing: I wish macros had support for a body variable like shortcodes in Zola, so you could use a macro like this:

{% macros::some_macro() %}
…
{% end %}

currently I think only one-line invocation is supported: {{ macros::some_macro() }}

This could be really handy in certain cases!

[Keats]

It would be https://jinja.palletsprojects.com/en/3.0.x/templates/#call in Jinja2. I agree it's very useful

[XAMPPRocky]

Doesn't have to be a breaking change, but something I would like, is if the filesystem layer of tera was abstracted out so that as a user, I can allow people to import files in their Tera templates that aren't directly in the filesystem, for example macros / files that are included with the binary, or available remotely through a url.

[Keats]

You can already add templates via https://docs.rs/tera/latest/tera/struct.Tera.html#method.add_raw_template unless you meant doing that in the template directly? If so that's not going to happen.

[XAMPPRocky]

unless you meant doing that in the template directly?

Yes I mean having the following work without using std::fs.

{% extends "base.html" %}
{% block title %}{% endblock title %}

If so that's not going to happen.

I think you should reconsider. Currently being tied to std::fs severely limits the environments and applications that you can use Tera as a library. For example; You can't render complete Tera projects in the browser because std::fs does not work in the browser, and there's no way catch the file system calls to redirect to in memory blobs. If Tera provided a way to catch these calls in the library you could write interactive editors and other tools without requiring a filesystem.

FWIW I think a minimal version of the change would have a small impact to Tera's API and code. I would be happy with a trait that provided a simple string to string lookup.

pub trait FileProvider {
     fn read<A: AsRef<str>>(&self, path: A) -> Result<String>;
}

// Default filesystem to use if none provided.
pub struct StdFs;

impl FileProvider for StdFs {
     fn read<A: AsRef<str>(&self, path: A) -> Result<String> {
          std::fs::read_to_string(path.as_ref()).map_err(From::from)
     }
}

[Keats]

Yes I mean having the following work without using std::fs.

You can already use Tera without using any files: https://tera.netlify.app/playground/ There's only one template in the playground but you could use as many as you want. Rhe name in extends just refer to a template name, which is taken from the filename. Nothing else happens with the filesystem after the templates have been read:

// we need to read it there because it's using the glob
let tera = Tera::new("templates/**/*")?;
// and we're done with the filesystem, it's never touched anymore by Tera

But we can also load it like in the example of https://docs.rs/tera/latest/tera/struct.Tera.html#method.get_template_names no files involved.

[XAMPPRocky]

Ah you're right, I just missed that extend and such use the template names and assumed it was filesystem based. Thanks for the clarification!

[Keats]

Anyone using the array form of include as in include ['a.html', 'b.html'] and ignore missing? cc @richardchien who implemented it. I can't find any examples of it in use in jinja2 and I can't think of a usecase either where I'm ok with missing content.

[stdrc]

Anyone using the array form of include as in include ['a.html', 'b.html'] and ignore missing? cc @richardchien who implemented it. I can't find any examples of it in use in jinja2 and I can't think of a usecase either where I'm ok with missing content.

When a static site generator needs to allow users to provide custom templates, this feature will be useful. I use it in my Jinja2 template and that's why I implemented it in Tera when I was rewriting my static site generator in Rust.

[Keats]

Hm Zola does support that transparently by moving the logic to the SSG rather than the template engine. I might remove it in v2 if there are no other compelling usecases.

[stdrc]

IMHO it's not a bad idea to be more compatible with Jinja2 as long as having a set of tests, but it's up to you anyway.🤔

[Geobert]

I don't think it's possible with current Tera but I asked this on the forum: https://zola.discourse.group/t/map-a-filter-on-an-array/1460/2

I would be an awesome addition for v2 :D