Tera v2 wishlist/changes

[Keats]

On top of https://github.com/Keats/tera/issues?q=is%3Aopen+is%3Aissue+label%3A%22For+next+major+version%22

Parser

Operator precedence and expressions ✅

Parentheses should work everywhere and precedence should make sense. This is mostly already implemented in a new parser (private for now).

Should something like {{ (get_page(path=“some-page”)).title }} work? Right now it requires going through set which is probably fine imo.

Better error messages ✅

The new parser is hand-written so we can provide detailed error for each type of error. I’m also thinking of spanning all expressions in the AST for better rendering errors but not 100% convinced on that yet, will need to try.

Whitespace management

Do trim_blocks and lstrip_blocks (https://jinja.palletsprojects.com/en/3.0.x/templates/#whitespace-control) by default that can be switched on/off as one thing and not two.

Indexing changes ❔

Right now you can access array indices by doing my_value.0 or my_value[0]. This is in line with Jinja2/Django but it feels a bit weird.

How about allowing only my_value[0] so array index is the same as dict index and more like all programming languages (except tuple access)? The dot syntax doesn’t work when the index is a variable anyway which is probably the main usage of indexing. It would be a big departure from Django/Jinja2 though.

Features

call

https://ttl255.com/jinja2-tutorial-part-5-macros/#call-block is a good example of that feature. I do find the naming in Jinja2 very confusing though so it would probably be called something different

Investigating valuable ❔

https://tokio.rs/blog/2021-05-valuable This should avoid an awful lot of cloning and would improve performances a lot. Still not clear whether it will work though.

Improving filters/tests/functions

They should be able to get values from the context implicitely the same way expression do so we don’t have to repeat things like lang=lang in Zola for example. Maybe a first argument which is something like get_ctx: |key: &str| -> Option<Value>? It would be nice if there was a way to define the arguments in a better way than the current macros too. Related issue: https://github.com/Keats/tera/issues/543

Also remove some weird things added to match Jinja2 like https://github.com/Keats/tera/issues/650#issuecomment-1003735940

--

Any feedback / other items you want to change?

[onkoe]

Please excuse me if this already exists, if you don't mind. I'd love to have optional arguments to macros that are non-bool. For example, let's say we're making a "button" macro for some Zola HTML:

{% macro button(label, href="#") %}
<a class="btn" href={{href}>
   <div>{{label}}
</a>
{% endmacro button %}

In my view, this should allow users to make "temporary" buttons like so:

{{ button::button(label="hi im a button!") }}

As-is, you need to give a value for all non-bool arguments to the macro, which can get real messy... real quick!

If this feature exists, could someone please guide me through it? Maybe I'm blind, but the docs didn't seem to mention "optional macro arguments" explicitly. Thanks for the fantastic project!

[Keats]

Hmm that exact example should already work. Any literal is allowed as default for optional arguments

[onkoe]

@Keats, I see! I'm going to post my whole problem, then. Please don't forget to mark it as unrelated if you find it unhelpful.

Here's the 'real version' of the example I gave:

{% macro button(label, type="default", color="primary", href="#", other_classes="", left-image="", right-image="") %} 

<link rel="stylesheet" href="{{ get_url(path="button.css", trailing_slash=false) }}"/>

{% if type == "default" %}
    <div class="btn-wrapper btn-wrapper-{{color}} {{ other_classes }}">
        <a class="btn" href="{{ href }}">
            <div>{{label}}</div>
        </a>
    </div>
{% elif type == "extended-fab" %}
    <div class="btn-wrapper btn-wrapper-{{color}} {{ other_classes }}">
        <a class="extended-fab-btn" href="{{ href }}">
            {% if left_image != "" %}
                <img src="{{get_url(path=left_image, trailing_slash=false) | safe}}" />
            {% endif %}

            <div>{{label}}</div>

            {% if right_image != "" %}
                <img src="{{get_url(path=right_image, trailing_slash=false) | safe}}" />
            {% endif %}

        </a>
    </div>
{% endif %}

{% endmacro button %}

After doing this, I get a funky little error message from Zola, requesting that my default values be boolean only.

Change detected @ 2023-10-19 19:09:43
-> Template changed /Users/barrett/Documents/barretts-club/templates/macros/button.html
Reloading only template
Error: Failed to build the site
Error:
* Failed to parse "/Users/barrett/Documents/barretts-club/templates/macros/button.html"
 --> 1:81
  |
1 | {% macro button(label, type="default", color="primary", href="#", other_classes="", left-image="", right-image="") %}
  |                                                                                 ^---
  |
  = expected `true` or `false`
Done in 7ms.

If I abide by its suggestions and use a boolean instead, it proceeds to go after my other optional arguments, such as href="#".

Is this more of a Zola issue, or am I misusing the macro system? Thanks for your quick response, by the way! 🤩

[Keats]

The issue is the naming of left-image and right-image parameters. Replace the dash with an underscore and it will work. No clue why the error is showing something else though.

[Keats]

https://github.com/Keats/tera2 has made some big progress with perf improvements and nice errors.

The next step is redesigning filters/tests/functions: https://github.com/Keats/tera2/issues/5 Maybe it's going to be like the current ones except we automatically pass a fn to grab data from the current context but curious to see other opinions!

[clarfonthey]

I didn't see it in the issue description or the tera2 readme, but it would be nice to have some way to prevent you from passing extra arguments to macros that aren't used. For example:

{% macro mymacro(one) %}
    {# ... #}
{% endmacro mymacro(one) %}
{{ self::mymacro(one="hello", two="world") }}

Should just be an error. I noticed this when changing the names of arguments and forgot a few calls.

[onkoe]

I'd love to see more static typing in Tera (and, thus, Zola)! Sometimes, it feels a bit unsafe or 'rowdy' to make specific things.

For example, one of my sites has a button macro, somewhat like a component in giant web frameworks. You can place arguments in any order, give incorrect data, or feed it some state that just isn't right.

For this, something like an {{ assert(arg.literal_type == "boolean") }} would be fantastic! Not sure how feasible it is, of course... ☺️

Just my ¢2. Thanks for the hard work on v2 - it's looking nice!

[Keats]

Should just be an error. I noticed this when changing the names of arguments and forgot a few calls.

That's a good idea! Tracking in https://github.com/Keats/tera2/issues/23 but that should be easy to implement.

You can place arguments in any order, give incorrect data, or feed it some state that just isn't right.

The any order is a feature. As for types you do have some tests for it in Tera v1 https://keats.github.io/tera/docs/#built-in-tests but it's not as great. Is it only for macros? We could potentially add type definition to them but I don't really want to go complex and start having type union, enums, optionals, string literals... It is appealing though

[bemyak]

Not sure how big it is, and even if it should be a part of Tera, but having the ability to add custom filters (and tests/functions?) as WASM plugins would be amazing! It will also (partially at least) solve the long-standing "Zola plugins" topic.

The idea is that you define a simple interface, with string input and output. Libs that implement this interface are compiled into wasm binaries and can be dropped into some folder (/plugins) to be loaded into Tera automatically. Users can share those easily, so we can have a great plugin ecosystem eventually, which covers even the niche use cases without the need to bloat Tera and Zola.

I think Zellij can be a good example of using WASM plugins: https://zellij.dev/documentation/plugin-lifecycle

[Keats]

I was more thinking of using something like lua or rhai than WASM for that usecase. Either way, it's not going to come with v2 straight away, it can be added later.

[clarfonthey]

Since Tera can just add arbitrary Rust code as filters/tests/functions, presumably that could be done as a separate crate/library rather than as part of the engine?

Since it would simply involve creating a wrapper that runs the WASM/Lua code as part of the call function.

[bemyak]

I think that implementing this kind of extension in Rust would be much easier than in Lua/Rhai because of the rich ecosystem. For example, implementing the deflate (de)compression filter took me only a handful LoC and was pretty straightforward. Doing this in Rhai would be much more involved since no ready-made library exists for it.

The other concern is security: do we allow the extensions to read arbitrary files from disk or make network calls? WASM solves this by running modules in a sandbox, and we explicitly grant permissions to them (or so I heard).

[Keats]

The downside of using Rust/wasm is that you get an opaque blob that vs a readable file. Sure it can do less but maybe it's an ok tradeoff?

[RafaelKr]

Twig has a use directive. It's described as Horizontal reuse is a way to achieve the same goal as multiple inheritance, but without the associated complexity:

The documentation describes its use cases and benefits pretty well: https://twig.symfony.com/doc/3.x/tags/use.html

Maybe something that could be taken into consideration for (future) implementation.

[Keats]

Horizontal reuse

That's not going to be added, it seems super niche.

Last call if people have ideas/needs for https://github.com/Keats/tera2/issues/5

[clarfonthey]

One thing that would be nice, but I'm not sure how it would be done, is the ability to make macros that return actual tera values instead of just strings. I know that you can technically JSON encode/decode to get around this, but there isn't actually a JSON decode function in tera by default.

[bemyak]

@clarfonthey, I agree, something like nushell typing would be really nice!

[Keats]

There's no JSON anymore in v2. It would have to be something other than macros though because macros are pretty explicitly only for txt output. What's the usecase?

[speatzle]

I would like to see some way to handle Enum Variants and their values with if or possibly via a match/switch block.

[clarfonthey]

There's no JSON anymore in v2. It would have to be something other than macros though because macros are pretty explicitly only for txt output. What's the usecase?

In my case, a lot of the macros I made ended up having to deal with multiple inputs/outputs, and this effectively means that things no longer can be macros in this state. Things like:

• Working with date ranges and then displaying them a particular way
• Trying to load values from Zola in a particular way where they were overridden in parent sections, but I couldn't return the result unless it was a string

And in general, you end up with macros being stringly-typed, where you can do things like return numbers, but they all have to be implicitly converted into strings and back at any macro boundary. This also makes refactoring macros difficult because sometimes you just can't split something in half, because you're holding onto more than one, or non-string variables, and you need to have a string to return.

You could argue that all of this is beyond tera's main use cases, but considering how you do have far more than basic functionality (like, for example, handlebars), I'd say it's potentially worth considering. Even a form of macro that simply let's you keep any local variables set when it returns could probably help.

[Keats]

I'm working on the functions (filters etc) and I'm running into some type issues: see PR at https://github.com/Keats/tera2/pull/35

The goal is to define a trait so we can borrow from the context while casting to the right type. Easy example would be a fn escape(&str) filter. I got it to work for owned values but I can't figure out the right incantation of traits to make it work with borrowed ones. If anyone has time to have a look that would be much appreciated. You can revert the last commit to get to where i was.

--

I would like to see some way to handle Enum Variants and their values with if or possibly via a match/switch block.

We don't have the concept of enums in the Value so I'm not sure how that would work

--

Re: macro with Value outputs I'm not sure about that. I think if you need a lot of logic, it's better to move it out of the template. Maybe it could be solved with better object manipulation in the language itself?

[Keats]

There's been lots of progress on filters: https://github.com/Keats/tera2/pull/36

The overall system is implemented and I've re-implemented some of the builtin filters from tera v1: https://github.com/Keats/tera2/pull/36/files#diff-52805b1327ccbe8f318467f14b1b1cf1e4573e08919d4023c18ceb8800dc1377R116-R141 Every filters that require a dependency will go in another crate so it can be changed without affecting tera itself.

Main things missing:

• a 3rd parameter to filters for a struct that will be passed automatically: I can't think of a good name right now but it will allow fetching values from the current context, get filters/tests to execute in other filters etc
• the map/group_by/filter filters: now that we have both Null (equivalent of None in Rust) and Undefined (something not found) semantics need to change a bit and I wasn't sure yet how to make them work (eg error on undefined or not, add a default if null etc). Any ideas?
• probably some filters I've missed or want to have another look at/add more tests like the indent one

There are some filters that I didn't reimplement yet just because I wasn't sure if anyone is actually using them (eg linebreaksbr for example). Definitely looking for more feedback there as well.

Once this is fully implemented, I'll add tests and function but it's going to be very straightforward/quick once the filters are done and we should have a mostly working Tera v2.

But it's not done yet! Now that there is https://github.com/mitsuhiko/minijinja I think we can diverge more from Jinja2/Django (truthfully they might have some of those features, I haven't checked).

I'd like:

• optional typing for macros
• easier support for vec/map manipulation in the templates: not sure exactly what yet
• global macros: macros you don't need to import, could be useful if you are using them for components and keep one in each file and you don't want to have to import "macros/button.html", "macros/input.html", etc on each template.
• validate as many things as possible at compile time: eg if you use {{ price | arrondi }} it should error when compiling the templates if that filter is not registered. I'm doing that for macros right now but not keeping the spans around so it's not as nice as it can be, eg i'm just saying the template X is calling macro Y which is not present. We're not going to get to askama level but that's fine, it's not the same usecase.
• ??

[onkoe]

Mini-bikeshedding moment: is the Null keyword settled on? It sounds a lot scarier than None, at least to me.

Speaking on the actual implementation, what is the difference between Null/None and Undefined? Neither give context about what went wrong, except "Undefined" representing that getting something failed. If you do decide to keep both keywords, clarification through documentation would be necessary here.

Finally, I'd prefer that lacking a value (and using it) prevents the site from rendering. Otherwise, you'd be inheriting a lot of JS-like baggage that'd make for some irritating bugs.

After all, how do I render Null? Sounds painful! 😄

[Keats]

Mini-bikeshedding moment: is the Null keyword settled on? It sounds a lot scarier than None, at least to me.

Mostly inspired by JS null/undefined tbh since they represent the same concepts. Eg if you have data "data": {age: 18, country_id: None} in you context:

{{ data.age }} -> prints 18
{{ data.country_id }} -> prints nothing
{{ data | country_id | default(value=1) }} -> prints 1
{{ data.doesnt_exist }} -> will error

Undefined is going to be only (I think) used internally, it won't be exposed to the end user in templates. It is different from Null since the field is there, just that it has a None value. I don't know if we should error in the case where we are printing a null value, it's not necessarily an error like undefined.

[Keats]

I've added indent/group_by/filter and made an issue to discuss filters specifically: https://github.com/Keats/tera2/issues/37

[Keats]

https://github.com/Keats/tera2 now has filters/tests/functions implemented and should be mostly working and is ready for some beta testing.

Next steps are:

• fixing whatever issues are found (please open issues on the tera2 repo rather than this thread so it's easier to track)
• add a format filter using the Rust format syntax
• add filter/tests/fn function compile time checks
• Easier modification of arrays/maps

And probably some more things like the global macros/macro types/rendering a macro specifically along the way. The macro types can be added later on so it will probably be worked last unless someone wants to implement it.