Twig template engine

[Nercury]

Hello!

I don't know if you are aware, but we are basically writing the same open-source project, Twig for Rust. The end result is going to look a bit weird - two Twig engines for Rust!

I am creating this issue because I can't find any other way to contact you. My wish is to figure some way forward so that we both be happy about it.

[colin-kiegel]

Hi, thanks for contacting me!

Ok, this is indeed interesting. :-) When I started this project I actually checked for similar projects, but couldn't find anything at all.

Would you suggest, that we merge our efforts? If so, how?

[Nercury]

Yes, merging our efforts would probably be the most sane of things (and very awesome).

I have one initial idea.

Some of twig implementation is actually PHP implementation. I found this part very nasty to deal with. For example, I had to rewrite code in rust that figures out if a string is numeric. This was based on PHP C implementation. However, I rewrote it with unicode in mind. In your code, I noticed php stripslashes. I avoided doing that so far, but I will have to basically re-implement this.

So, maybe we can move these things into some kind of common lib. Name it template_ops or something like that (I would avoid PHP or Twig in name).

Next, we can probably merge the lexer somehow. My lexer is direct re-implementation that has performance issues because of utf8 string indexing. Your lexer looks a lot cleaner. I don't actually care as long as it spews the same token stream and passes my tests ;)

As for the Parser, my approach is to try and avoid simulating inheritance completely, instead of that relying on strict type hierarchy. I decided this is possible by counting the number of instanceof in original Twig :)

As for the parser and runtime work, I think that my current attempt is still very big experiment which might work or may fail.

[colin-kiegel]

Ok, cool.

Yes, there seems to be quite some php-specific stuff in twig. And I am also a bit afraid of what might still be ahead. At the same time I am wondering whether everyone would be enthusiastic with php-quirks reimplemented in a rust library. So maybe there will be situations where you want to choose between full php-twig compatibility and some rust alternative. The if-statement (or the expression within) could be a candidate to be duplicated with slightly different behaviour in something like core-php-compat and core-rust (not sure about that, yet).

I like the idea to merge our repositories incrementally, by extracting common stuff into a helper-library and exchanging implementations.

I suggest, that we try to streamline some basics first, like:

• error handling + definitions
• lexer tokens

If we synchronize this, it will be much easier to exchange implementations like the lexer. What do you think?

My error handling is basically this:

• return err!(ParserErrorCode::Logic, "Unexpected end of token stream"), where filename:line:column and module-path is collected by the err!-macro automatically. It also supports formatting err!(..., "Hello {name}", name = "world"). This is something I find quite useful.
• and I use try!( .... ) a lot, where try! will convert different error-types for me, because my error-types implement a couple of Into-traits, e.g. a SyntaxError implements Into, via this macro: impl_convert_exception!(SyntaxErrorCode, LexerErrorCode, LexerErrorCode::SyntaxError);

Definitions are in: src/error/* - but I am sure, it has room for improvement. It definitely needs some cleanup, due to boiler-plate when defining new error definitions. I will have a look at your way of dealing with errors. It would be great, if we find a common way here. That will make everything else a lot easier! What do you think?

I will have a look at your code, too. If you want me to change anything in my codebase (like error-/token-definitions), feel free to send me a pull request. :-)

[Nercury]

For the PHP quirks, I suggest we remove them if the removal would not break PHP-Twig template. For example, I think we can easily add proper unicode support to all these low-level functions.

Twig itself is quite good at hiding PHP quirks, but maybe it is possible to simply choose not to implement something if some quirk bleeds through. I have yet to find something majorly bad.

I would be reluctant, however, to provide incompatible implementation. I think it's better to document what won't work instead of providing implementation that silently works differently.

For helper library, I have invited you to rust-web organization I have just created :).

It looks like my error handling is way crazier than yours. I am keeping all errors in enums/structs and then converting them to string on actual display. There are 3 groups: template for lexer and parser errors, engine for loading/caching, and runtime which is a bit in flux. On top of that, there is master twig Error which can contain any of these children, and in case of engine/runtime, the chain of error causes. Of course, all chilren are implementing Into trait to be converted to "master" error.

I have found one idea useful: separating location from actual error. There is this At struct which wraps actual error inside. To convert to it, there is method on error named at, which requires location in file. So I do this:

TemplateError::ExpectedTokenTypeButReceived(
    (expected.into(), Received::EndOfStream)
).at(line)

Note that I only have line numbers, but I was planning to expand this to line/col later.

Tokens are also a bit different. I was keeping string tokens as slices that reference original string. So the full list of lexed tokens does not exist, only the iterator over them, which lexes while iterating.

That presents one limitation: I can't transform strings in token when lexing (for stripslashes/etc). That has to moved to later phase.

I browsed your code a bit already. Can you explain the idea of jobs?

[colin-kiegel]

I am so excited, thank you. :-)

I joined rust-web and created a twig team and twig repository. I would recommend we leave the twig repository more or less empty right now. We might add a readme pointing to both our current implementations. And we can use the issue-tracker there to track our streamlining progress. If we have some basic layout, we can start to push actual code there. Are you ok with this suggestion?

Compatibility: I agree, the primary goal should be compatibility with PHP-Twig templates. I was just thinking about additional flags, which could be set voluntarily like a "strict mode", where some php quirks would be rejected to target an audience of non-php-folks, who would otherwise choose a different template engine. But these are merely vague thoughts about future steps and not really relevant right now. The twig template engine should be flexible enough to support different modes - like a mustache-mode with 100% mustache-compatibility. The default mode should be 100%-twig-compatibility, if possible.

Jobs: The original reason to introduce jobs was to separate mutable and immutable stuff. E.g. after initializing the lexer/parser/... (loading extensions and stuff) these objects are supposed to stay immutable - at least during a job-run. While each job will have a highly mutable state. This is preparation for running different jobs in parallel. And I am also highly concentrated on avoiding runtime-stuff like rc, cell, etc. - which is not always easy, but so far, it lead me to interesting design decisions. Jobs originated in an attempt to avoid rc/cell-stuff - and it was only afterwards, that it occured to me, it would be quite useful for parallelization. This is why I duplicated this concept for lexer/parser and runtime.

[colin-kiegel]

I initialized https://github.com/rust-web/twig with a basic skeleton. We can use pull requests here to integrate all parts, we agree on. :-)

I will try to take a look at error handling this week..

PS: The complete merge will probably take some time, I guess.

[colin-kiegel]

PPS: One long-term "goal" of mine is JIT-compilation with LLVM - where each template would be compiled to machine code dynamically linked to the template engine during runtime. But I have absolutely no LLVM-knowledge so far - I guess this will be a lot of work (hopefully fun, too). ^^

[Nercury]

Great!

My idea was to create a very minimal runtime that can execute a very simplistic instruction set.

So, when you ask the twig to "execute" the template, it would go like this:

• If there is cached instructions for this template, and environment hash for cached instructions matches current env hash, run these instructions.
• If not, lex, parse, compile it into instructions, and do the same.

The plan here was to eliminate all indirection related to extensions (everything is resolved into instructions), and abstract the "compilation" away from engine.

So right now I have this little-rs project that has two backends: the interpreter, that theoretically can work, and compile-to-rust, which requires refactoring everything (because I used goto :) ).

PS: Yes, there is going to be way more talking now until we build our new rust template engine

[colin-kiegel]

Ok, cool. Your runtime idea seems far more evolved than mine. I understand it's some kind of byte code executed by a virtual machine interpreting this bytecode. Feel free to correct me, if I am wrong. :-)

I was going to do implement "execution" in two variants:

• my very very simple approach to get things going: Each node in the AST will have some execution logic - i.e. the tree will just execute recursively. Of course this has a lot of bad indirection - execution of child nodes will be a vtable call. But this should be easy to implement and could help to detect errors in more sophisticated approaches.
• future plan: compiling to machine code - no virtual machine involved. The target would be to get templates as fast as if they were written in C from the start. But I really don't know how difficult that will be. Maybe it's better to compile to rust first as an intermediate representation.

I guess our execution-approaches are kind of orthogonal to each other, right now. I would suggest we keep both approaches, so we can kind of double check if everything is working as expected: Something like "throwing random ASTs at both execution backends" could be used for automatic testing.

[williamdes]

Hi !

:ping_pong: 5 years later

I need Twig because it is cool, etc..

Do you think one of the implementations could be finished ? I do not know a lot about Rust and only did start coding in Rust.

[Nercury]

Wow, looking back on it 5 years later, it looks like this discussion was where it ended. I am sorry.

Last thing I remember about trying to re-implement 100% compatible twig engine was that I was deep in PHP C code looking at what happens when you apply + operator to a string and a float, and realizing that I would need to re-implement the same nonsense in rust, which would make the engine compatible, but did I really want that? That's when it ended for me :)

Nowadays, if I were to write a template engine, I would create a new one and use the twig only for inspiration.

[williamdes]

Do you mean that porting twig is no more an idea now ? Does it mean that you should archive your project ?

[Nercury]

I am no longer interested, that's all. But this is not my project (my own version is under my account, and you are welcome to copy chunks of code if you want).

[williamdes]

Maybe you would be interested to transfer your repository to @code-lts so someone can pick the maintenance some day ?