"Html" Lexer

[kbond]

With the popularity of Twig Component HTML Syntax I wanted to discuss the possibility of adding an official html lexer in Twig core. What we're doing in twig components is sort of a hack: converting <twig:... to {% .... An official lexer would remove this conversion process.

We've already added the block tag (<twig:block name="title">) in twig components. We can envision users asking for more official Twig tags as html tags.

I know this would not be an easy or quick task and there would be a lot to consider for each tag:

<twig:if ...?>
    ...
<twig:else>
    ...
<twig:endif>

Alternatively, if it's decided to not include in core, what about opening the lexer up for extension?

[weaverryan]

To clarify, there are 2 possible levels to this proposal

1) We add this HTML lexing to core and allow for things like <twig:if> as Kevin said.

2) OR... we add something (in core? Or via an extension in Lexer) to support the <twig:Foo> syntax properly in a way where TwigComponents could leverage it. So maybe we don't go as far as <twig:if>, but <twig:Foo> would work, and then perhaps there is then some way to "plug into" Twig's core to be notified when this syntax is found (so that the TwigComponents code could take over).

[faizanakram99]

yes please, the current syntax of twig is sometimes unreadable, for example consider this statement here, it isn't really clear when an attribute starts and when it ends, with html syntax it will be ':disabled="fieldType != 'select'"' (much easier to read)

{% if form.choices is defined %}
    {{ form_row(form.choices, {'attr': {':disabled': 'fieldType != "select"'}, 'row_attr': {'x-show': 'fieldType == "select"'} }) }}
{% endif %}

[ericmorand]

@kbond so you propose to add the tag <twig:> to the list of official tags of the langage? Or did I misunderstood?

[kbond]

I think Ryan's second proposal (2) is probably the best starting point. I believe this would simplify twig components code and, assuming we get support/adoption, perhaps sell the idea of html syntax in core (Ryan's first proposal).

[ericmorand]

What would the first proposal ("html syntax in core") mean outside of the Symfony framework? How would such syntax translate when used this way:

index.php

<?php

require './vendor/autoload.php';

$loader = new \Twig\Loader\ArrayLoader([
    'index.twig' => '<twig:if foo == true>Here<twig:else>There<twig:endif>'
]);
$env = new \Twig\Environment($loader, []);

$env->display('index.twig', []);

php index.php

[kbond]

IIRC, the way we have html-syntax working in twig components is: translate the html syntax to standard twig syntax, then compile down to php as normal.

I'm quite out of my depth here but I believe html-syntax in core could remove the "translate the html syntax to standard twig syntax" step. So essentially, your example would just work (assuming this new lexer is enabled by default).

[fabpot]

Having this "alternative" syntax for Twig itself is probably a no-go. That would be a very different templating engine. I'm not even sure how the "translation" would work as each tag has its own way of parsing what comes after the tag (so even if we can do the work for core tags, what about tags provided by third parties? Would it mean that they would need to provide the parsing logic for both the current syntax and the "HTML" one?).

Noob question: what are the reasons for having this syntax new for Symfony UX vs using plain Twig?

[stof]

AFAIK, this was introduced in symfony/ux to replace the previous syntax based on a {% twig_component %} tag that was very verbose, except that a tag could have exposed a much less verbose syntax instead (as a token parser can support almost anything inside the tag)

[kbond]

Having this "alternative" syntax for Twig itself is probably a no-go.

Yep, totally understand this. This proposal should now just focus on Ryan's point (2) above. @smnandre I think has a better understanding on what's needed than me.

Noob question: what are the reasons for having this syntax new for Symfony UX vs using plain Twig?

It started as basically an experiment to match other frontend frameworks and Blade Components. Ryan and I wanted to use this type of syntax ourselves and based on feedback now that it has been in Twig Components for a while now, we aren't alone.

[stof]

@kbond the issue is that it is basically creating a different templating language (except that you make it compile to a Twig template instead of being standalone), which is why you have to hack Twig for that.

[WebMamba]

And what do you think about adding an extension point into the twig Lexer? So Twig stays as it is, but you let the ability for others to create a new templating system from Twig?

what are the reasons for having this syntax new for Symfony UX vs using plain Twig?

Symfony UX is used a lot by front-end developers and they are already familiar with this syntax since it's inspired by Vue and other front-end frameworks. And even for back-end developers the syntax make things more easy to read when working with component.

[smnandre]

On one hand, I’m fully aware and critical of our current implementation for all the reasons discussed in #4215 .

On the other hand, I completely support what @WebMamba mentioned: we frequently receive requests for more "HTML-like syntax," and there’s genuine enthusiasm for this approach.

That said, despite a few attempts on my end to make both syntaxes coexist within the compiler/parser/lexer, I’m struggling to find a clean, secure, and effective way to implement this.

[smnandre]

This proposal should now just focus on https://github.com/twigphp/Twig/issues/3867#issuecomment-1660790814 above

Not sure to understand, as we already offer .

@kbond Could you illustrate with some usage example that would be supported ?

[smnandre]

One thing I've spent some time exploring.. Twig allows the definition of custom "tags" using the Lexer (refer to the Twig documentation here).

$twig->setLexer(new \Twig\Lexer($env, [
    'tag_variable' => ['{[', ']}'],
]));

{# you can now echo with `{[` instead of `{{` #}

{[ include(...) ]}

However, this approach would require some kind of context-aware chain of Lexers, which would involve writing a lot of code.

But even then, it wouldn't provide the same HTML-like syntax that we currently use. Moreover, these settings would be exclusive, meaning there's absolutely no way to mix them within the same template.

I'm not even touching on the potential nightmare this could cause in terms of documentation, support, and so on.

So, in the end, I'm afraid @stof is right:

that it is basically creating a different templating language

This doesn't mean we can't offer features and improve developer experience (DX); we can certainly find other ways!

But as of now, I really don't see how HTML syntax could be integrated into Twig’s core. I'd be glad to be proven wrong, though.

[stof]

@smnandre changing the lexer configuration is a no-go in the Symfony ecosystem (as that's also basically creating a different templating language). There's a good reason why symfony/twig-bundle does not provide configuration settings for those Twig settings: using non-default value would make the project incompatible with all the ecosystem of bundles shipping Twig templates (including symfony/webprofiler-bundle and symfony/twig-bridge). And the same is true for any other ecosystem using Twig (for instance, if Drupal decides to use Twig, they would have to settle on a single configuration for the lexer, and it would be simpler to settle on the default one). I would even suggest to deprecate those settings.

But as of now, I really don't see how HTML syntax could be integrated into Twig’s core. I'd be glad to be proven wrong, though.

I don't see it either. As said previously, this HTML syntax is a different templating language by itself.

[kbond]

Not sure to understand, as we already offer .

Really just meant to pivot this issue to only discuss adding extension points to twig itself that could make what we're doing in twig components easier.

[smnandre]

One thing that could help is some access of private code from the Lexer.. to replicate parts of the process without having to duplicate the code.

But if we want to add new features to the HTML syntax, i'm pretty sure we need to add some HTML parsing with custom tags (php 8.4 could help here) ..

Also i'm willing to experiment using a Loader as "extension" point.

[fabpot]

I'm going to close this one as I don't see anything we can do here.