Idea: power lambdas

[jgonggrijp]

2023-10-31 update: the newest version of my proposal is here.

2022-10-21 update: the following excerpt from https://github.com/mustache/spec/issues/138#issuecomment-1286225095 summarizes how I currently envision power lambdas.

The way I currently think of it, lambdas receive a second argument which somehow (i.e., in an implementation-defined way) makes the following things possible:

• retrieve any context frame;
• retrieve a list of all keys visible in the current (full) context;
• resolve any key against the current (full) context;
• identify the context frame in which a key is resolved;
• render a template of choice against the current context.

Whereby lambdas must not modify the pre-existing contents of the context, and implementations are welcome to actively prevent this if the programming language can enforce it. However, lambdas can (already) push a new frame on the stack, which still has the net effect of changing what's available in the context.

2022-10-16 update: added playground links and savestates.

This is a feature I've meant to implement for a while, and which I was planning to propose to the spec after implementing it. However, it seems relevant to the discussion about dynamic partials that is currently taking place with @anomal00us, @gasche and @bobthecow in #134 and #54, so I decided to describe the idea now already.

The context name resolution algorithm described in the current interpolation, section and inverted section specs states that if any name in a dotted path happens to refer to a function (method), that function is called in order to obtain the next frame from which to resolve the next name in the path (or the final result). In case of a section or inverted section, the function is also passed the contents of the section as a string argument, if it is capable of accepting an argument. Illustration with JavaScript lambda code:

{{#key.lambda.otherKey}}Hello {{name}}{{/key.lambda.otherKey}}

{
    key: {
        lambda(section) {
            // section will be the string 'Hello {{name}}'
            return {
                otherKey: { name: 'John' }
            };
        }
    }
}

Hello John

Try the above example in the playground by pasting the following code:

{"data":{"text":"{\n    key: {\n        lambda(section) {\n            // section will be the string 'Hello {{name}}'\n            return {\n                otherKey: { name: 'John' }\n            };\n        }\n    }\n}"},"templates":[{"name":"","text":"{{#key.lambda.otherKey}}Hello {{name}}{{/key.lambda.otherKey}}"}]}

The lambdas spec adds to this that if the last name in the path is a function and it returns a string, then that string is rendered as a template before finally being substituted for the interpolation or section tag.

{{#key.lambda}}Hello {{name}}{{/key.lambda}}

{
    key: {
        lambda(section) {
            // section will be the string 'Hello {{name}}'
            return 'I changed my mind about the template, {{name}}';
        }
    },
    name: 'John'
}

I changed my mind about the template, John

Try the above example in the playground by pasting the following code:

{"data":{"text":"{\n    key: {\n        lambda(section) {\n            // section will be the string 'Hello {{name}}'\n            return 'I changed my mind about the template, {{name}}';\n        }\n    },\n    name: 'John'\n}\n"},"templates":[{"name":"","text":"{{#key.lambda}}Hello {{name}}{{/key.lambda}}"}]}

I would like to suggest another optional extension spec on top of this, which I'll dub "power lambdas" for now. It amounts to passing an additional argument to functions that can accept it, containing a (read-only) view of the full context stack. I'll illustrate a couple of advanced features that could be implemented using power lambdas below. For now I'll assume that the context stack is passed as the most conventional sequence type of the programming language, with the top of the context stack as the last element. It could also be left to the implementation whether the stack frames are ordered top or bottom first.

Access to properties of lower context stack frames that have been hidden by higher frames:

{{#nested1}}
    {{#nested2}}
        {{name}}
        {{parentScope.name}}
        {{rootScope.name}}
    {{/nested2}}
{{/nested1}}

{
    parentScope(section, stack = section) {
        return stack[stack.length - 2];
    },
    rootScope(section, stack = section) {
        return stack[0];
    },
    name: 'John',
    nested1: {
        name: 'Lizzy',
        nested2: {
            name: 'Deborah'
        }
    }
}

        Deborah
        Lizzy
        John

Hash iteration by key-value pairs like in Handlebars's #each helper:

{{#nested}}{{#byKey}}
    The {{key}} is called {{value}}.
{{/byKey}}{{/nested}}

{
    byKey(section, stack) {
        var context = stack[stack.length - 1];
        var pairs = [];
        for (var key in context) {
            pairs.push({key, value: context[key]});
        }
        return pairs;
    },
    nested: {
        'police officer': 'Jenny',
        'doctor': 'Hildegard',
        'hairdresser': 'Tim'
    }
}

    The police officer is called Jenny.
    The doctor is called Hildegard.
    The hairdresser is called Tim.

Dynamic partials:

{{!template.mustache}}
{{animal}} goes:
    {{#dereference.>}}animal{{/dereference.>}}

{{!cow.mustache}}
Moo!

{{!dog.mustache}}
Woof!

{
    dereference(section, stack) {
        // This example presumes that there is a way to render a template with a prepared
        // template stack instead of just a plain view that will become the root of a new
        // context stack. Offering such an interface will be attractive for implementations
        // that support power lambdas.
        var name = renderMustache('{{' + section + '}}', stack);
        return {
            '>': function() { return '{{>' + name + '}}'; }
            // I'm oversimplifying here, you could do this for all sigils without code
            // duplication.
        };
    },
    animal: 'cow'
}

cow goes:
    Moo!

[bobthecow]

First, I don't think addition additional optional params to your lambda implementation goes against the spec, as long as the base case is compatible. You should just be able to implement this.

Second, specifying exactly what happens inside lambdas, down to the params, feels like it goes against the principles of Mustache. The Mustache way, to me, kind of defers on these sorts of matters to what feels "right" for each implementation's language, if that makes sense?

So mustache.php implements lambdas with a second "lambda helper" parameter, which exposes a method for rendering a template with the current stack as context. I went back and forth with a really similar idea to this proposal, and on what felt like too "logic-y", and came to the conclusion that exposing the context stack just felt wrong. I've considered adding a context lookup method to the helper, though, as that feels cleaner and more Mustachey.

[jgonggrijp]

Thanks for sharing your insight! Yes, makes sense.

Just to be clear, I wasn't worried that my implementation might be incompatible with the spec because of the additional argument. I just wanted to have an implementation to back my proposal.

I can see why enforcing a particular form for the second argument would be too restrictive for implementers, so I agree a hypothetical spec shouldn't do that. On the other hand, two of the three examples I gave above are only possible if the lambda can access each individual stack frame. So in order for an implementation to qualify as supporting power lambdas, I believe it should offer some way to do that. I also like your idea of offering a context lookup helper, and your existing support for rendering against the current stack.

[agentgt]

TLDR; Maybe block parameters could be used as stack resolved parameters?

I'll throw my 2 cents on this one given some experience based on the Java world of mustache and handlebars.

Most people with no experience of mustache expect or want section lambdas (currently 1 arity) to work very similar to handlebars helpers or just normal function calls.

e.g.

{{#some arg1 arg2 ...}}some body{/some}}

The above syntax is way too much logic and obviously breaks regular mustache but in general what they want is to some part of the context (aka {{.}} and not parents) plus some optional parameters that are usually string as well as finally possibly some body contained within the lambda.

In the mustache spec world and mustache.java you only get the unprocessed body of the lambda. In mustache.java this is Function<String,String>. On the other hand JMustache passes the immediate context object as well as the body which I find far more useful.

The classic use case for this is date formatting and or escaping or possibly creating nontrivial href links. In JMustache you would do it like:

{{#someDate}}
{{dateFormat}}someFormatStringThatCannotBeDynamicInterpolation{{/dateFormat}}
{{/someDate}}

The someDate object gets passed to the dateFormat lambda (as well the format string). However because of how hideous laborious notation wise the above is as well as nonstandard we just ended up building richer models (ie generating the formatted date apriori) or using handlebars.

Consequently in my current implementation of mustache (which has many challenges because it is statically checked and thus the templates/partials/lambdas cannot be generating dynamic templating) I have been toying with the following:

{{dateFormat}}{{$date}}.{{/date}{{$fmt}}"iso"{{/fmt}} rest of the body {{/dateFormat}}

Basically block tags inside lambdas do dotted named based resolution unless they are in quotes. The above is also currently technically legal mustache as allowed block tag location is nebulous. The other option is to allow the block parameters to expand eagerly and leave the body untouched with the expectation that the lambda just deals with a string name value pair.

See in my situation unlike @bobthecow I am generating static code w/o special context stack types. The goal of my mustache implementation is to be typesafe. Thus I don't want to just pass the whole stack as some generic stack object but instead resolve to an actual type.

To make this clear my implementation would at compile time convert the above to a method call:

// the template has to be analyzed at compile time so the return object is the context
@TemplateLambda("The time is {{.}}")  
public String dateFormat(LocalDateTime date, String fmt) {
// return formatted string
}

[jgonggrijp]

Hey @agentgt, nice suggestion. I invited you to join here in #131, but you already found it.

I have been thinking about something very similar to what you're describing, but would suggest different syntax:

{{#dateFormat}}{{#date}}*someDate{{/date}{{#fmt}}iso{{/fmt}} rest of the body {{/dateFormat}}

(Note that the {{dateformat}}{{/dateformat}} outer pair in your original example is not allowed by current syntax, but maybe that was just a typo on your end.)

The trick here is that dateFormat is a lambda that returns a new object with date and fmt methods. That object goes on the top of the stack, so those methods are invoked as lambdas, which allows you to apply the arguments in a curried fashion. The date method would store the value and return nothing, while the fmt method would take the previously stored value and return the formatted result, so it ends up interpolated inside the section.

The *dateFormat names a key in the context, analogous to the recently added dynamic names spec, while iso is taken as a literal string. The interpretation of the *dateFormat notation would be the responsibility of the date method, but that can only be successfully implemented if lambdas can somehow access the context stack. Otherwise, all ingredients are already supported by the current spec.

The above is also currently technically legal mustache as allowed block tag location is nebulous.

Just to clarify, blocks are allowed everywhere. The meaning depends on whether the block is directly nested with a parent tag pair or not. If not directly nested within a parent pair, it marks a part of the template that can be configured from the outside. If it is directly nested within a parent pair, it serves to provide that outside configuration (but is itself also still configurable).

Your challenge of implementing safely typed templates is very interesting. I would love to see your code.

[agentgt]

I have been playing around with jstachio's lambdas which for a variety of reasons has to be a little different than javascript or other dynamic implementations and found some issues with lists.

How jstachio does lambdas is that the contents of the lambda block are used as a template and the lambda returns an object that is effectively push on to the context stack (current context for the inputted template). I think this is vastly superior to the current spec of the lambda returning a template as there are many more things that can go wrong.

If the above idea were put into the spec an easy backward compatibility is if the lambda returns an object and takes two parameters like:

function(input, currentContext){ 
   return {"item": currentContext}; 
}

input is the contents of the lambda block and currentContext is the top of the context stack.

Here is how that looks in JStachio:

    /*
     * { "items" : [ 1, 2, 3], "lambda" : function(input, item) {...no analog in current spec..} }
     */
    static final String template = """
            {{#items}}{{#lambda}}{{item}} is {{stripe}}{{/lambda}}{{/items}}""";

    @JStache(template=template)
    public record Items(List<Integer> items) {
        @JStacheLambda
        public LambdaModel lambda(Integer item) {
            return new LambdaModel(item, item % 2 == 0 ? "even" : "odd");
        }
    }
    /*
     * In jstachio if you return an object it is then pushed on the stack
     * and the contents of the of the lambda block are used as a sort of inline partial.
     * 
     * This is in large part because we cannot handle dynamic templates and also
     * because I think it is the correct usage is as the caller knows how it wants to render
     * things and avoids the whole delimeter nightmare.
     */
    public record LambdaModel(Integer item, String stripe) {}

    @Test
    public void testName() throws Exception {
        String expected = "5 is odd";
        String actual = JStachio.render(new Items(List.of(5)));
        assertEquals(expected, actual);
    }

Anyway this works great with one giant problem.... there is now way to get the entire list. That is there is no spec way to pass the entire list.

The above is important because one of the biggest complaints when using mustache is the need to know index information while in a list. Such as first, last, and numeric index albeit that last one is less common. Of course you could redecorate the model and depending on implementation you might even be able to use lambdas as an accumulator to figure out first and maybe index but in general it is a huge annoyance.

So ideally a lambda would provide you with that list information but in my implementation it is not possible as the lambda cannot get the entire list. I will probably add fake fields like:

{{#items.this}}{{#lambda}}{{/lambda}}{{/items.this}} 

This would force the list to be treated as an object instead of a list.

So my question is if you do do: *items using the asterisk notation for context lookup will it get the entire list?

[jgonggrijp]

Hey @agentgt, nice to discuss these matters with you. Indeed, JStachio is quite different from the JavaScript-oriented implementations I have seen. Interesting to see how JStachio.render operates.

As a general remark: please feel absolutely free to not implement parts of the spec that you feel are infeasible or unfitting for your situation, and also to implement behavior that is not (yet) in the spec. @bobthecow put it elegantly a few posts up:

(...) The Mustache way, to me, kind of defers on these sorts of matters to what feels "right" for each implementation's language, if that makes sense?

Some detail remarks:

How jstachio does lambdas is that the contents of the lambda block are used as a template and the lambda returns an object that is effectively push on to the context stack (current context for the inputted template).

Most people do not realize that there are two variants of the specification for lambdas: the obligatory variant in the interpolation and sections modules, and the optional variant in the ~lambdas module. The first, obligatory variant prescribes exactly what you are describing here. So in that sense, JStachio already behaves as specified!

The dynamic template replacement thing is what is specified in the optional ~lambdas module. Wontache implements both behaviors, as illustrated in the following example.

Data

{
    /* lambda pushes a new stack frame */
    today: function(ignored) {
        return {
            day: 20,
            month: 'October',
        };
    },
    /* lambda returns a new template */
    emphasize: function(section) {
        return '<em>' + section + '</em>';
    },
}

Template

It is {{#emphasize}}{{#today}}{{month}} {{day}}{{/today}}{{/emphasize}}.

Output

It is <em>October 20</em>.

Copy the following code to the playground in order to try the above example, as well as a variant with today nested outside instead of inside emphasize (which gives the same result):

{"data":{"text":"{\n    /* lambda pushes a new stack frame */\n    today: function(ignored) {\n        return {\n            day: 20,\n            month: 'October',\n        };\n    },\n    /* lambda returns a new template */\n    emphasize: function(section) {\n        return '<em>' + section + '</em>';\n    },\n}"},"templates":[{"name":"today-inside","text":"It is {{#emphasize}}{{#today}}{{month}} {{day}}{{/today}}{{/emphasize}}."},{"name":"today-outside","text":"It is {{#today}}{{#emphasize}}{{month}} {{day}}{{/emphasize}}{{/today}}."}]}

So these interpretations of lambdas are not necessarily in conflict with each other. However, when they are, you should probably prioritize the stack-pushing variant over the template-replacing variant. You are doing the right thing!

Anyway this works great with one giant problem.... there is now way to get the entire list. That is there is no spec way to pass the entire list.

Right, that's why I'm proposing to enable this, somehow.

(...)

So ideally a lambda would provide you with that list information but in my implementation it is not possible as the lambda cannot get the entire list.

To be clear, is it not possible in your implementation because of the spec, or would it still not be possible if power lambdas made it into the spec?

So my question is if you do do: *items using the asterisk notation for context lookup will it get the entire list?

Could you illustrate this question with a template in which you demonstrate this usage of the *items notation?

[agentgt]

Sorry for the late reply on this @jgonggrijp . I must have missed your questions.

First some update.

JStachio is pretty much 1.0.0 ready. Its documentation is here: https://jstach.io/jstachio/

Here is a glorified example:

 @JStache(template = """
     {{#people}}
     {{message}} {{name}}! You are {{#ageInfo}}{{age}}{{/ageInfo}} years old!
     {{#-last}}
     That is all for now!
     {{/-last}}
     {{/people}}
     """)
 public record HelloWorld(String message, List<Person> people) implements AgeLambdaSupport {}

 public record Person(String name, LocalDate birthday) {}

 public record AgeInfo(long age, String date) {}

 public interface AgeLambdaSupport {
   @JStacheLambda
   default AgeInfo ageInfo(
       Person person) {
     long age = ChronoUnit.YEARS.between(person.birthday(), LocalDate.now());
     String date = person.birthday().format(DateTimeFormatter.ISO_DATE);
     return new AgeInfo(age, date);
   }
 }

The above might be fairly confusing if you are not familiar with Java but the things prefixed with @ are annotations and essentially immutable and static but are visible to compiler plugins. JStachio is essentially a compiler plugin.

So hopefully you can now see how in JStachio unlike almost all other implementations templates cannot be dynamically created. Consequently the lambda either returns an object which will be pushed on to the stack and then the lambda section body is used kind of like a partial or it returns a raw string. Either way it does have access to the head of the context stack (as well as the raw body).

A current real world limitation that JStachio does have is that it cannot render the section body and then decorate like say wrapping it with <em> tags. I'm exploring some ways to deal with that issue.

To be clear, is it not possible in your implementation because of the spec, or would it still not be possible if power lambdas made it into the spec?

It might be accessible if you get access to the entire stack. e.g. Currently my implementation does not allow lambdas to have whole stack like parameters but I plan on adding that. However it still does not fix the list problem.

Using the above Persons list and let us assume I have a lambda that accepts a list of persons how do I reference the list?

{{#people}}
{{#peopleLambda}}
{{/peopleLambda}}
{{/people}}

Two major problems arise:

1. peopleLambda cannot access people (list of person) in any way if the list is empty.
2. Assuming the list (people) is not empty the top of the stack is now a single person for peopleLambda and peopleLambda will be executed however many people there are.

So in short I cannot see how a lambda can get access to a list of anything without being executed multiple times or not at all. That is probably OK for the most part ignoring the most desired feature request of mustache: index information (which JStachio does support).

Could you illustrate this question with a template in which you demonstrate this usage of the *items notation?

I can't recall how I got that notation but I think it was the idea that lambdas could take parameters in prefix style (ie normal function call) instead of the current stack postfix style.

e.g. postfix currently in regular mustache and I think in your current power lambda in wontache: {{#parameter}}{{#lambda}}body{{/lambda}}{{/parameter}}

what was talked about I think at some point: {{#lambda}}{{*parameter}}{{/lambda}}. I think that is where the asterisk came from.

That is mustache and current lambdas are "cake eat" where as most function call in languages (ignoring fortran) are "eat cake". With the prefix notation we can access the whole list:

{{#peopleLambda}}{{*people}}{{/peopleLambda}}

But is probably not in the spirt of Mustache.

Anyway I'm fine with your current recommendations for power lambda. I think really what is needed is a standard way to access index information and if power lambdas can do that across the board that would be nice but I think it is easier of the spec add some "virtual" keys as optional similar to handlebars (e.g. @index).

[jgonggrijp]

Right, so I was suggesting you don't do

{{#peopleLambda}}{{*people}}{{/peopleLambda}}

but (subtle difference)

{{#peopleLambda}}*people{{/peopleLambda}}

which would enable peopleLambda to access the entire list (people), even if it is empty. I imagine the implementation of peopleLambda to go something like this in JavaScript:

function peopleLambda(sectionText, contextMagic) {
    var contextKey = sectionText.slice(1); // substring 'people' in the above example
    var theList = contextMagic.lookup(contextKey);
    return 'The length of this list is ' + theList.length + '.';
}

In my suggested version, the *people part of the template has no special meaning to the Mustache engine; the lambda itself is doing the interpretation. I'm using the literal part of the template text to pass arguments to the lambda, with an asterisk signifying that the argument should be looked up in the context. For a lambda that should always look up its argument in the contenxt, you could also leave off the asterisk and simplify the implementation further:

{{#peopleLambda}}people{{/peopleLambda}}

function peopleLambda(contextKey, contextMagic) {
    var theList = contextMagic.lookup(contextKey);
    return 'The length of this list is ' + theList.length + '.';
}

However, I imagine either approach might be difficult of even impossible for JStachio. If the notation {{*people}} improves matters for you, then by all means go for it.

I have seen notations like {{*x}} and {{>*x}} elsewhere, but with a different meaning. Especially in #54. The similarity is still close enough that you might be able to justify its reuse here, though.

Edit to add: my compliments on your readable JavaDoc!

[agentgt]

{{#peopleLambda}}*people{{/peopleLambda}}

Ok now I remember. I think this actually would be possible with special hints that the section body has parameters and the parameters have to be in a certain format. The compiler then parses that format. Otherwise runtime reflection and parser could be used but that is against the spirit of jstachio. Let us ignore runtime and dynamically constructing templates for now because obviously if we are given the stack and dynamic renderer we can almost do anything.

I believe that is why I was proposing reusing block notation (or another new sigil) in the lambda sections as parameters (which reminds me I need to test parents calling lambdas with blocks as I'm not sure what now happens in JStachio). The idea being a new notation to pass a template (section body) and parameters by using block notation.

Using the person example let us assume our lambda function looks like (using typescript-like to show types):

function peopleLambda(people: Person[]): DecoratedPerson[] {
}

interface DecoratedPerson {
  person: Person
  index: number;
}

If we choose to support multiple context parameters:

{{#peopleLambda}}
{{$people}}*people{{/people}}
{{#.}}
<em>{{index}} {{person.name}}
{{/.}}
{{/peopleLambda}}

If you can only pick one binding on the stack I suppose the notation could be:

{{#peopleLambda}}
{{*people}}
{{#.}}
<em>{{index}} {{person.name}}
{{/.}}
{{/peopleLambda}}

The idea being the parameters are not rendered and removed from the section body but are looked up in the context stack and passed to the lambda. That is the compiler does the context lookup instead of the lambda author.

That is I rather avoid passing special things like "contextStack" and "renderer" to the lambda.

Anyway the above is rather complicated so I have avoided implementing it. Also because there is a spec format for passing parameters around: handlebars.

I will play around more to see what works and doesn't and let you know my findings. Thanks for the response!

[jgonggrijp]

Yes, given your design choices and preferences, something like {{#peopleLambda people=*people}} or {{#peopleLambda people=people}} (as opposed to {{#peopleLambda people="people"}} which would be the literal counterpart in the latter case), as in Handlebars, might actually be your safest bet. It is less in the spirit of Mustache to add more structure to the internals of a tag, but it is easier to avoid conflicts with present and future semantics of tags in other Mustache implementations in that way.

If you do pass the parameter through a separate tag, I recommend inventing something new over reusing block tags. Block tags inside (lambda) sections already have a meaning, as illustrated in this playground savestate:

{"data":{"text":"{\n    name: 'Luke',\n    lambda() {\n        return {\n            name: 'darling',\n        };\n    },\n}"},"templates":[{"name":"main","text":"{{<parent}}\n    {{$block}}sprinklers{{/block}}\n{{/parent}}"},{"name":"parent","text":"{{#lambda}}\nUse the {{$block}}force{{/block}}, {{name}}.\n{{/lambda}}"}]}

data

{
    name: 'Luke',
    lambda() {
        return {
            name: 'darling',
        };
    },
}

main.mustache

{{<parent}}
    {{$block}}sprinklers{{/block}}
{{/parent}}

parent.mustache

{{#lambda}}
Use the {{$block}}force{{/block}}, {{name}}.
{{/lambda}}

[agentgt]

Apparently it works in my implementation (blocks in lambda calls). So abusing block notation would be a bad idea indeed!

     public record Person(String name) {
     }

    @JStache(template = """
            {{<parent}}
                {{$block}}sprinklers{{/block}}
            {{/parent}}""")
    @JStachePartials(@JStachePartial(name = "parent", template = """
            {{#lambda}}
            Use the {{$block}}force{{/block}}, {{name}}.
            {{/lambda}}
            """))
    public record PersonPage(String name) {

        @JStacheLambda
        public Person lambda(Object ignore) {
            return new Person("darling");
        }
    }

    @Test
    public void testParentLambda() throws Exception {
        String expected = """
                Use the sprinklers, darling.
                """;
        String actual = JStachio.render(new PersonPage("Luke"));

        assertEquals(expected, actual);
    }

[jgonggrijp]

Of course, the power lambdas idea is not really new. I just encountered a similar description by @pvande from 2012: https://github.com/mustache/spec/issues/41#issuecomment-7337582

[jgonggrijp]

Also related: #19.

[determin1st]

the crucial part of lambdas is the argument specification. the management of the stack, its values, rendering, substitution etc can be done by passing the mustache object or the interface to it as the first parameter to any lambda. the second argument should be reserved, string type for hints or additional arguments, the third should bear first section content (when applicable, not for inverted or variable)

function lambda(object $m, string $arg, string $sect);

the primary application of lambda is template preparation, not final rendering, because calling lambda is not a performant operation

hey {{name}}, you have a {:#ansi_color red:}message{:/ansi_color:}!

[jgonggrijp]

@determin1st Why not put the section contents first, then the object/magic interface, and finally, optionally, the argument string? That would be more backwards-compatible.

Your second code snippet seems to have both {{-}} and {:-:} delimiters, what's up with that?

[agentgt]

the management of the stack, its values, rendering, substitution etc can be done by passing the mustache object or the interface to it as the first parameter to any lambda.

And what is that object? That seems far more likely something that the spec should not even mention since that stack or Mustache object will vary greatly.

My personal opinion is that the first parameter should be the top of the stack. The second one the contents of the section which would be message. Or vice versa.

The return value is where it gets more complicated and I think the optional part of the spec because of backward compat gets this wrong:

An optional part of the specification states that if the final key in the name is a lambda that returns a string, then that string replaces the content of the section. It will be rendered using the same delimiters (see Set Delimiter below) as the original section content. In this way you can implement filters or caching.

Because of the above in the ancient original spec is why you need some sort of special mustache object being passed but I think it is wrong or at least confusing to many and current spec has it correct where a template is returned. Still I think that is mostly wrong but due to limitations of various languages it is much better than getting a omnipotent "mustache" object and using it to render.

The return value for most cases of a lambda should be an object that gets put/pushed on the stack and the contents of the section should be used as a template and not some dynamically constructed template (e.g. the case of returning the String).

The above works for most use cases with the exception of wanting to surround something. For that I had to do special things because JStachio does not allow you to dynamically create templates. You can see how it is done with templates that can refer to the section as a partial like {{>@section}} https://jstach.io/doc/jstachio/current/apidocs/io.jstach.jstache/io/jstach/jstache/JStacheLambda.html

Regardless I rather basic lambdas be not coupled to a specific Mustache implementation or have tons of parameters. I think Handlebars fills that niche much better.

[agentgt]

BTW this ( I changed your delimiters) :

{{#ansi_color red}}message{{/ansi_color}}

Could be done like:

{{#ansi_color}}{{$red}}message{{/red}}{{/ansi_color}}

Your lambda just parses the section code looking for blocks. Yes it is a hack but you could write this code in any mustache implementation. Actually don't even need to use blocks and just create your own DSL.

For example in most programming languages you format messages with a mini DSL (e.g. %s).

You could just do {{#ansi_color}}%red(message){{/ansi_color}} or whatever syntax you like or your host language supports.

[agentgt]

Oh and I forgot to mention if you support returning objects (which again I think is the correct way to do lambdas):

{{#ansi_color}}{{#color.red}}message{{/color.red}}{{/ansi_color}}

This assumes ansi_color returns an object/lambda of color and color has a lambda on it called red. You see where I'm going with this.

That is in JStachio you can do this:

{{#ansi_color.color.red}}message{{/ansi_color.color.red}}

Lambda returning lambdas (which kind of reminds of some old OOP languages that actually do not have multiple parameters and just pass a single message).

This doesn't entirely map to all languages easily but I bet most could do it.

I think @jgonggrijp mentioned something like the above already.

EDIT You could just even remove the color lambda/object.

{{#ansi_color.red}}message{{/ansi_color.red}}

[determin1st]

Why not put the section contents first, then the object/magic interface, and finally, optionally, the argument string? That would be more backwards-compatible.

putting section as the last argument, allows to omit its placement at invokation and make callback definition more flexible:

function lambda(object $m, string $arg, string|null $sect=null);
function lambda(object $m, string $arg, string|false $sect=false);
function lambda(object $m, string $arg);

all those valid at least for php. having it first, prescribes to put certain value as a sign of "section is not applicable". i find this marker useful:

[# some data
  'some_lambda' = (function(object $m, string $arg, ?string $sect=null) {
    if ($sect === null)
    {
      # okay, this is a variable or inverted block
    }
    else
    {
      throw ErrorEx::fatal(
        'this is not an intended use of some_lambda(), '.
        'please rewrite your template!'
      );
    }
  }),
]

Your second code snippet seems to have both {{-}} and {:-:} delimiters, what's up with that?

those are {: :} delimiters for preparation. people call it precompilation step. step 1 => step 2 scheme. in my opinion, you should have nice/readable templates for modification, which often makes them unusable at the final render. so i designate preparation for removing all the indentation, render some lambdas, special symbols etc..

[determin1st]

The above works for most use cases with the exception of wanting to surround something. For that I had to do special things because JStachio does not allow you to dynamically create templates. You can see how it is done with templates that can refer to the section as a partial like {{>@section}} https://jstach.io/doc/jstachio/current/apidocs/io.jstach.jstache/io/jstach/jstache/JStacheLambda.html

Regardless I rather basic lambdas be not coupled to a specific Mustache implementation or have tons of parameters. I think Handlebars fills that niche much better.

okay, i checked how you use context and those magic with iteration but it's still an object argument to the lambda callback, so i see no contradiction. its not obligatory to pass the main instance or whatever, it's an object that helps lambda do stuff related to template rendering

BTW this ( I changed your delimiters) :

{{#ansi_color red}}message{{/ansi_color}}

Could be done like:

{{#ansi_color}}{{$red}}message{{/red}}{{/ansi_color}}

Your lambda just parses the section code looking for blocks. Yes it is a hack but you could write this code in any mustache implementation. Actually don't even need to use blocks and just create your own DSL.

For example in most programming languages you format messages with a mini DSL (e.g. %s).

You could just do {{#ansi_color}}%red(message){{/ansi_color}} or whatever syntax you like or your host language supports.

no, that wont work in my implmentation, because i have added OR and SWITCH sections to the syntax:

{{^has_failed argument}}
  FALSY
{{|}}
  TRUTHY
{{/has_failed}}
{{^as_well_as argument}}
  FALSY HERE
{{/as_well_as}}

will receive NO sections, because it's a negation, it only selects one. i call it FALSY OR block. this one (TRUTHY OR block):

{{#return_array 1,2,3}}
  {{#array}}{{.}}{{/array}}
{{|}}
  FALSY
{{/return_array}}

will receive the first section. this one (SWITCH OR block):

{{#select tag}}
  NOT FOUND AND TRUTHY
{{|one}}
  1
{{|}}
  NOT FOUND AND FALSY
{{|two}}
  2
{{|0}}
  0
{{/select}}
{{^also_have argument}}
  NOT FOUND AND FALSY
{{|Joe}}
  Im Joe
{{|Sam}}
 Im Sam
{{/also_have}}

will not receive any section

[determin1st]

Oh and I forgot to mention if you support returning objects (which again I think is the correct way to do lambdas):

{{#ansi_color}}{{#color.red}}message{{/color.red}}{{/ansi_color}}

This assumes ansi_color returns an object/lambda of color and color has a lambda on it called red. You see where I'm going with this.

That is in JStachio you can do this:

{{#ansi_color.color.red}}message{{/ansi_color.color.red}}

Lambda returning lambdas (which kind of reminds of some old OOP languages that actually do not have multiple parameters and just pass a single message).

This doesn't entirely map to all languages easily but I bet most could do it.

I think @jgonggrijp mentioned something like the above already.

EDIT You could just even remove the color lambda/object.

{{#ansi_color.red}}message{{/ansi_color.red}}

anything that lambda returns for TRUTHY OR block (except the string which does substitution) renders as usual. but here you're starting to mix dot notation with arguments and this will potentially turn into a big big mess. those lambdas that appear in the path shouldnt receive anything, imo, they are traversion helpers, nothing more. it's better to remove, not mix, like, for example remove closing tag name requirement:

{{#some.path.to.heavens.ansi_color red}} text {{/}}

[agentgt]

@determin1st

Do you have a link to your implementation?

That might give me more context. Thanks!

[determin1st]

@agentgt its not fixed yet, ill drop working tests by the end of the week. today i came to the next thought that section parameter must be an integer - identifier - because template sections are already parsed, identifier could be resolved to the text easily:

function lambda(object $m, string $arg, int $sectionId=-1);

getting text

$text = $m->giveMeTextFrom($sectionId);# in lambda

+speedup

$m->render($text) === $m->renderAlreadyCompiled($sectionId);

[agentgt]

@determin1st I'm not sure I follow including section ids. It seems like your adding a lot more extensions that I'm just not aware of yet so perhaps my comments are not geared correctly for your implementation.

However this:

{{#ansi_color.red}}message{{/ansi_color.red}}

is actually syntactically equivalent to this:

{{#ansi_color}}
{{#red}}
message
{{/red}}
{{/ansi_color}}

Let us ignore return values for a second and do this with JSON-like model which is what Mustache is based on:

{ 
"ansi_color" : {
   "red" : function(section) { return "<span class='red'>" + section + "</span>" }
   // other colors
   }
} 

You can test this on the wontache playground:

{"data":{"text":"{ \n\"message\" : \"hello\",\n\"ansi_color\" : {\n   \"red\" : function(section) { return \"<span class='red'>\" + section + \"</span>\" }\n   // other colors\n   }\n} "},"templates":[{"name":"example1","text":"{{#ansi_color.red}}{{message}}{{/ansi_color.red}}"}]}

https://jgonggrijp.gitlab.io/wontache/playground.html (thanks again @jgonggrijp for the great tool!)

anything that lambda returns for TRUTHY OR block (except the string which does substitution) renders as usual. but here you're starting to mix dot notation with arguments and this will potentially turn into a big big mess. those lambdas that appear in the path shouldnt receive anything, imo, they are traversion helpers, nothing more.

I disagree. Dotted paths open sections. Sections can be lambdas. The issue is going deeper than two levels without constructing a giant object tree or may not be possible for other lambdas. ansi_color is taking a symbol and not something with high cardinality like an integer.... speaking of which that is a major fucking with problem with parameters. Deciding what is literal and not is why Handlebars is far more complicated.

Now the reason I'm saying return an object that gets pushed onto the stack over a template is that returning strings is confusing. As an end user my question is the String going to be escaped and interpolated or is it treated as raw and how do I make that distinction and how do I do this across languages?

Furthermore pushing something on the stack feels far more natural and would allow you to chain lambdas easier but I understand why the optional default return the template approach was chosen. I'm also biased as this currently works in JStachio.

Furthermore while

{{#ansi_color.color.red}}message{{/ansi_color.color.red}}

May not work currently... if you do allow modification of the stack with a lambda the follow could very easily be allowed (and many currently do allow pushing stuff on the stack):

{{#ansi}}
{{#color}}
{{#red}}{{/red}}
{{/color}}
{{#opacity}}100{{/opacity}} {{! let us pretend this is possible with ansi }}
{{#build}}message{{/build}}
{{/ansi}}

You just have the lambdas take the special "mustache" parameter and shove an object that has color and opacity on to the stack as a lambdas and return empty templates. What the above is in many languages is the "builder" pattern and it is used when you do not want a metric ton of parameters. In our case its because we don't want to have to make Mustache into Handlebars.

EDIT I guess my point is if you allow modifying the stack than you can easily simulate argument passing while syntactically not breaking existing mustache implementations

Speaking of which @jgonggrijp was dynamic name lambdas ever talked about? That is if dynamic name lambdas would be allowed {{#*ansi_color}}...{{/*ansi_color}} which I believe would again be more of reason why returning an object that gets pushed seems more natural (because the lambda in that case would be returning a template name (or something that would resolve a template name for JStachio's case).

[agentgt]

Perhaps this wording or slant might make it more apparent of what I'm trying to say: Ignoring inverted sections all sections are actually lambdas the way I'm describing them.

Ignoring actual implementations at an abstract academic level every time you do

{{#section}}{{stuff}}{{/section}}

Whatever section is gets pushed on to the stack and the contents in between are used as a template AND you can do this recursively (I mean it is a stack after all)!

Not only that for you to open or get section you need the top of the stack.

Do you see why I think just returning a string is not right? Because regular sections actually do more than just return the section contents. They push something onto the stack.

You see how that is unifying and makes it easier to explain especially over adding new syntax for arguments (which to be frank I found most people can just get by with decorating their model some more)?

[jgonggrijp]

@determin1st

(...) today i came to the next thought that section parameter must be an integer - identifier - because template sections are already parsed, identifier could be resolved to the text easily:

function lambda(object $m, string $arg, int $sectionId=-1);

This is getting very much into implementation details. Not all Mustache engines might identify their sections with integers. In fact, a section could be entirely implicit in generated code.

getting text

$text = $m->giveMeTextFrom($sectionId);# in lambda

Getting the text is already a solved problem with the current situation, since it is passed as an argument. Moreover, this interface opens the option for a lamda to retrieve a different section. How is the lambda supposed to know which id to pass in that case? Should lambdas even be that powerful in the first place?

I'm not saying lambdas should never render alternative template text, but if they do, I feel that the programmer should actually know the template text, rather than dynamically retrieving it like a black box that is hidden behind an integer id.

+speedup

$m->render($text) === $m->renderAlreadyCompiled($sectionId);

You can achieve the same speedup without the section id; just go $m->renderCurrentSection();.

@agentgt

(...)

However this:

{{#ansi_color.red}}message{{/ansi_color.red}}

is actually syntactically equivalent to this:

{{#ansi_color}}
{{#red}}
message
{{/red}}
{{/ansi_color}}

Let us ignore return values for a second (...)

Hold on. These forms are sometimes equivalent, but not in the general case. Consider the following view:

{ 
    "message" : "hello",
    "red" : function(section) { return "<span class='red'>" + section + "</span>" },
    "ansi_color" : {}
}

With the dotted notation, this will give no output, while the nested notation will give the intended output. Playground savestate:

{"data":{"text":"{ \n    \"message\" : \"hello\",\n    \"red\" : function(section) { return \"<span class='red'>\" + section + \"</span>\" },\n    \"ansi_color\" : {}\n}"},"templates":[{"name":"dotted","text":"{{#ansi_color.red}}{{message}}{{/ansi_color.red}}"},{"text":"{{#ansi_color}}{{#red}}{{message}}{{/red}}{{/ansi_color}}","name":"nested"}]}

https://jgonggrijp.gitlab.io/wontache/playground.html (thanks again @jgonggrijp for the great tool!)

:heart:

(...)

Furthermore while

{{#ansi_color.color.red}}message{{/ansi_color.color.red}}

May not work currently...

I think it could, actually.

if you do allow modification of the stack with a lambda the follow could very easily be allowed (and many currently do allow pushing stuff on the stack):

{{#ansi}}
{{#color}}
{{#red}}{{/red}}
{{/color}}
{{#opacity}}100{{/opacity}} {{! let us pretend this is possible with ansi }}
{{#build}}message{{/build}}
{{/ansi}}

Yes, also. You can make this work even without power lamdas, just using the current spec. This playground savestate illustrates one way to do it (you could simplify and/or sophisticate it):

{"data":{"text":"{ \n    message: 'hello',\n    ansi: function(sectionText) {\n        return {\n            color: {\n                red: function(sectionText) {\n                    this.selected = 'red';\n                    return '';\n                }\n                // green, blue, and so forth\n            },\n            opacity: function(sectionText) {\n                this._opacity = sectionText;\n                return '';\n            },\n            build: function(sectionText) {\n                return (\n                    '<ansi color=\"' +\n                    this.color.selected +\n                    '\" opacity=\"' +\n                    this._opacity +\n                    '\">{{' +\n                    sectionText +\n                    '}}</ansi>'\n                );\n            }\n        };\n    }\n}"},"templates":[{"name":"stateful","text":"{{#ansi}}\n{{#color}}\n{{#red}}{{/red}}\n{{/color}}\n{{#opacity}}100{{/opacity}} {{! let us pretend this is possible with ansi }}\n{{#build}}message{{/build}}\n{{/ansi}}"}]}

EDIT I guess my point is if you allow modifying the stack than you can easily simulate argument passing while syntactically not breaking existing mustache implementations

Agreed.

Speaking of which @jgonggrijp was dynamic name lambdas ever talked about?

Not in depth, but it was mentioned as a possibility in #134 or one of its preceding discussions. The optional spec for dynamic names does not forbid dynamic sections, by the way.

That is if dynamic name lambdas would be allowed {{#*ansi_color}}...{{/*ansi_color}} which I believe would again be more of reason why returning an object that gets pushed seems more natural (because the lambda in that case would be returning a template name (or something that would resolve a template name for JStachio's case).

Not quite. The way dynamic names are currently specified, the name is resolved dynamically before the tag does its particular job. So if you were to have this view/data,

{
    message: 'hello',
    ansi_color: function() { return 'red'; },
    red: {
        message: 'surprise!'
    }
}

and the external template red.mustache,

<ansi color="red">{{message}}</ansi>

then this template.mustache

{{#*ansi_color}}{{message}}{{/*ansi_color}}

should theoretically render

surprise!

which I suspect is not what you had in mind. Put differently, the dynamic names spec simply describes how to perform an indirect key lookup in the context; it does not state that the indirectly found key is always interpreted as a template name. What to do with a key still depends on the sigil, so if the sigil is #, we are going to do another context lookup, not a template lookup.

You can select a partial or parent template dynamically in a lambda (already), but the order of control is different. red.mustache can stay the same as above, but in the template.mustache, you just do a straight, non-dynamic partial call (can be either variable or section):

{{#ansi_color}}{{/ansi_color}}

and then the lambda generates template code that selects the external template dynamically using a partial or parent tag:

{ 
    message: 'hello',
    selected_color: 'red',
    ansi_color: function() {
        return '{{>' + this.selected_color + '}}';
    }
}

savestate:

{"data":{"text":"{ \n    message: 'hello',\n    selected_color: 'red',\n    ansi_color: function() {\n        return '{{>' + this.selected_color + '}}';\n    }\n}"},"templates":[{"name":"red","text":"<ansi color=\"red\">{{message}}</ansi>"},{"name":"template","text":"{{! could also just be a variable interpolation\n    resolving to lambda }}\n{{#ansi_color}}{{/ansi_color}}"}]}

(...)

Do you see why I think just returning a string is not right? Because regular sections actually do more than just return the section contents. They push something onto the stack.

Pushing something onto the stack is both more powerful and less powerful than returning a new template string at the same time. It is more powerful, because it lets you change the context and inject additional lambdas. It is less powerful, because it does not allow you to radically change the structure of the template. Essentially, power lambdas would be needed in order to let us have both at the same time.

[agentgt]

Hold on. These forms are sometimes equivalent, but not in the general case. Consider the following view:

I agree I just didn't want to go too much into the differences of how once you enter dotted path it is stricter but thank you for pointing it out.

Pushing something onto the stack is both more powerful and less powerful than returning a new template string at the same time. It is more powerful, because it lets you change the context and inject additional lambdas. It is less powerful, because it does not allow you to radically change the structure of the template. Essentially, power lambdas would be needed in order to let us have both at the same time.

What I gathered that lambdas (again at a very abstract level here) can do other than plain sections:

1. is read the contents of the section
2. create a new template

My assumption which I want check with you is that most use "The create new template part" (other than the obvious of just leaving it untouched) is to wrap or repeat a section template.

Are there other use cases that I might be missing?

In my implementation you cannot dynamically create templates (which I think is a valuable feature but I can see why in a javascript environment not so).

So I deal with this by allowing one to reference the section as a partial.

That is if you wanted to wrap ansi_color you do:

{
    ansi_color: function(stack) {
        // some magic that puts selected color on stack:
        stack.color = 'red'
        return '<span class="{{color}}">{{> @section }}</span>';
    }
}

That is section is a magic partial (you can also do parent partials). I think this is way more powerful and declarative but I just cannot figure how to portray that in the spec without introducing some magic name.

I suppose it doesn't matter that much with Javascript because if you can dynamically bind templates you could do the above and just add a template called @section dynamically but that seems error prone.

I thought about doing something like:

{{> * }}

That is make the asterisk without a name resolve to the section body.

Probably not a good idea but just wanted to throw around what I have been mulling on.

[determin1st]

@agentgt

{{#ansi}}
{{#color}}
{{#red}}{{/red}}
{{/color}}
{{#opacity}}100{{/opacity}} {{! let us pretend this is possible with ansi }}
{{#build}}message{{/build}}
{{/ansi}}

.....above is in many languages is the "builder" pattern....

okay, i understand, but again i prefer to reduce excessive mental weight which is involved by doing such a templating, not find a smart path to workaround into result. block sections are ALREADY PARSED when lambda callback is invoked, so i see the usage of section renderers as the first class scenario, not re-re-reparsing it, which will also drop performance. that's why

   falsy: {{^func arguments}} section {{/func}}
  truthy: {{#func arguments}} section {{/func}}
variable: {{func arguments}}

i see arguments as a nice and necessary extension to lambdas

EDIT I guess my point is if you allow modifying the stack than you can easily simulate argument passing while syntactically not breaking existing mustache implementations

i dont remember i forbid something. actually my current approach is to shot ownself's leg.

You see how that is unifying and makes it easier to explain especially over adding new syntax for arguments (which to be frank I found most people can just get by with decorating their model some more)?

pushing values to stack is working already, i dont understand what you mean. what author should do is testing of all possible types.. i did some for inverted lambda block (which looks like {{^lambda argument}}..{{|}}..{{/lambda}}):

LAMBDA FALSY OR
#1: path not found      ok (first section rendered)
#2: not a function      ok (nothing is rendered)
#3: boolean false       ok (first section rendered)
#4: boolean true        ok (second section rendered)
#5: empty string        ok (first section rendered)
#6: non-empty string    ok (second section rendered)
#7: integer zero        ok (first section rendered)
#8: integer non-zero    ok (second section rendered)
#8: array empty         ok (first section rendered)
#9: array non-empty     ok (second section rendered)
#10: object             ok (second section rendered)

currently second section (OR) is not in the spec, so it means nothing would render (empty string).

[agentgt]

i see arguments as a nice and necessary extension to lambdas

I really didn't do a good job explaining why

{{#func arg1 arg2...}}

Is complicated so I will try.

First how do you plan on literals and what literals do you plan on supporting? Or is there no literal support?

That is I assume arg1 and arg2 are resolved from the stack? Can you pass more than one argument (I assume so given the plural)?

Then it begs the question if you can do dot path on arguments.

See lots of complexity.

However if you maybe limit to one argument and no literals that might reduce the complexity but I don't think that gets you much than what is already possible other than selecting a different part of the stack.

pushing values to stack is working already, i dont understand what you mean. what author should do is testing of all possible types.. i did some for inverted lambda block (which looks like {{^lambda argument}}..{{|}}..{{/lambda}}):

Apologies but I'm not sure I follow either. I'm not entirely sure I understand the pipe syntax. I think once you get your implementation ready for review I can probably provide better comments. Sorry about that and confusing you.

[determin1st]

@jgonggrijp

This is getting very much into implementation details. Not all Mustache engines might identify their sections with integers. In fact, a section could be entirely implicit in generated code.

identifying section renderer (function) by its text is non-performant task - you have to have a hash over that text plus you have to resolve hash to renderer. identifier (or an index) is the most optimal way of getting both text and renderer. which.. actually you dont need!!! if your implementation does arguments i mentioned. so yes, a little encouragement for good implementations. dont want lambdas - skip, want lambdas - do good

Getting the text is already a solved problem with the current situation, since it is passed as an argument. Moreover, this interface opens the option for a lamda to retrieve a different section. How is the lambda supposed to know which id to pass in that case? Should lambdas even be that powerful in the first place?

text is not needed. because you have renderer. you only need text when you want to build something over it, parse it, re-parse it, re-reparse it.. not good.

You can achieve the same speedup without the section id; just go $m->renderCurrentSection();.

nope, with this syntax, you have to drop identifiers into context while template's been rendered. well, maybe it's good for debugging, but not for performance, especially for scripting engines.

[determin1st]

First how do you plan on literals and what literals do you plan on supporting? Or is there no literal support?

anything reasonable, alphanumeric recommended

That is I assume arg1 and arg2 are resolved from the stack? Can you pass more than one argument (I assume so given the plural)?

nope, they're packed into a single string you have to parse yourself inside the lambda, define your own format.

Then it begs the question if you can do dot path on arguments.

See lots of complexity.

not really. use it as a key to complex data, not data itself

However if you maybe limit to one argument and no literals that might reduce the complexity but I don't think that gets you much than what is already possible other than selecting a different part of the stack.

function lambda(object $m, string $arg);

see, single $arg ument

Apologies but I'm not sure I follow either. I'm not entirely sure I understand the pipe syntax. I think once you get your implementation ready for review I can probably provide better comments. Sorry about that and confusing you.

's not pipe symbol but | OR symbol, nice pair to / closing tag

[agentgt]

@determin1st this is just a kind suggestion... don't worry about optimization yet. Premature optimization is evil. Especially because you are also trying to add features that are not even spec'ed out yet.

I can assure you do not need to turn sections into integers or whatever to make Mustache haul ass. I say this basically as the author of the fastest Mustache implementation (well ignoring Rust handlebars).

In fact the recommendations I'm making here are so that you can precompile easier and have a lot less dynamic behavior which includes not having lambdas alter section body to new templates (which requires recompile every time) as well as having to go up and down the stack looking for arguments.

Taking the top of the stack as well as pushing something on the top is fast.

nope, they just a single string you have to parse yourself inside the lambda, define your own format.

Yeah that is not going to work for a spec. It is an implementation detail.

What I think you should have a look at is Handlebars. If you have a good portion of Mustache implemented in theory Handlebars would be a better spec to target.

I say this all kindly. I don't want to discourage you and only want to help but I'm just nervous by the comments of performance and altering mustache syntax for convenience that I think will largely confuse your users.

I also don't know what is possible in PHP these days so there is that as well :)

[determin1st]

@agentgt

I can assure you do not need to turn sections into integers or whatever to make Mustache haul ass. I say this basically as the author of the fastest Mustache implementation (well ignoring Rust handlebars).

ive got a speed test for different engines written in php and js. they loop over some json files (commonly supported spec). ill provide those later. maybe youll be able to lineup it with your implementation, or provide instructions how to do that

In fact the recommendations I'm making here are so that you can precompile easier and have a lot less dynamic behavior which includes not having lambdas alter section body to new templates (which requires recompile every time) as well as having to go up and down the stack looking for arguments.

any new template that is rendered inside lambdas shouldnt be cached. also, i think that template recursions arent necessary, as lambdas do it explicitly.

Yeah that is not going to work for a spec. It is an implementation detail.

vice versa, spec friendly. no details of the format, have your own.

What I think you should have a look at is Handlebars. If you have a good portion of Mustache implemented in theory Handlebars would be a better spec to target.

its too much logic in there, in fact ive abandoned a lot of mustache syntax. "partials" <*> i dont understand that. excessive tripple mustaches.. changing delimiters inside the template.. i see expressions in handlebars - dont need them if you have lambda arguments, right

I say this all kindly. I don't want to discourage you and only want to help but I'm just nervous by the comments of performance and altering mustache syntax for convenience that I think will largely confuse your users.

I also don't know what is possible in PHP these days so there is that as well :)

im the only user, so no worries there :]

[agentgt]

Those comments helped a lot with context. Thank you!

any new template that is rendered inside lambdas shouldnt be cached. also, i think that template recursions arent necessary, as lambdas do it explicitly.

"partials" <*> i dont understand that

Yeah this is the part that I have pretty strong opinions on and given your concern of optimization a cache miss is expensive.

I'm just trying to convince @jgonggrijp and others that while it is in the spec to return a template if we do power lambdas I want a method for people to not manually concatenate strings to create a new template.

Of course my opinions are not really because of optimization but because I think template mangling/creation at runtime (e.g. while the engine is running) is asking for problems.

We have a templating language after all. Let us use it instead of concat-ing. Given my belief that most sections -> template altering is just to wrap or decorate most of that can be accomplished with partials.

{
 "red" : function(section) { return "<span class='red'>" + section + "</span>" }
}

That is to say while you can cache that returned template as it appears it won't alter much should we really be encouraging people do that?

Thus I consider this superior:

{
 "red" : function(section) { return "<span class='red'>{{> * }}</span>"; }
}

You can replace that asterisk with whatever bikeshed name you like but it represents the section body as a template.

{
 "user_input": "bad stuff",
 "red" : function(section) { return '<span class="red">{{> * }}</span>'; }
}

And if power lambdas can somehow be distinguished from regular lambda calls then I think it should support what I'm showing above to encourage users to use Mustache to construct strings.

If power lambdas can push even a single thing on the stack than we get iteration.

{
 "user_input": "bad stuff",
 "red" : function(section, stack) { stack.list = [1,2,3]; return "{{#.}}<span class='red'>{{> * }}</span>{{/.}}" }
}

In the current way lambdas work without section body partials which is what I'm basically proposing that power lambdas have you would have to manually loop through the array constructing the template using the host language.

I feel that is very wrong and is just asking for someone to concatenate a variable and bypass escaping which is a major security concern for many and should be for Mustache.

For example

return <span class='red'>' + section + stack.user_input + '</span>'

You see user_input is not escaped.

Ignoring the pushing stuff on the stack I don't think it would be much work to put what I'm proposing into wontache. Maybe I will fork it to refresh my very rusty JS skills.

---

Sorry for EDITS.

[jgonggrijp]

@agentgt

What I gathered that lambdas (again at a very abstract level here) can do other than plain sections:

1. is read the contents of the section

2. create a new template

My assumption which I want check with you is that most use "The create new template part" (other than the obvious of just leaving it untouched) is to wrap or repeat a section template.

I honestly don't know what most use it for, but I share your impression that most usage examples tend to just wrap something around the template and return it.

Are there other use cases that I might be missing?

Well, in your previous post (before the one I'm quoting here, I mean), you mentioned using dynamic lambdas to resolve partial/parent templates. I replied that {{#* notation would be interpreted differently by the spec. Then, I demonstrated a different way in which a lambda might still by used to resolve an external template dynamically. That use case involves completely replacing the section content, rather than just wrapping it. I'll repeat the savestate here so you can review it:

{"data":{"text":"{ \n    message: 'hello',\n    selected_color: 'red',\n    ansi_color: function() {\n        return '{{>' + this.selected_color + '}}';\n    }\n}"},"templates":[{"name":"red","text":"<ansi color=\"red\">{{message}}</ansi>"},{"name":"template","text":"{{! could also just be a variable interpolation\n    resolving to lambda }}\n{{#ansi_color}}{{/ansi_color}}"}]}

In my implementation you cannot dynamically create templates (which I think is a valuable feature but I can see why in a javascript environment not so).

Of course, not including a feature can be a deliberate and perfectly valid choice.

So I deal with this by allowing one to reference the section as a partial.

That is if you wanted to wrap ansi_color you do:

{
    ansi_color: function(stack) {
        // some magic that puts selected color on stack:
        stack.color = 'red'
        return '<span class="{{color}}">{{> @section }}</span>';
    }
}

That is section is a magic partial (you can also do parent partials). I think this is way more powerful and declarative but I just cannot figure how to portray that in the spec without introducing some magic name.

Wait, how would you write that example code in Java with JStachio? While this restricts the options somewhat, it seems like you are still generating a template at runtime.

I suppose it doesn't matter that much with Javascript because if you can dynamically bind templates you could do the above and just add a template called @section dynamically but that seems error prone.

Yes, even in JavaScript, it is not trivial. I just realized that I've been meaning to inline sections in the generated code (so instead of every section being a separate function, the whole template is a single function and each section is a loop inside that function with a little bit of extra sophistication) and this is really difficult to combine with a power lambda being able to render the section ahead of time, or even just with a regular lambda being able to read the section template. I don't really have an answer to that question yet.

I thought about doing something like:

{{> * }}

That is make the asterisk without a name resolve to the section body.

Probably not a good idea but just wanted to throw around what I have been mulling on.

Strictly speaking, that notation already has a meaning: it fetches a context property with the empty string as key and then uses the value of that property as the name of an external template, which is then interpolated as a partial. I guess you could get away with that, though, since most users are probably careful enough not to have context entries with empty string keys.

@determin1st

I think I'm missing at least some of your point, so I will only comment on the parts of your posts that I'm sure I understand right now. I look forward to seeing your implementation, which will probably clarify things. My impression is that you are trying to solve a problem I'm not (fully) aware of yet (besides performance).

Getting the text is already a solved problem with the current situation, since it is passed as an argument. Moreover, this interface opens the option for a lamda to retrieve a different section. How is the lambda supposed to know which id to pass in that case? Should lambdas even be that powerful in the first place?

text is not needed. because you have renderer. you only need text when you want to build something over it, parse it, re-parse it, re-reparse it.. not good.

I understand that having a renderer is better than re-parsing the template text, but you didn't address my other point. To clarify: my other point is independent from the question whether you use a renderer or the raw template text. I think it would be problematic if a lambda could fetch any section (be it renderer or text) by numeric id. This would give lambdas too much access, in a way. Let me illustrate this with an example.

The following template has two sections, one nested inside the other:

{{#outer}}
    {{#inner}}
        {{variable}}
    {{/inner}}
{{/outer}}

Let's say the numeric id of the outer section is 1 and the numeric id of the inner section is 2. Let's also say that in the context, inner is a lambda.

Per your proposal, inner would receive three arguments: $m (the magic object), $arg (which is an empty string in this case), and $sectionId (which is 2). Now, suppose that the body of the inner lambda is this line of code:

$m->renderAlreadyCompiled($sectionId - 1);

To me, this looks like a stack overflow. I can also easily imagine many other possible ways to shoot yourself in the foot with this. Hence my question: do you really need to address the section by numeric id inside the lambda? Why not just provide access only to the current section, through a method with a fixed name?

@agentgt

@determin1st (...)

nope, they just a single string you have to parse yourself inside the lambda, define your own format.

Yeah that is not going to work for a spec. It is an implementation detail.

To be brutally honest, something like that could make it into the spec. In a way, that is exactly what happened with the section text being passed to a section lambda: "here's the raw text, parse it yourself if you want".

I'm not in favor of doing that with tag internals, just saying that such things are possible in principle. :-)

As for passing exactly one argument (which may have internal structure) to a lambda, I think pipes/filters would be a more elegant way to do that. See https://github.com/mustache/spec/issues/41#issuecomment-7312138.

@determin1st

im the only user, so no worries there :]

Ah, please don't rule out that others might want to use your engine. It sounds like it's going to be special, so even if it is just for academic interest, please do consider sharing it.

@agentgt

I'm just trying to convince @jgonggrijp and others that while it is in the spec to return a template if we do power lambdas I want a method for people to not manually concatenate strings to create a new template.

Ah, that reminds me of something else I wanted to suggest.

Assuming a situation where power lambdas receive a magic object with special facilities, you could offer a mechanism for wrapping the already rendered section in a callback. Here's a JavaScript version of it, so I don't confuse anyone by writing bogus Java code:

{
    someLambda: function(sectionText, magic) {
        magic.wrapAfterRender(function(renderResult) {
            return '<b>' + renderResult + '</b>';
        });
    }
}

Now, if the template is

{{#someLambda}}hello{{/someLambda}}

the output would be

<b>hello</b>

. This would be perfectly compatible with all the other stuff I have proposed for lambda partials. Not to say that is a requirement, though!

You have a point about the escaping, by the way. I agree with both of you that the template-as-a-string override strategy is rather crude. If we can completely remove the need for it with power lambdas, I'm all for it.

[agentgt]

Wait, how would you write that example code in Java with JStachio? While this restricts the options somewhat, it seems like you are still generating a template at runtime.

Hopefully this clarifies: https://jstach.io/jstachio/io.jstach.jstache/io/jstach/jstache/JStacheLambda.html

 @JStacheLambda(template="<span class="{{>@section}}">{{>@section}}<span>")
 public boolean isEven(int index) {
     return index % 2 == 0;
 }

Notice the return value gets put on the stack and the template is derived from the annotation.

See the issue in general with JStachio and I guess you could argue Javascript is you want to return a template AND whatever you want to push onto the stack.

Or you pass an IN/OUT parameter aka magic object where pass the template and new node.

JStachio gets to cheat on this front because Java has annotations and the returned template can and should mostly be static.

A mostly static analog to this in Javascript would be to return a template name instead of an actual template. That is in someway why I was asking about dynamic templates and I think I accidentally distracted us away from the problem.

And yes you can do this in Javascript assuming we can push stuff on the stack by pushing a template name on the stack and returning a template of {{> *. }} (notice the .) but what you can't do is have the named template get access to the section with out concat.

Because {{> * }} does not make since outside of this case I was going to use it.

I don't think it is actually that far off or surprising in meaning to be honest given we can do this if something is pushed on the stack {{> *. }}.

What I'm asking is to make the mental jump that the section body which is a String has been pushed on the stack with the key as empty string even if that isn't what really happens underneath.

Does that make sense?

e.g. after lambda has pushed or not we might have:

{
"" : 'hello'
}

hello being the section body content from your previous example.

So yes it is more than just pushing on to the stack. We are also ostensibly pushing an empty string key with the section body.

EDIT I know it doesn't entirely make since because you don't normally push key and value on the stack but singular object so this is more like a merge conceptually (implementations can decide how they want to do this).

e.g. I push some object like the following on the stack:

{
"foo": "bar"
}

My returned template sees it more like.

{
"" : "hello",
"." : { "foo" : "bar" }
}

Again in my implementation the returned template comes from an annotation but for others this could be fed to an object that is passed in or returned or whatever.

[determin1st]

Thus I consider this superior:

{
 "red" : function(section) { return "<span class='red'>{{> * }}</span>"; }
}

If power lambdas can push even a single thing on the stack...

you return to "pushing to stack" all the time, lets push it in the runtime.. the template

{{#wrapper red}}
  {{tagOpen}}
    content
  {{tagClose}}
{{/wrapper}}

the lambda

function wrapper(object $m, string $arg)
{
  $text = $m->getFirstSectionText();# dont need that
  $id = $m->getFirstSectionId()# dont need that
  return $m->renderFirstSection([# pushing to the stack here
    'tagOpen'  => '<span color="'.$arg.'">',
    'tagClose' => '</span>',
  ]);
}

im abandoning my previous statement that passing integer section identifier is the most optimal solution. i figured how it can be obtained implicitly and it's better for the api. so lambda callback should look like:

function lambda(object $m, string $a): mixed;

getting back to {: :} or other custom delimiters used in PREPARATION phase, the template

{[#ansi_color red]} content {[/ansi_color]}

the lambda

'ansi_color' => (function(object $m, string $arg) use ($ANSI) {
  return $ANSI['foreground'][$arg] . $m->getFirstSectionText() . $ANSI['reset'];
}),

here, "template as a string" isnt wrong, because there is no caching in PREPARATION.

[agentgt]

@determin1st I can't tell if it is a mistake or you implementation just does not do any escaping but tagOpen and tagClosed would be escaped in most implementations. That is the < and > etc would be replaced with HTML entities like < and >. Regardless you are still concatenating to make the template e.g. the "snipped".$arg."snipped".

To go a little off topic here so I can understand better about your implementation I group Mustache implementations like this:

1. STATIC based
2. TRANSFORM based
3. REFLECTIVE based

STATIC

• Object tree is known in advance and cannot deviate in structure or typing
• Template is know in advance and cannot be changed at runtime

The are very few in this group. JStachio being one and some Handlebars in Rust are some others. I want to make a Dart version and perhaps Typescript or Javscript using JSONSchema/JSONNet to describe the object tree.

For many languages this requires some access to the symbolic tree of the languages or a schema.

Because you know so much apriori these implementations are usually the fastest because you are basically implementing how some would manually traverse the object tree without all the checks.

TRANSFORM

Functional programming languages that do not like reflection prefer this model. These implementations provide an Object Tree type and you as the user of the library transform your model into their Mustache Object Tree model.

I imagine the OCaml and Haskell version use this model. These are often just as fast as static if the transformational step is not counted. Luckily in these languages transformations are optimized (e.g. persistent data structures etc).

Some versions of Javascript Mustache I guess could be argued to fit this model where the Object tree is pure JSON (and thus introspection is just pattern matching JSON structure).

REFLECTIVE

This is 90% of mustache implementations. Object tree structure is unknown. Templates can also be created dynamically although I would not make that a requirement of this grouping.

Most of these implementations require reflection or at best traversal of document-like format aka JSON.

I'm fairly sure yours is in the reflective category but all this PREPARATION stuff is starting to make me think yours is not. I also do not know what is possible with PHP these days.

Which category is your implementation in or do does it not fit any of those descriptions?

On another note when I mention "stack" it is confusing because there are two stacks which maybe the confusion.

There is the "stack" that is the Mustache template syntax you parse. While for most it is an AST it can be seen as a stack as well.

Then there is the "stack" that is the object tree traversal. This is mostly what I mean when I say "stack".

It is confusing because often times when you parse in STATIC or TRANSFORM you are walking both stacks at the same time apriori in what I think is your PREPARATION mode but not sure.

[jgonggrijp]

@determin1st So if I understand correctly, your implementation makes two passes over the template, one as a kind of preprocessing/macro expansion step and one for the final rendering.

What does the use ($ANSI) notation, between the function parameter list and the function body, mean in PHP?

I'm also interested in your answer to @agentgt's question about static/transform/reflective.

@agentgt I'm not 100% sure I understand your distinction between static and transform, but I suspect the original Ruby implementation might also fall in the transform category. Although it also has full runtime reflection.

To distinguish between the two types of stack, you could perhaps qualify them as "context stack" and "syntax stack".

[agentgt]

@jgonggrijp I guess imagine that grouping as more of continuum 😄

Even JStachio has some optional TRANSFORM aspects to it (e.g. use this special object type the engine knows about).

With dynamic languages pattern matching / dynamic dispatch verses reflection can get nebulous but what I mean for the TRANSFORM is that the cardinality of the dispatching is constrained. JSON is one way to constrain dispatch because there are only so many types.

EDIT

The distinction between TRANSFORM and STATIC is that the transformation to the constrained model happens automatically at compile time as opposed to requiring the user transform whatever data to that constrained model. So STATIC is basically TRANSFORM.

I admit it can get pretty confusing as various languages support so many things and "compile" can mean a lot of things.

EDIT 2

I a guess another way to put it is that the STATIC way does not have to dispatch or pattern match at all (let us ignore "properties" in some languages aka unified access).

{
"a" : ""
}

In the static model we always know that tree we get will have the "a" key as a String.

In the transform model that entire JSON is some NODE and usually implementations use sum types like an EBNF.

eg:

type NODE = STRING | BOOL | NULL | NUMBER | OBJECT | LIST | NODE

They do not know that "a" will even be there. However unlike the reflective model where it could be anything they just need to recursively pattern match the above.

With the reflective model you have to ask the type "tell me everything about you". In the transform model you ask "which of these types are you"?

[determin1st]

@agentgt

.....does not do any escaping but tagOpen and tagClosed would be escaped in most implementations. That is the < and > etc would be replaced with HTML entities like < and >. Regardless you are still concatenating to make the template e.g. the "snipped".$arg."snipped".

yes, that's a feature i want to deviate a bit. make it opt in instead of opt out, meaning that {{&userInput}} is whats going to be escaped. im not sure how to do that, maybe an option to the instance that inverts default behavior. it is based on HTML and proposition that templates are typed by users. developer knows exactly what "stage" currently is and whether it s a user input or template composition.

To go a little off topic here so I can understand better about your implementation I group Mustache implementations like this:

1. STATIC based
2. TRANSFORM based
3. REFLECTIVE based

oke, let me explain how i see stage things in the modern template EVALuator. HTML-only days are over, so it must be generic view.

stage 1: preparation

templates should be somehow beautiful and easy to read and to modify - for example, check how beautiful are json spec files - long one-liners with zero indentation, yaml files, in this case, arent the source of truth, because they bypass stage 1 implicitly. json is what we need, big lengthy json objects with infinite one-liners.. (put some irony here).

i think you agree that template storage represents a hasmap - name=>template content, it can be filename (still a name) or json object key, also a name bound with content - string type. php arrays are better than JSON files, aesthetically. i bet any language does better job in template sourcing - better representation for a developer.

single template, one may wish to see is:

  {:BR:}{:TAB:}
  Hey, {{name}}, you are {:#ansi_color red:}{{age}}{:/ansi_color:} years old!

this tiny indentation brings an issue to non-HTML sources. "remove indentation around standalone blocks" feature in mustache doesnt fully do OUTdentation. the issue is obscure in HTML, because HTML usually eats any whitespace in the view. same as in yaml files, you dont see it. i think that explicit outdentation is required. mine has two methods for this stage:

function outdent(string $template): string;
function prepare(string $template, mixed $data=null, string $customDelimiters=''): string

the process may look like

foreach ($templateStorage as $name => &$template) {
  $template = $m->prepare($m->outdent($template), $initialData, '{: :}');
}

initial data may look like

[
  'BR' => "\n",
  'TAB' => "\t",
  'ansi_color' => (function(..){..}),
]

now template is explicitly indented for the target source (console/terminal that supports ansi sequences)

stage 2: compilation or cache preSET

this is completely optional. the only benefit of this stage is significant performance jump which is related to addressing mechanism - instead of copying potentially large strings and calculating hashes over them, you use identifiers. this not required for experiments or few template renderings. mine has a method:

function set(string $template): int

the process is similar

foreach ($templateStorage as $name => &$template) {
  $template = $m->set($template);
}

now templates are integers. getting text back is possible.

stage 3: rendering

is also optional. experiments and playgrounds dont need methods that use cache. few template renders are fine with prepare(). otherwise, yours is some kind of an app that renders periodically and able to benefit from caching and memoization. mine has:

function render(string $template, mixed $data=null): string;
function get(int $template, mixed $data=null): string;

rendering may look like

$result = $m->render($templateStorage[$name], $data);
# or when precached
$result = $m->get($templateStorage[$name], $data);

regarding to lambdas, i suppose that most of them will live a short life at the stage 1 and wont be invoked periodically as projected for stage 3. there is no need for constant ansi wraps or other special/control character insertion in a thoughtful template.

@jgonggrijp

So if I understand correctly, your implementation makes two passes over the template, one as a kind of preprocessing/macro expansion step and one for the final rendering.

yes, stage 1 and 2 can be treated as a single step.

What does the use ($ANSI) notation, between the function parameter list and the function body, mean in PHP?

it only passes external variable for use. in JS that is achieved by closure scopes, phps have that use () clause.

@agentgt

type NODE = STRING | BOOL | NULL | NUMBER | OBJECT | LIST | NODE

They do not know that "a" will even be there. However unlike the reflective model where it could be anything they just need to recursively pattern match the above.

With the reflective model you have to ask the type "tell me everything about you". In the transform model you ask "which of these types are you"?

i see it as optimization properties. i did some type memoization which is based on template then data, not data then template. im not determined about actual structure yet. for example {{lambda argument}} is clearly a function - because it has an argument - no need to typecheck here, so ill select fragility over implicit stability

also consider cache fragility:

$template = $m->outdent('

  {{#list}}
    {{.}}
  {{/list}}

');
echo $m->render($template, ['list'=>[1,2,3,4,5]]);# prints 12345, template is cached
echo $m->render($template, ['list'=>['one','two','three']]);# FATAL! - expecting integers in the array
echo $m->prepare($template, ['list'=>['one','two','three']]);# prints onetwothree, template cache is ignored

[jgonggrijp]

@determin1st I think I'm starting to understand your aim, and I find it very interesting. Please do keep us updated.

Just one nitpick:

(...)

single template, one may wish to see is:

  {:BR:}{:TAB:}
  Hey, {{name}}, you are {:#ansi_color red:}{{age}}{:/ansi_color:} years old!

this tiny indentation brings an issue to non-HTML sources. "remove indentation around standalone blocks" feature in mustache doesnt fully do OUTdentation. (...)

Regarding the standalone block indentation, I suppose you mean my proposal in #130/#131. That one isn't in the spec yet, because it is somewhat controversial, so it is not technically a Mustache feature yet. Although I did implement it in Wontache and you can try it in the playground.

That being said, it does do full outdentation, if the block has intrinsic indentation. Here is an example:

{{! template.mustache }}
* a
* b
* {{<parent}}{{$block}}
                {{! notice this big indent }}
                c{{/block}}{{/parent}}
* d
* e

{{! parent.mustache }}
{{$block}}x{{/block}}

{{! output }}
* a
* b
* c
* d
* e

Playground savestate:

{"data":{"text":"{}"},"templates":[{"name":"template","text":"* a\n* b\n* {{<parent}}{{$block}}\n                {{! notice this big indent }}\n                c{{/block}}{{/parent}}\n* d\n* e"},{"name":"parent","text":"{{$block}}x{{/block}}"}]}

[determin1st]

@jgonggrijp

Regarding the standalone block indentation, I suppose you mean my proposal in #130/#131. That one isn't in the spec yet, because it is somewhat controversial, so it is not technically a Mustache feature yet. Although I did implement it in Wontache and you can try it in the playground.

That being said, it does do full outdentation, if the block has intrinsic indentation. Here is an example:

{{! template.mustache }}
* a
* b
* {{<parent}}{{$block}}
                {{! notice this big indent }}
                c{{/block}}{{/parent}}
* d
* e

{{! parent.mustache }}
{{$block}}x{{/block}}

{{! output }}
* a
* b
* c
* d
* e

i have outdent() function which is 3 lines long and i have 67 lines of code that says "clear standalone blocks" and which does nothing after outdent is applied, its a dead code.

that's why, i dont understand why it moves into the engine. indentation and outdentation should be explicit, because when additional parsing applies automatically on prepared content (for example, from those json files) it does nothing useful, except wasting CPU cycles. so my bet is for explicit/manual preparation.

let's better get rid of the name requirement in the block's closing tag:

array: [{{@block argument}}
  {{.}}{{^_last}},{{/}}
{{/}}]

also you use examples with strange syntax {{$block}} isnt default mustache.

[jgonggrijp]

@determin1st

i have outdent() function which is 3 lines long and i have 67 lines of code that says "clear standalone blocks" and which does nothing after outdent is applied, its a dead code.

that's why, i dont understand why it moves into the engine.

One reason that comes to mind is that one template may interpolate the result of another template using {{>partial}} or {{<parent}}{{/parent}} tags. When this happens, the user of your engine does not get the opportunity to intervene with a $m->outdent call. How do you plan to handle that, anyway?

Also, reindentation of partials was already in the spec before I proposed those rules for blocks. I just generalized the existing rules for partials to parents and blocks.

indentation and outdentation should be explicit, because when additional parsing applies automatically on prepared content (for example, from those json files) it does nothing useful, except wasting CPU cycles. so my bet is for explicit/manual preparation.

The way I proposed it, block outdentation only needs to happen once, when the block is first compiled. Likewise, indentation only needs to happen when a block or parent is expanded, and this can be cached because the indentation happens before rendering. I don't see why CPU cycles would be necessarily wasted.

let's better get rid of the name requirement in the block's closing tag:

array: [{{@block argument}}
  {{.}}{{^_last}},{{/}}
{{/}}]

I find that code less readable, but let's not get sidetracked. How is the omission of closing tag contents related to block reindentation?

also you use examples with strange syntax {{$block}} isnt default mustache.

Ah, now I'm not sure whether we are talking about the same type of "block". {{$block}} is official syntax, although it is optional. It made it into the spec relatively recently, so you are forgiven for not recognizing it; see #125. Since yesterday, the mustache(5) manpage on the official Mustache website is also finally current again, so you can a more informal explanation of it over there.

The @ sigil is not in the spec, anywhere. You must have gotten that from an engine that implemented it as an unofficial extension. What does it mean?

[determin1st]

@jgonggrijp

One reason that comes to mind is that one template may interpolate the result of another template using {{>partial}} or {{<parent}}{{/parent}} tags. When this happens, the user of your engine does not get the opportunity to intervene with a $m->outdent call. How do you plan to handle that, anyway?

im not a big fan of inheritance/partials/parents, i simply dont understand that syntax, but outside inclusions can be handled with lambdas:

{{include header.mustache}}
{{include body.mustache}}

and code

function include(object $m, string $a) {
  return $m->prepare($m->outdent(file_get_contents(TEMPLATE_DIR.$a)));
}

Also, reindentation of partials was already in the spec before I proposed those rules for blocks. I just generalized the existing rules for partials to parents and blocks.

if people would start with their intent or use case, much of the complexity be gone.

The way I proposed it, block outdentation only needs to happen once, when the block is first compiled. Likewise, indentation only needs to happen when a block or parent is expanded, and this can be cached because the indentation happens before rendering. I don't see why CPU cycles would be necessarily wasted.

well, in this case, because its hash/checksum calculations over potentially long strings

I find that code less readable, but let's not get sidetracked. How is the omission of closing tag contents related to block reindentation?

it can be put into the engine instead of in/outdentation, which can be extracted as additional functionality or opt-in functionality (which applies automatically as you wish but with dedent=true or reindent=1|2|4|8|16 option to the engine)

also you use examples with strange syntax {{$block}} isnt default mustache.

Ah, now I'm not sure whether we are talking about the same type of "block". {{$block}} is official syntax, although it is optional. It made it into the spec relatively recently, so you are forgiven for not recognizing it; see #125. Since yesterday, the mustache(5) manpage on the official Mustache website is also finally current again, so you can a more informal explanation of it over there.

i see.. a block begins with $ it says.. these wordings overload my meanings. how do you call a section that have multiple sections inside? i call blocks anything that terminates with /

The @ sigil is not in the spec, anywhere. You must have gotten that from an engine that implemented it as an unofficial extension. What does it mean?

@ blocks are for explicit iteration/traversal (either index=>value or key=>value) with helper _ variables (probably not best pairing, what's better?).

[jgonggrijp]

@determin1st

im not a big fan of inheritance/partials/parents, i simply dont understand that syntax, but outside inclusions can be handled with lambdas:

{{include header.mustache}}
{{include body.mustache}}

and code

function include(object $m, string $a) {
  return $m->prepare($m->outdent(file_get_contents(TEMPLATE_DIR.$a)));
}

That code example explains a lot, thanks! Now I also understand why you need arguments.

For what it's worth, the spec's {{>header}} is roughly equivalent to your {{include header.mustache}}. {{<header}}{{/header}} would be equivalent as well. {{<}}{{/}} exists as separate syntax from {{>}} because it can have internal structure, which enables it to override parts of the external template. That's what the {{$}}{{/}} blocks are for. I guess your implementation could use the argument string for that purpose as well.

To illustrate, this pair of templates:

{{! header.mustache }}
# {{title}}

{{! fullDocument.mustache }}
{{>header}}

{{>body}}

would be equivalent to this single template:

# {{title}}

{{>body}}

and this pair of templates:

{{! body.mustache }}
{{#paragraphs}}
{{content}}
{{$toplink}}[Back to top](#top){{/toplink}}

{{/paragraphs}}

{{! fullDocument.mustache }}
{{>header}}

{{<body}}
    {{$toplink}}We need urgent funds! [Donate now](/donate.php){{/toplink}}
{{/body}}

would be equivalent to this single template:

{{>header}}

{{#paragraphs}}
{{content}}
We need urgent funds! [Donate now](/donate.php)

{{/paragraphs}}

Here is a playground savestate to play with:

{"data":{"text":"{ \n    title: 'Silly page',\n    paragraphs: [\n        {content: 'The quick fox jumps over the lazy dog.'},\n        {content: 'To be or not to be, that is the question.'}\n    ]\n}"},"templates":[{"name":"header","text":"# {{title}}\n"},{"name":"body","text":"{{#paragraphs}}\n{{content}}\n{{$toplink}}[Back to top](#top){{/toplink}}\n\n{{/paragraphs}}\n"},{"name":"fullDocument","text":"{{>header}}\n\n{{<body}}\n    {{$toplink}}We need urgent funds! [Donate now](/donate.php){{/toplink}}\n{{/body}}"}]}

if people would start with their intent or use case, much of the complexity be gone.

Please clarify: in what way could complexity have been avoided here?

The way I proposed it, block outdentation only needs to happen once, when the block is first compiled. Likewise, indentation only needs to happen when a block or parent is expanded, and this can be cached because the indentation happens before rendering. I don't see why CPU cycles would be necessarily wasted.

well, in this case, because its hash/checksum calculations over potentially long strings

Templates and blocks can be addressed by their names, which are generally short strings.

(...)

i see.. a block begins with $ it says.. these wordings overload my meanings. how do you call a section that have multiple sections inside?

I would call the inner sections "nested sections" or "subsections". I don't consider it special for a section to contain other sections, so I never thought of a way to refer to that specifically. Maybe "outer section" or "supersection".

i call blocks anything that terminates with /

Makes sense. I don't really know of a concise, unambiguous way to describe that, to be honest. When an opening and a closing tag belong together, like {{#section}}{{/section}} or {{$block}}{{/block}}, I tend to call that a "tag pair", but in that case it isn't obvious whether that includes only the tags or also the contents between those tags.

By itself, {{/}} is called "End Section Tag", even if you use it to close a block or a parent. I mean, that is how it is called in the respective specs. But this is not at all helpful in finding a word for the whole thing.

Do you have any suggestion? Maybe we can introduce additional jargon somewhere in the spec.

@ blocks are for explicit iteration/traversal (either index=>value or key=>value) with helper _ variables (probably not best pairing, what's better?).

I cannot think of anything better. You might be able to implement that usecase with lambdas as well, though. In fact, that is one of the things I'm hoping to achieve with power lambdas.

[determin1st]

@jgonggrijp

I cannot think of anything better. You might be able to implement that usecase with lambdas as well, though. In fact, that is one of the things I'm hoping to achieve with power lambdas.

heres what i tested

  numbers: 
  {{@numbers}}
    {{.}}
    {{^_last}},{{/}}
  {{/}}

  {{BR}}

  numbers: 
  {{#iterate numbers}}
    {{.}}
    {{^last}},{{/}}
  {{/}}

both print 1,2,3,4,5 (without the last comma). the second says iterate numbers, its a lambda. template itself is fed with this data:

$data = [
  'numbers' => [1,2,3,4,5],
  'BR' => "\n",
];
$data['iterate'] = (function ($m,$a) use ($data) {
  # prepare
  $list = $data[$a];
  $last = count($list) - 1;
  $help = ['last' => ($last === 0)];
  # add helper
  $m->helper($help);
  # compose
  for ($x='',$i=0; $i <= $last; ++$i)
  {
    $help['last'] = $i === $last;
    $x .= $m->render($list[$i]);
  }
  return $x;
});

so, the $m->helper() pushes helper to the stack. the difference from the main instance is that it automatically popped out from the stack, after the lambda completes. still, i see the place for @ syntax as it simplifies iteration, but not fully determined of the naming of _ assisted/helper values prefix.

I would call the inner sections "nested sections" or "subsections". I don't consider it special for a section to contain other sections, so I never thought of a way to refer to that specifically. Maybe "outer section" or "supersection".

i call blocks anything that terminates with /

Makes sense. I don't really know of a concise, unambiguous way to describe that, to be honest. When an opening and a closing tag belong together, like {{#section}}{{/section}} or {{$block}}{{/block}}, I tend to call that a "tag pair", but in that case it isn't obvious whether that includes only the tags or also the contents between those tags.

By itself, {{/}} is called "End Section Tag", even if you use it to close a block or a parent. I mean, that is how it is called in the respective specs. But this is not at all helpful in finding a word for the whole thing.

Do you have any suggestion? Maybe we can introduce additional jargon somewhere in the spec.

right, here is what i think about blocks and sections

according to that, block equals to section, because only one section is possible in the block in the current specification.

[jgonggrijp]

@determin1st

I cannot think of anything better. You might be able to implement that usecase with lambdas as well, though. In fact, that is one of the things I'm hoping to achieve with power lambdas.

heres what i tested

  numbers: 
  {{@numbers}}
    {{.}}
    {{^_last}},{{/}}
  {{/}}

  {{BR}}

  numbers: 
  {{#iterate numbers}}
    {{.}}
    {{^last}},{{/}}
  {{/}}

both print 1,2,3,4,5 (without the last comma). the second says iterate numbers, its a lambda. (...)

Great example code. With power lambdas as I envision them, it could look and work nearly identical to your second example, with the subtle but important difference that it uses a dotted name instead of an argument and you need to unpack the value explicitly:

{{#iterate.numbers}}
    {{value}}
    {{^last}},{{/last}}
{{/iterate.numbers}}

I wrote above that I was hoping to achieve this with power lambdas, but for this particular use case, I actually think that filters would be even better (see https://github.com/mustache/spec/issues/41#issuecomment-7312138). A filter is like a lambda that directly receives another value from the context as an argument, so the implementation is greatly simplified (JavaScript because I'm not fluent enough in PHP):

{{# numbers | iterate }}
    {{value}}
    {{^last}},{{/last}}
{{/ numbers | iterate }}

{
    numbers: [1, 2, 3, 4, 5],
    iterate: function(array) {
        var wrapped = [];
        for (var i=0, l=array.length; i < l; ++i) {
            var w = {index: i, value: array[i]};
            if (i === 0) w.first = true;
            if (i === l - 1) w.last = true;
            wrapped.push(w);
        }
        return wrapped;
    }
}

(...) I don't really know of a concise, unambiguous way to describe that, to be honest. When an opening and a closing tag belong together, like {{#section}}{{/section}} or {{$block}}{{/block}}, I tend to call that a "tag pair", but in that case it isn't obvious whether that includes only the tags or also the contents between those tags. (...) Do you have any suggestion? Maybe we can introduce additional jargon somewhere in the spec.

right, here is what i think about blocks and sections

according to that, block equals to section, because only one section is possible in the block in the current specification.

Great picture! Now I realize we were also not talking about the same thing with "section". 😆

Let's try to align your picture with the jargon in the spec.

• Block: alas, that word already has a different meaning in the spec. Conversely, the spec has no word for what you call a "block" (at least not in the general case). We should find a different word for it, so both you and the spec can use it.
• Section: the spec would call this "content". If the spec would allow multiple of those separated by {{|}}, like your implementation does, it might call them "alternative contents" or perhaps "branches" (since you seem to be using them as branches of an if/else or a switch statement). The spec could not use the word "section" for them, because that already means something else, too: a section is what you call a "block", but only if the sigil is #.
• Type of the block: the spec would call this "tag type", or "sigil" if we are referring to the symbol.
• Block's terminator: the spec would call this "End Section Tag" if referring to the whole tag or "slash" if referring only to the sigil.

[determin1st]

@jgonggrijp

...filters would be even better (see #41 (comment)). A filter is like a lambda..

{{# numbers | iterate }}
    {{value}}
    {{^last}},{{/last}}
{{/ numbers | iterate }}

again, naming is misleading. filter is what takes 10 elements, but returns 5, it "filters" them, removes, reduces, makes the whole smaller. i tried similar template:

numbers: 
{{#arrayIterator numbers}}
{{value}}
{{^last}},{{/}}
{{/}}

with this code

$data = [
'numbers' => [1,2,3,4,5],
];
$data['arrayIterator'] = (function ($m,$a) use ($data) {
# prepare
$list = $data[$a];
$last = count($list) - 1;
$pair = [];
# compose
for ($x='',$i=0; $i <= $last; ++$i)
{
$pair[] = [
'last'  => $i === $last,
'value' => $list[$i],
];
}
return $pair;
});

how to name it? i found one - combinator, it takes two and combines them into one. but that is specifics of this particular lambda, not a general case. i wouldnt call every (lambda)(argument) or (lambda argument) a combinator or alike

(...) I don't really know of a concise, unambiguous way to describe that, to be honest. When an opening and a closing tag belong together, like {{#section}}{{/section}} or {{$block}}{{/block}}, I tend to call that a "tag pair", but in that case it isn't obvious whether that includes only the tags or also the contents between those tags.

i put more details:

• "tag pair" is irrelevant, it can be {{#hello}}..{{/world}} now as i changed the pairing algorithm

Let's try to align your picture with the jargon in the spec.

• Block: alas, that word already has a different meaning in the spec. Conversely, the spec has no word for what you call a "block" (at least not in the general case). We should find a different word for it, so both you and the spec can use it.

its misleading. you wrote earlier {{#block}}..{{/block}} but its not the block, the block is {{$block}}..{{/block}} according to the docs

• Section: the spec would call this "content". If the spec would allow multiple of those separated by {{|}}, like your implementation does, it might call them "alternative contents" or perhaps "branches" (since you seem to be using them as branches of an if/else or a switch statement). The spec could not use the word "section" for them, because that already means something else, too: a section is what you call a "block", but only if the sigil is #.

naah, each section has its own type, so they arent only content. branches grow from the tree, one from another. blocks arent trees, they are lists. i simply go to the dictionary: https://dictionary.cambridge.org/dictionary/english/section one of the parts that something is divided into

• Type of the block: the spec would call this "tag type", or "sigil" if we are referring to the symbol.

tag is a label, label to something. tag type not equals to type tag, but type to what? what object/entity/thing it classifies?

• Block's terminator: the spec would call this "End Section Tag" if referring to the whole tag or "slash" if referring only to the sigil.

okay, but section doesnt always end with the {{/}} it may end at the start of another section.

did some performance mesurements, @agentgt you may be interested in those

[agentgt]

@determin1st I expect the perf on most dynamic implementations particular in scripting languages aka REFLECTIVE to be about the same and about one order of magnitude difference which looks to be the case here.

Do you have the test code you used for the benchmark?

It's hard to make any sort of apples to apples comparison especially if threading comes into play and implementation details but here is a Java templating benchmarking test of various template engines. In Java we are talking nanoseconds here for template rendering: https://github.com/agentgt/template-benchmark (JStachio renders somewhat complex templates a million templates a seconds on so so hardware).

https://github.com/agentgt/template-benchmark/blob/utf8/results.png