Proposal on how to specify indentation of blocks

[jgonggrijp]

In #75 and #125, the inheritance spec was discussed at length. An issue that has been discussed, but undecided, is how it interacts with the notions of standalone tags and indentation.

Setting the stage

Currently, it is specified that when a standalone partial tag is indented, the entire corresponding partial template will be indented by the same amount before rendering and interpolation.

template.mustache

    {{>partial}}

partial.mustache

Hello there!

output

    Hello there!

Try the above example in the playground by pasting the following code:

{"data":{"text":"null"},"templates":[{"name":"template","text":"    {{>partial}}"},{"name":"partial","text":"Hello there!"}]}

A parent tag is essentially a partial tag with a matching closing tag, allowing it to have internal structure.

template.mustache

{{<partial}}{{/partial}}

output

Hello there!

Try the above example in the playground by pasting the following code:

{"data":{"text":"null"},"templates":[{"name":"template","text":"{{<partial}}{{/partial}}"},{"name":"partial","text":"Hello there!"}]}

A block tag essentially enables us to define and expand an embedded partial template inside a larger template. Such a block can be passed to a parent template.

template.mustache

{{<introduction}}{{$title}}Professor{{/title}}{{/introduction}}

introduction.mustache

I'm {{$title}}Darth{{/title}} {{$name}}Vader{{/name}}.

output

I'm Professor Vader.

Try the above example in the playground by pasting the following code:

{"data":{"text":"null"},"templates":[{"name":"template","text":"{{<introduction}}{{$title}}Professor{{/title}}{{/introduction}}"},{"name":"introduction","text":"I'm {{$title}}Darth{{/title}} {{$name}}Vader{{/name}}."}]}

From the recursion spec, it follows that blocks are passed across multiple inheritance levels even when intermediate levels leave the block implicit. While not directly relevant to the current issue, I will be exploiting this in some of my examples, because it enables me to separate the indentation of the parent tag from the indentation of the block tags on both ends.

template.mustache

{{<intermediate}}{{$name}}Moder{{/name}}{{/intermediate}}

intermediate.mustache

{{<introduction}}{{/introduction}}

output

I'm Darth Moder.

Try the above example in the playground by pasting the following code:

{"data":{"text":"null"},"templates":[{"name":"template","text":"{{<intermediate}}{{$name}}Moder{{/name}}{{/intermediate}}"},{"name":"introduction","text":"I'm {{$title}}Darth{{/title}} {{$name}}Vader{{/name}}."},{"name":"intermediate","text":"{{<introduction}}{{/introduction}}"}]}

The problem

In the previous section, I intentionally omitted indentation in the demonstration of parent and block tags, because there is currently no specification of how that should behave. Since a parent tag pair without blocks is basically just a partial, the following interpretation would probably be uncontroversial:

template.mustache

    {{<partial}}{{/partial}}

output

    Hello there!

Adding blocks to the mix, most people will probably also agree with the whitespace in the following example.

template.mustache

{{<intermediate}}
{{$greeting}}
high five
{{/greeting}}
{{/intermediate}}

intermediate.mustache

Hi,
    {{<invitation}}{{/invitation}}

invitation.mustache

please give me a:
    {{$greeting}}
    hug
    {{/greeting}}

output

Hi,
    please give me a:
        high five

The above examples are relatively clear-cut because all tags are standalone and the block is only indented in the place of expansion (in invitation.mustache). Things become fuzzier when indentation appears in more places,

template.mustache

{{<intermediate}}
    {{$greeting}}
        high five
    {{/greeting}}
{{/intermediate}}

invitation.mustache

please give me a:
    {{$greeting}}
        hug
    {{/greeting}}

or when some tags don't clear on both sides.

template.mustache

{{<intermediate}}{{$greeting}}
high five
{{/greeting}}{{/intermediate}}

invitation.mustache

please give me a:
    {{$greeting}}hug{{/greeting}}

Hence, in order to promote consistent behavior across different Mustache implementations, I'm suggesting to specify how template inheritance and whitespace should interact (credits to @gasche for first suggesting this). I'll open the discussion with my current ideas of what might be the best way to approach this. Before I do that, though, let me introduce some concepts that I think are relevant.

Exploring the problem space

Template space vs. comment space

Comment tags introduce "dead" space within a template that is never rendered.

before {{! never rendered }} after

To highlight this distinction, let's paint the comment space black.

before {{!████████████████}} after

Per the spec, the contents within a pair of parent open/close tags default to being comment space, too.

{{<parent}} never rendered {{interpolation}} {{/parent}}

{{<parent█████████████████████████████████████████████}}

This also visually highlights the similarity between a parent and a partial.

{{>parent}}

By way of exception, block tags can "punch holes" in the comment space within parent tag pairs, creating what could be considered "template islands", with "comment barriers" in between them.

a {{<parent}} b {{$block}} c {{/block}} d {{/parent}} e

a {{<parent███████$block}} c {{████████████████████}} e

Within a regular, continuous stretch of template space, everything is linearly ordered. As a result, everything that comes before a particular tag will directly influence the position of everything that comes after that tag.

If this wasn't here... {{#section}} ... this would be more to the left. {{/section}}

Comment barriers provide a visual way to remind us that there is no such direct relationship within and across parent tags. Blocks need not appear in the same order within a parent tag pair as within the corresponding parent template, or might even be ignored. Also, a block may be significant within a parent tag pair, but play only a minor role within the parent template.

{{<parent████$block}} If this wasn't here... {{███████████████████}} ... this might as well be just as much to the left.

We could say that the larger template outside of the parent tag pair and the template islands within them act like "parallel worlds" that don't really interact, until everything "lands" in the parent template.

Parameter blocks vs. argument blocks

A block outside any parent tag pair marks a place in the template that can be configured. It acts as a parameter of the template. If not configured from the outside, its contents serve as the default.

{{$parameter}} default {{/parameter}}

A block inside a parent tag pair provides the external configuration for such a parameter. It acts as an argument for another template.

{{<parent████$parameter}} argument {{███████████████████████}}

Definition vs. expansion

As I mentioned before, the contents of a block can be thought of as an embedded partial template. To illustrate, compare the following pair of templates:

block.mustache

{{<parent████$block}}Hello there!{{█████████████████}}

parent.mustache

{{$block}}{{/block}}

with the following:

block.mustache

Hello there!

parent.mustache

{{>block}}

The output is the same in both cases (Hello there!) and in both cases, content from one template is inserted in the other. In both cases, this content is defined inside block.mustache while parent.mustache is the place where it is expanded. The main differences are (1) which template is "outermost" and (2) how much of a template file is dedicated to the definition of the "transplanted" content. In partials, the definition is always the entire file, while in blocks, it never is.

It is important to distinguish definition from expansion, because the site of expansion can be identical to the site of definition, which might hide the distinction. In blocks, this happens when a parameter block is never configured from the outside and the default ends up being rendered.

{{$block}}Hello there!{{/block}}

There are several asymmetries between block definitions and expansions, and between parameter and argument blocks. A parameter block is a site of both definition and expansion, while an argument block is only a site of definition. Every parameter block ends up being expanded into, but not every defined block is expanded. A block defined in argument position can only be expanded elsewhere, while a block defined in parameter position can never be expanded elsewhere.

Clearance

For most tag types, it is specified that they should not introduce blank lines just by existing. This is why

a
{{!████}} b
c

renders into

a
 b
c

while

a
{{!████}}
c

renders without the middle line.

a
c

We say that the latter comment is "standalone" while the former is not. For those tag types where it is specified unambiguously, standalone always means that there is start clearance, i.e., no content except for whitespace between the preceding line start and the opening delimiter ({{) of the tag, as well as end clearance, no content expect for whitespace between the closing delimiter (}}) of the tag and the following line end.

For parent and block tag pairs, a more nuanced concept of standalone is likely necessary, due to the previous distinctions between template and comment space, parameter and argument blocks, and definition and expansion. Let's first consider the state space for a single tag ({{!}} below). There are four possible combinations depending on whether it clears on the start and end or not:

┌───────────┬─────────────────────┐
│ start     │    end clearance    │
│ clearance │   yes   ╷    no     │
│      ╶────┼─────────┼───────────┤
│       yes │ a       │ a         │
│           │ {{!}}   │ {{!}} b   │
│           │ b       │           │
│      ╶────┼─────────┼───────────┤
│        no │ a {{!}} │ a {{!}} b │
│           │ b       │           │
└───────────┴─────────┴───────────┘

In the original tags such as section, comment and change delimiter, we consider the yes/yes case in the top left as standalone and the other three as "not standalone". In order to distinguish the other three cases from each other as well, let's name all combinations with a letter: B when the tag clears on both delimiters, S when it clears only at the start, E when it clears only at the end and N when it clears on neither delimiter.

In blocks, the opening and closing tags may independently have each of the four types of clearance, so we end up with sixteen possible combinations for the block as a whole:

┌─────────┬─────────────────────────────────────────────────────────────┐
│ opening │                     closing tag                             │
│ tag     │     B     ╷     S     ╷        E        ╷        N          │
│       ╶─┼───────────┼───────────┼─────────────────┼───────────────────┤
│       B │ a         │ a         │ a               │ a                 │
│         │ {{$}}     │ {{$}}     │ {{$}}           │ {{$}}             │
│         │ b         │ b         │ b {{/}}         │ b {{/}} c         │
│         │ {{/}}     │ {{/}} c   │ c               │                   │
│         │ c         │           │                 │                   │
│       ╶─┼───────────┼───────────┼─────────────────┼───────────────────┤
│       S │ a         │ a         │ a               │ a                 │
│         │ {{$}} b   │ {{$}} b   │ {{$}} b {{/}}   │ {{$}} b {{/}} c   │
│         │ {{/}}     │ {{/}} c   │ c               │                   │
│         │ c         │           │                 │                   │
│       ╶─┼───────────┼───────────┼─────────────────┼───────────────────┤
│       E │ a {{$}}   │ a {{$}}   │ a {{$}}         │ a {{$}}           │
│         │ b         │ b         │ b {{/}}         │ b {{/}} c         │
│         │ {{/}}     │ {{/}} c   │ c               │                   │
│         │ c         │           │                 │                   │
│       ╶─┼───────────┼───────────┼─────────────────┼───────────────────┤
│       N │ a {{$}} b │ a {{$}} b │ a {{$}} b {{/}} │ a {{$}} b {{/}} c │
│         │ {{/}}     │ {{/}} c   │ c               │                   │
│         │ c         │           │                 │                   │
└─────────┴───────────┴───────────┴─────────────────┴───────────────────┘

We can identify each combination by concatenating the letters for the opening and closing tags, so from left to right, the combinations at the top row would be named BB, BS, BE and BN. If we want to promote consistent behavior across implementations, the specification will need to address unambiguously what should happen with the whitespace in all sixteen combinations.

Proposal

I suggest a small set of rules, which should unambiguously specify what happens with whitespace in all cases and which I hope will also be acceptable to most people. The following rules were in part inspired by ideas previously shared by @gasche.

0. Standalone tags

Individual block and parent tags should not introduce blank lines in the output when they clear on both delimiters. I'm numbering this rule zero, because it is not really a new rule.

1. Standalone pairs

The pair of a parent tag and its matching end section tag as a whole is standalone if the opening tag clears at the start and the closing tag clears at the end (so that's BB, BE, SB or SE). This corresponds to the intuitive notion that such a parent pair should behave like a single standalone partial tag. Another, independent justification of this rule is that the internal delimiters of the pair should not be taken into account because they are in comment space.

Like in standalone tags, the external clearance around a standalone pair should be considered part of the tags rather than part of the literal contents within the surrounding template.

The exact same notion of a standalone pair is also applicable to parameter blocks, because these are sites of template expansion, too. It does not apply to argument blocks, because those are not template expansion sites and also because their outer delimiters are in comment space.

The outermost pair in each of the following examples is standalone:

{{<parent}}{{$block}}
{{/block}}{{/parent}}

{{$block}}{{/block}}

    {{$block}}
    {{/block}}

2. Intrinsic indentation

If the opening tag of an argument block or a standalone parameter block clears at the end (so that's BB, BS, BE, BN, EB, ES, EE or EN for argument blocks but only BB or BE for parameter blocks), then the intrinsic indentation of the block is the whitespace at the start of the first line after the opening block tag.

In all remaining cases, the intrinsic indentation of a block is the empty string.

The justification for taking only the opening side of the internal clearance into account is that the absence of start clearance at the closing tag is equivalent to an external partial template that lacks a final line end.

In all examples below, the intrinsic indentation of the block is four spaces:

{{$block}}
    content
{{/block}}

{{$block}}
    content{{/block}}

{{<parent}}{{$block}}
    content
{{/block}}{{/parent}}

In all examples below, the intrinsic indentation of the block is the empty string:

{{$block}}content{{/block}}

    {{$block}}content{{/block}}

    {{$block}}
content
    {{/block}}

    a {{$block}}
    content
    {{/block}}

    {{$block}}
    content
    {{/block}} b

{{<parent}}
    {{$block}}content{{/block}}
{{/parent}}

3. Deindentation at block definition time

The intrinsic indentation of a block definition is removed from each of its lines before anything else happens with its contents.

The following example block,

    {{$block}}
        text
        {{#section}}
            text
        {{/section}}
    {{/block}}

is equivalent to an external partial template with the following content.

text
{{#section}}
    text
{{/section}}

4. Indentation at template expansion sites

This rule is a generalization of the indentation rule that already exists for standalone partial tags.

The indentation of a partial tag, parent pair or parameter block pair is determined as follows. If it is a block with intrinsic indentation, then the intrinsic indentation is used. Otherwise, if the tag or pair is standalone, then the indentation of the single tag or the pair's opening tag is used. In all remaining cases, the empty string is used.

Note that the indentation used is always whitespace that is not part of the literal contents of the template.

The indentation thus determined is added to each line of whichever (external) template or block ends up being expanded in the position of the tag or pair, before that template or block is rendered.

In each of the following examples, four spaces of indentation will be added to each line of whichever template or block ends up being expanded on site.

    {{>partial}}

    {{<parent}}{{/parent}}

    {{$block}}{{/block}}

{{$block}}
    default content
{{/block}}

In each of the following examples, no indentation (the empty string) will be added.

{{>partial}}

{{<parent}}{{/parent}}

{{$block}}
{{/block}}

Full circle

Let's revisit some of the examples that appeared earlier in this post, and see how they would render according to the above rules. The playground implements those rules faithfully, so playground savestates are provided.

The straightforward example

template.mustache

{{<intermediate}}
{{$greeting}}
high five
{{/greeting}}
{{/intermediate}}

intermediate.mustache

Hi,
    {{<invitation}}{{/invitation}}

invitation.mustache

please give me a:
    {{$greeting}}
    hug
    {{/greeting}}

output

Hi,
    please give me a:
        high five

Try the above example in the playground by pasting the following code:

{"data":{"text":"null"},"templates":[{"name":"template","text":"{{<intermediate}}\n{{$greeting}}\nhigh five\n{{/greeting}}\n{{/intermediate}}"},{"name":"intermediate","text":"Hi,\n    {{<invitation}}{{/invitation}}"},{"name":"invitation","text":"please give me a:\n    {{$greeting}}\n    hug\n    {{/greeting}}"}]}

The complex indentation example

template.mustache

{{<intermediate}}
    {{$greeting}}
        high five
    {{/greeting}}
{{/intermediate}}

invitation.mustache

please give me a:
    {{$greeting}}
        hug
    {{/greeting}}

output

Hi,
    please give me a:
            high five

Try the above example in the playground by pasting the following code:

{"data":{"text":"null"},"templates":[{"name":"template","text":"{{<intermediate}}\n    {{$greeting}}\n        high five\n    {{/greeting}}\n{{/intermediate}}"},{"name":"intermediate","text":"Hi,\n    {{<invitation}}{{/invitation}}"},{"name":"invitation","text":"please give me a:\n    {{$greeting}}\n        hug\n    {{/greeting}}"}]}

The mixed clearance example

template.mustache

{{<intermediate}}{{$greeting}}
high five
{{/greeting}}{{/intermediate}}

invitation.mustache

please give me a:
    {{$greeting}}hug{{/greeting}}

output

Hi,
    please give me a:
        high five

Try the above example in the playground by pasting the following code:

{"data":{"text":"null"},"templates":[{"name":"template","text":"{{<intermediate}}{{$greeting}}\nhigh five\n{{/greeting}}{{/intermediate}}"},{"name":"intermediate","text":"Hi,\n    {{<invitation}}{{/invitation}}"},{"name":"invitation","text":"please give me a:\n    {{$greeting}}hug{{/greeting}}"}]}

Discussion

What do you think? Can you think of edge cases where the proposed rules would give undesirable results? Can you think of rules that would give acceptable results in more cases, or simpler rules that would give equally acceptable results?

I hereby offer to write the specs if we reach an agreement. In that case, I will try them with my own implementation before submitting a pull request.

CC to all implementers and contributors who have been recently active around the spec (in random order): @gasche @softmoth @adam-fowler @Danappelxx @pvande @splumhoff @bobthecow @sayrer @spullara

[gasche]

(Thanks for the ping through #131!)

In my best experiment with indentation, I used two guiding principles:

• a tag, or pair of matching stag, is "visible" if its rendering emits output; "standalone" is redefined to "not visible"
• partials (with or without parameters) and blocks have indentation; when rendering a line, I add the indentation of the partials it is included in, and remove the indentation of the parameter blocks

(I did use a rule such that "if the start-block tag is followed by whitespace and a newline, then its indentation is the indentation of the first non-whitespace line below", similar to your "intrisic indentation" concept.)

Your system is, on the surface, more complex, but it has the advantage of being solely driven by the (whitespace) syntax of the input, without any semantic consideration. ("visible" or not depends on how the item will be rendered.) I think it makes it simpler to implement, and to integrate into existing implementations.

I don't know if the two approaches give the same results in all cases, or most cases. Intuitively I think they are very similar.

Three remarks:

• I should dig up my own implementations around this to give as a pointer. (TODO)
• We should probably build prototype implementations of whichever spec we have in mind, to test on various examples to understand how these things behave.
• Some of the notions (standalone pairs) would naturally apply to other tags (section tags, for example), and this would in fact improve the result in these settings. You have not proposed this, I suppose for backward-compatibility reasons.

[gasche]

Pointers to my previous explorations in this area:

• My specification of how to measure "indentation" is given in this github comment. I think it is essentially identical to the notion of "intrisic indentation" that you propose, which suggests that the proposal is "natural".
• The implementation of the "visibility" criterion is in this patchset (which ended up not merged in ocaml-mustache upstream as I preferred to stick to the standard behavior). (I understand that the code may be less accessible to people unfamiliar with OCaml, but the comment at least gives the ideas.)

[jgonggrijp]

I have updated the opening post to include links to and savestates for my newly developed Mustache playground. I hope this will reduce barriers for readers to experiment with the examples.

The playground is powered by my own implementation, Wontache. Because of that, it fully implements the latest spec as well as the rules proposed here. It aims to be a general, more powerful alternative to the "demo" on the official Mustache website. Whitespace is handled correctly, lambdas are supported and it is possible to enter multiple named templates alongside each other, so that partial and parent tags can be used effectively. And as demonstrated above, it is possible to share sessions with other people by copy-pasting chunks of JSON code.

Feedback welcome (@gasche)!