Grammar details

[svanimpe]

I've written a TextMate grammar (https://github.com/svanimpe/stencil.tmbundle) for Stencil and intend to use this to build a Visual Studio Code extension. Before I can finish the grammar, I need an answer to the following questions:

• [x] What are valid property names? Implemented as [^\s\.,|%}]+, meaning any string that does not contain whitespace, a dot (used as a separator), a pipe (indicates the start of a filter), a percent sign (as that may indicate the end of a tag) or a closing brace (as that may indicate the end of a variable).
• [x] What are valid filter names? Implemented as [^\s:|%}]+, meaning any string that does not contain whitespace, a colon (indicates the start of the parameters), a pipe, a percent sign or a closing brace.
• [x] What are valid tag names? Implemented as [^\s%]+, meaning any string that does not contain whitespace or a percent sign.
• [x] What are valid string literals and what escape characters are supported? Implemented as anything between double quotes. I've added general support for escape characters (without checking if they're actually valid) so nested quotes are supported.
• [x] What are valid number literals? Implemented as -?(0|[1-9]\d*)(\.\d+)?, which is simpler than what Swift supports, but it should be sufficient. I've also added support for numbers as properties and numbers as filter parameters.
• [x] Are filter parameters always strings? Answer: they can be strings, numbers or variables. I've added support for all of these.
• [x] Can filters have multiple parameters? Answer: yes, using comma as a separator. I've added support for this.
• [x] Are there any language constants like true, false or self? Answer: currently not, but this may change.
• [x] What is allowed inside a {{ }} variable? Answer: literals, properties and filters. Expressions may be supported in the future.
• [x] Are spaces around operators required? Answer: yes. My grammar requires this as well.
• [x] Are spaces around number literals required? Answer: yes. My grammar requires this as well.
• [x] Are spaces inside tags required? Answer: no. I've updated the grammar to reflect this.

[svanimpe]

@kylef or @ilyapuchka Can I please get some feedback on this?

[AliSoftware]

I haven't looked at the PR and grammar in details, but what I can say is:

• For now I think all filters and tags names fit your RegEx, but I think future filters could potentially contain _ in their name too maybe
• Some boolean filters also provide a negative counterpart which by convention starts with !. See #160
• Technically I think some filters could support numbers as parameters (even if I don't think any filters provided either by Stencil not StencilSwiftKit have one? but future filters might)
• Filters can have multiple parameters. One example of that is the replace filter provided by StencilSwiftKit
• About spaces, you can write {%for x in list%}{{x}}{%endfor%} without any space if you like (even if it's nicer to read with spaces imho)

[svanimpe]

@AliSoftware Thanks.

Can you show an example of a filter with multiple parameters? Is this filter:param:param or filter:param,param, or something else?

[AliSoftware]

That's actually a very good question, since it seems that we don't use the replace filter anymore in the templates we have in SwiftGen, so I don't have any example template using replace, and that syntax isn't documented anywhere it seems. But according to the source code here, it splits the parameters of a filter using ,. So that would be something like {{"Hello world"|replace:"world","Steven"}} I guess :wink:

[ilyapuchka]

Here is what I can tell to the extent of what I remember from implementation details. Might not be entirely correct. I would suggest to also refer to tests.

What are valid property names? The current regex is [a-zA-Z][a-zA-Z0-9]* for a single level, with multiple levels separated by dots.

any string that can be a dictionary key, . is used as key path separator

What are valid filter names? The current regex is [a-zA-Z][a-zA-Z0-9]*.

any string except | which is used as filters separator

What are valid tag names? The current regex is [a-zA-Z]+.

any string except {%, %}. it also depend on how the tag is parsed, some of them like for have more complex syntax than just a tag name. Usually tags are closed with end\(tag name) tags. Some of the tags have other special tokens like else or empty

What are valid string literals and what escape characters are supported? I currently support only \ \n \t \".

everything inside double or single quotes is parsed as a string literal. As far as I can tell there is nothing implemented for escaping yet.

What are valid number literals? The current regex is -?(0|[1-9]\d*)(.\d+)?, which means an optional minus sign, followed by an integer that does not start with 0 (unless 0 is the only digit), followed by an optional fractional part. There is no support for exponential, binary or hexadecimal notation.

anything that can be converted to Int or Float in Swift. I would refer Swift docs for that or just try different representations in a playground or REPL. I think exponential and hex form (0x) is actually allowed, but (strangely) not binary (0b) or octet (0o), which are only supported as literals

Are filter parameters always strings? I currently only support strings.

they can be string or number literals or variables names

Can filters have multiple parameters? I could not find any examples of this, so I currently support only one parameter.

yes, comma is used as a separator, spaces are not allowed (there is PR for that #178)

Are there any language constants like true, false or self? None are mentioned in the documentation.

not in the master, there is PR for that - #164

What is allowed inside a {{ }} variable? I currently support properties with optional filters, string literals and number literals, but not expressions (containing operators).

yes, expressions are not supported on master, there is PR for that - #164

Are spaces around operators required? Currently yes.

yes, spaces are required between operators

Are spaces around number literals required? Currently yes.

if they are used as part of other expression, i.e. with operator or in tag then yes. Space is used as default separator for tokens. Other separators are |, ,, :

Are spaces inside tags required? Currently no, but I'd like to require this, to promote readable code.

they are not required, but I agree that with spaces its more readable

[svanimpe]

@ilyapuchka Thanks for your input!

(properties) any string that can be a dictionary key, . is used as key path separator

I assume , and | (and whitespace) are not allowed too?

One more question: in {% for key,value in dict %} no space is allowed after the comma? I am making this assumption because the parser requires token in at index 2: https://github.com/kylef/Stencil/blob/2e80f70f6742c0ae44e32d5a56697e6392fa27df/Sources/ForTag.swift#L13 If there were a space after the comma, there would be an additional token, no?

[ilyapuchka]

| is not allowed as it will be used to separate filter from variable name. I guess any other characters are fine. Similarly filter names should not have : as it is used to separate filter name from its arguments. And as arguments are separated with , they are not allowed in variable names if they are used as filter arguments. For simplicity in the grammar I would not allow any of separators characters in variables/filters/tags etc names. Spaces are currently not allowed in filter expressions and variables lists, there is PR that fixes that #178

[svanimpe]

@ilyapuchka @AliSoftware I've updated my questions to include your answers and how I've decided to implement it.

With that, the code is ready to release and my Visual Studio Code extension can now be dowloaded from the Marketplace or from within Visual Studio code itself :)

https://marketplace.visualstudio.com/items?itemName=svanimpe.stencil

In the future, can you please @mention me or open an issue in https://github.com/svanimpe/vscode-stencil if you make changes to Stencil that require an update to my grammar?

[ilyapuchka]

👍