Filter literals

[samdeane]

The above commit provides a very trivial implementation of literals (issue #37), by simply checking whether the first character of the identifier is a quote.

It seems to work, but may have all sorts of unintended consequences...

Just made the request for the purposes of discussion really - I'm not necessarily expecting this code to be merged...

[groue]

Hi, Sam. I'm currently in Collioure, a wonderfully nice city of the French Riviera near Spain, and spend most of my time at the beach, enjoying the sun and the sea. I'll be back by next Monday, in one week. Until then, your pull requests will remain frozen :-) I promise I'll give a good look at them when I'm back :-) Your work looks very promising, thanks a lot!

[groue]

Hi, Sam.

Just made the request for the purposes of discussion really - I'm not necessarily expecting this code to be merged...

I'm glad to hear that:-)

It seems to work, but may have all sorts of unintended consequences...

Well, I don't see unintended consequences here...

However, I try to make sure GRMustache is not half-baked. Most common use cases and usual Mustache pitfalls are covered, and the library is hookable enough to let the user hack in, in confidence, with loads of sample code (maybe too much). Confidence is key, for me: the confidence that one builds when one uses a solid, reliable, non-surprising library, and realizes that the eventual bugs are his. A lot of work has been put in this crazy lib, so that the end-user doesn't have to even think about it.

OK. So, reliable literals. At least strings & numbers, of course.

Strings:

• All strings should have a literal, including the string containing a single quote. Backslash-escaping is the most natural syntax, since we're in Objective-C "\"".
• OK, so backslash must be escaped as well: "\\".
• It's possible that the user expects \n and \t etc. to be supported, and tries to use them: OK, there's no reason not to support them, so let's support all the standard C escaping. We would also support raw newlines and tabs, of course, there is no reason not to, after all.
• Syntax for escaped unicode literals ? This would go too far, let's wait for a feature request. GRMustache can already read templates encoded in all Foundation-supported encodings, through GRMustacheTemplateRepository: people will embed raw :smile: and :poop:.
• Thinking about it, since we're a Mustache implementation, and that Mustache is supposed to be a cross-platform language, we may, instead of the C string syntax, opt for the well-known, solid, de-facto standard JSON string syntax, which is consistent with the C syntax for most common characters. Ha, crap, raw newlines are not supported. Anyway, it looks like a way better solution, don't you think?
• Syntax errors? Sure. "foo is not a valid string. Haaa... But it may be a valid key in an object... Crap, this is a serious difficulty. I don't have any answer yet.

Numbers:

• If we use JSON string syntax, JSON number syntax is a logical choice.
• Since we support numbers, and we're an objective-C library, what about adding YES and NO to the built-in literals? Or true and false? OK, I don't know. But some filters may take boolean arguments, after all, and 0 and 1 are valid, but sub-optimal ways to express them. Hummm... JSON supports true, false, and null. Sold.
• Syntax errors? Sure. 1x is not a valid number. Haaa... But it may be a valid key in an object... Crap, this is a serious difficulty. I don't have any answer yet.

Performance:

• I try to avoid parsing when rendering, and the GRMustache parser and compiler emit ready-made objects that already know exactly what they have to do. So strings & numbers should be parsed by the GRMustacheExpressionParser class. GRMustacheExpressionParser basically is a state machine, it should not be too hard to add a few states to parse strings and numbers.

Thinking again about "foo and 1x, I realize that "foo" and 1 are nasty identifiers: when literals are not supported, they are keys in some object. Uncommon ones, I agree. Still, I'm not sure to like literals to prevent the user to express the "foo" and 1 keys in an object: this is a clear incompatibility. I don't know how to solve this.

Well, Sam. We have read my raw thoughts, with a few turnovers on the string syntax, and open questions :smile:

Now you understand why issue #37 has not found yet its solution: it's a lot of work, and introduces an incompatibility :sweat_smile:. I hope you understand why I want literals to be serious ones.

[groue]

In some other implementations (PHP, maybe some other ones), a tag like {{ items.0 }} renders the first element of the items array. This is because, roughly speaking, foo.bar is evaluated as foo["bar"]. So items.0 is evaluated as items["0"], that is identical to items[0] in some weakly typed languages.

So there is an actual practice, today, to use numbers as a way to access the Nth item of an array. This is of course not specified, but this should be taken in account. Maybe this feature should be added to GRMustache, for the sake of cross-platform-compatibility.

Assuming support for numeric literals, the same 0 in {{ 0 }} and {{ items.0 }} would have different semantics (literal, and key).

I feel uncomfortable with this.

[samdeane]

Ok, I understand your points about consistency and reliability, parsing etc.

You are already supporting arguments for filters, however, so you must already be doing some parsing, and there may be some subset of characters which are currently illegal to use as a filter argument. Actually I was half expecting the quote character to be one of them, so I was pleasantly surprised when it turned out not to be.

It seems reasonable therefore to continue with your existing parsing rules for filter arguments (whatever they are), and to defer the decision about whether an argument is a literal or a reference to an object in the context until the argument is actually used - that's the approach I took by just waiting for hasValue:withContext:protected:error: to be called. This has minimal performance implications as far as I can see.

Since this is a text-based system, and since any filter that we're passing an argument to is likely to have been implemented by us, it seems perfectly ok to me to say that literals must be text. If the filter implementation wants to coerce them to be numbers or anything else, that's up to it. I admit that this isn't quite as pretty as supporting numerical literals directly, but I don't think it's the main priority.

So the only problem is how to recognise a literal.

It seems to me that the small change of saying that an identifier is a literal if it is wrapped with quote characters should be safe in many cases, since identifiers starting with quotes aren't likely to be common.

However, if you wish to allow for ultimate flexibility, you could allow the literal delimiter character to be an option that is passed in to the engine (or set somehow - a pragma might also be a useful way to be able to set it).

You could therefore make the default behaviour that literals not be supported; only by setting the literal delimiter would the user of the engine indicate that they wanted to use literals. By setting it, they would implicitly be guaranteeing that the delimiter character that they chose was not going to be used as a legitimate identifier.

I do understand your quest for purity with the design, but pragmatically I needed them now, so this patch was the solution I've come up with as a first attempt. The suggestions above would make it a little bit better I think, but of course I can just continue to maintain my fork if you don't think that we've arrived at a perfect solution yet :)

[samdeane]

One other comment.

You said:

Assuming support for numeric literals, the same 0 in {{ 0 }} and {{ items.0 }} would have different semantics (literal, and key).

I'm not sure why you'd ever want to interpret the 0 as a literal in either of those cases.

The only place I can see there being any point interpreting a literal is inside a filter use - and I don't think that case it going to be nearly as open to ambiguity.

[groue]

The only place I can see there being any point interpreting a literal is inside a filter use

Well, some other people will see something else. For example:

"Set delimiters" tags {{=<% %>=}} allow to change the syntax of Mustache tags. Some guy once tried to set delimiters to... the current delimiters: {{={{ }}=}}. GRMustache used to choke on this tag. Some would wonder why would anybody want to use this void and useless tag: well, go read issue #38.

I don't buy any "I can't see..." argument any longer.

For imaginative people, those who can see what I can not see, I need the grammar of expressions to be regular, consistent, predictable, with a few simple building rules, and no artificial exception. If a and b are valid expressions, then a(b) is also a valid expression, and reciprocally. If f("foo") is a valid expression, so is "foo" (literal outside of a filter). If items.0 is a valid expression, so are items and 0, with the same semantics as {{#foo}}{{bar}}{{/foo}} vs. {{foo.bar}} (access through context stack vs. direct access to foo's bar).

So the only problem is how to recognise a literal. [...]

Your ideas are very interesting. I was keen to throw the whole literal idea away, and... well, you may have found a solution!

{{ 'name }} (using the single quote as literal marker) would simply render "name". {{ f(blah, 'foo) }} would call the f filter with the value of blah, and the raw "foo" string.

Thanks!

This will directly send GRMustache into the Greenspuns Tenth Rule Of Programming, you know ? :smile:

[groue]

{{ 'name }} (using the single quote as literal marker) would simply render "name". [...]

Sorry, I was mislead. Should we have support for literals, we would need support for strings, including strings containing white space. The quote in {{ 'name }} is not a literal marker, it is a quoting character, just as its Lisp counterpart. The quoting character lets the user provide expressions as data: {{ f('g(x)) }}, {{ f('g('x)) }} (maybe useful, but this is not the topic of this issue).

[groue]

Here is some sample code:

// This object turns literal strings into strings.
@interface MustacheLiterals : NSObject
@end

@implementation MustacheLiterals
- (id)valueForKey:(NSString *)key
{
    // When asked for `"foo"`, return `foo`
    NSUInteger length = [key length];
    if (length > 2 && [key characterAtIndex:0] == '"' && [key characterAtIndex:length-1] == '"') {
        return [key substringWithRange:NSMakeRange(1, length - 2)];
    }
    // Not a string literal
    return nil;
}
@end

// Add a MustacheLiterals object to the base context of all templates,
// so that all templates can use string literals.
MustacheLiterals *literals = [[MustacheLiterals alloc] init];
GRMustacheConfiguration *configuration = [GRMustacheConfiguration defaultConfiguration];
configuration.baseContext = [configuration.baseContext contextByAddingObject:literals];

// Render `foo: {{ "foo" }}, FOO: {{ uppercase("foo") }}`
NSString *templateString = @"foo: {{ \"foo\" }}, FOO: {{ uppercase(\"foo\") }}";
GRMustacheTemplate *template = [GRMustacheTemplate templateFromString:templateString error:NULL];

// foo: foo, FOO: FOO
NSString *rendering = [template renderObject:nil error:NULL];

I think the pull request can be closed now.

[jorisroling]

I love this capability. But... I found that the template {(contacts: uppercase("contacts.png")}} does not work because of the . (dot) in it. Removing that allows it to work (showing "contacts: CONTACTSPNG").

Any idea?

[groue]

Oops, @jorisroling, you're right. Without proper built-in support for literals, I'm forcing application code to provide it, and we run into this kind of issue. At least the "bug" is in the application code, not in the lib.

The gist of the problem, the reason why I'm reluctant to provide litteral support right away can be expressed this way:

1. Being, I hope, a quality library - and that means that the few provided services are reliable and complete, keeping bad surprises away, GRMustache can not provide a louzy support for literals. Handlebars.js provides string, integers and boolean literals, and this is indeed the bare minimum.
2. Unfortunately, Integers literals create a problem:
   1. Accessing the first item of an array through the items.0 syntax is supported by a non-negligible number of other Mustache implementations. Enough for this syntax to be a candidate for specification (should it rise from its ashes). BTW, items.[0] is the equivalent Handlebars.js syntax.
   2. The {{a.b}} syntax is deeply linked to {{#a}}{{b}}{{/a}}. Both a and b are of the same nature: they are keys, that can be found alone, or joined by a dot, and are used for querying the context stack: "hey, top of the stack, do you have something called a? b?".
   3. From the two last points above, I infer that 0 is a key: items.0 -> "hey, items, do you have something called 0?". Yes indeed, 0 is a key, not a literal.
   4. 0 being a key, it's always a key. Just as b, which is a key in both {{a.b}} and {{#a}}{{b}}{{/a}}.
3. 0 being a key, I can not provide support for integer literals. Hence I can not provide support for string literals. And we're doomed.

The only way out is to ditch the items.0 syntax (supported by a few Mustache implementations, not by GRMustache right now), for the reason of being a conceptual monstruosity (which it is).

I've already done that in the past (ditching monstruosities)... And each time GRMustache gets more isolated from the Mustache family. The dream of a usable cross-platform Mustache language is melting under the acid rains...