Feature: Subexpressions

[Swivelgames]

Sorry for the rapid emails.

Another feature, if I were to have my ideal wish, would be to allow mustache inside mustache.

{{myfunc {{anotherFunc "thing"}}}} -> {{myfunc "result_of_anotherFunc"}}

Sadly, this would probably take rewriting the entire parser.

Originally posted by @fidian in https://github.com/tests-always-included/mo/issues/61#issuecomment-1496361833

I felt compelled to split this off from #61, since I've too been mulling over how something like this could be implemented.

First and foremost, this might be another item to reference from Handlebars. It introduced a similar feature with Subexpressions, which might be perfect for this:

{{outer-helper (inner-helper 'abc') 'def'}}

[Swivelgames]

I know we're a bit all over the place, but hear me out: If we were to implement this, it might actually resolve #58, #61, and #62, especially if we interpolate inner first:

It would allow Mustache inside Mustache:

h_var="hello"

uppercase() {
    echo "${1^^}"
}

concat() {
    echo "$@"
}

This: {{concat (uppercase (h_var)) "world" "!"}}
Becomes: {{concat "HELLO" "world" "!"}}
And outputs: "HELLO world !"

And especially if we interpolate inner first, it would also allow:

x="test"
test="foo"
declare -a arr1
arr1[test]="bar"
ref="arr1"

{{(x)}} => {{test}} => "foo"
{{myfunc (test) }} => {{myfunc "foo"}}
{{myfunc ((x)) }} => {{myfunc "foo"}}
{{arr1.(x)}} => {{arr1.test}} => "bar"
{{(ref).test}} => {{arr1.test}} => "bar"
{{(ref).(x)}} => {{arr1.test}} => "bar"

[Swivelgames]

@fidian Implemented in #65 and here's what it looks like:

Environment:

x="repo"
i=2
repo=( "resque" "hub" "rip" )
quote() {
    echo "'${MO_FUNCTION_ARGS[@]}'"
}
double_quote() {
    echo "\"${MO_FUNCTION_ARGS[@]}\""
}

Template:

X: {{x}}
Function: {{quote (x)}}
Specific Element: {{repo.(i)}}
Loop:
{{#(x)}}
  <b>{{double_quote (quote (.))}} {{.}}</b>
{{/(x)}}

Output:

X: repo
Function: 'repo'
Specific Element: rip
Loop:
  <b>"'resque'" resque</b>
  <b>"'hub'" hub</b>
  <b>"'rip'" rip</b>

[fidian]

Very interesting.

[Swivelgames]

@fidian Just updated to fix the edge cases!

Thoughts? :slightly_smiling_face:

[fidian]

First off, I think that we'd want to preserve Mustache syntax, so {{h_var}} would treat h_var as a name, look up its value, then place that into the template. Also, when calling functions, I think {{my_func h_var}} should pass the value of $hvar to my_func. I shall continue to use this idea in the following discussion. This blurs into how we treat quoted strings because {{my_func "string" x}} should receive the value "string" and $x, so not all parameters are names and not all parameters are values. I'd like to have consistency regardless of where the parameter is located and I'll try to explain this a bit with examples.

How does one differentiate between a name and a value? If I remember right, {{(x)}} would look up $x, replace (x) with the value of $x, then evaluate the expression again. So, if $x was set to "PWD", then {{(x)}} would result in the current working directory. Thus, x is a name, (x) looks up the name and replaces it with the value of $x, which results in {{PWD}}. PWD is treated as a name again and its value is inserted into the template. That seems straightforward enough for me.

h_var="hello"

uppercase() {
    echo "${1^^}"
}

concat() {
    echo "$@"
}

Mustache: {{concat (uppercase (h_var)) "world" "!"}}

At this point, concat, uppercase, h_var are names where as "world" and "!" are values because they are quoted. (h_var) looks up the name and replaces the subexpression with hello, but is that a name or a value? I presume the parenthesis act like eval, so (h_var) is replaced with hello in the expression.

Mustache: {{concat (uppercase hello) "world" "!"}}

When we get to this level, it appears that hello is a value, not a name because there's no variable $hello. This is where it gets confusing because it seems that the first item in the parenthesis is considered a name but subsequent items are considered values.

[Swivelgames]

I see what you're saying now. Definitely a breaking change, but it does make it more consistent with other implementations. Granted, it complicates things a bit, but I think it promotes consistency and encourages quoting.

In which case:

{{concat "hello" "world"}} => concat "hello" "world"
{{concat hello "world"}} => concat "$hello" "world"
{{concat (hello) "world"}} => concat "$$hello" "world"

This alone wouldn't need much tweaking to the existing implementation either. It would really just affect moCallFunction, to encourage it to interpolate barewords as variables.

[Swivelgames]

Well... I ate my words on that one :joy:

That was quite a bit bigger of an undertaking than I had initially anticipated... mainly because of the need to quote/unquote things.

[fidian]

First, I would like to thank you for your patience, ideas, and the work you've put into this project.

I've been playing around on a branch to see if I could make the parser more intelligent. I think it's done, but I want to go through the code again and scour it for problems before I really release it to the world. It's on the rewrite-parser branch. Take a look if you like. Your ideas and the stream of emails has encouraged me to revisit this and tackle many outstanding issues, especially with regard to standalone tags, whitespace problems, parsing arguments, and function calls. I really like how you suggested (x) could mean we want to look up the value of $x and my changes expanded that slightly to include braces.

Foundationally, I tried to follow Mustache syntax, but interpreted it a little differently. {x} now will insert the value of $x and (x) will insert a name of $x. Both can be nested. I also allow for concatenation automatically, single quoted, and double quoted strings. How does this matter? {{x}} will just insert the value of $x, and if x=123 then "123" is put into the result. {{{x}}} is parsed in two phases; first {x} is looked up and changed to $x as a value, then {{'123'}} is evaluated in the template, which results in "123". The outward behavior of {{x}} and {{{x}}} appear the same, just like before. However, this gives me a good starting point for differentiating between wanting a name and wanting a value. Here's an example of syntax.

x=123
a=AAA
b=BBB
AAA=3a
BBB=3b
AAABBB=mixed

func() {
    echo "__${1-}__${2-}__"
}

{{x}} --bash--> $x --result--> 123
{{{x}}} --mustache--> {{'123'}} --result--> 123

{{func a b}} --bash-> func "$a" "$b" --result--> __AAA__BBB__
{{func 'a' "b"}} --bash--> func "a" "b" --result--> __a__b__
{{func (a) (b)}} --bash--> func "$AAA" "$BBB" --result--> __3a__3b__
{{func (a b)}} --bash--> func "$AAABBB" --result--> __mixed____
{{func {a} {b}}} --bash--> func "AAA" "BBB" --result--> __AAA__BBB__
{{func {a b}}} --bash--> func "AAABBB" --result--> __AAABBB____

With this setup, there's only a few rules that need to be explained.

• Treat {a} as a value, effectively converting {a} into 'AAA' and interpret as the value "AAA"
• Treat (a) as a name, effectively converting (a) into AAA, then using it like $AAA
• Single quoted strings can contain anything except single quotes.
• Double quoted strings are like single quoted strings but the following standard escape sequences may be used: \\, \", \r, \t, `\n'
• Multiple items in a mustache tag, set of braces, or parenthesis are concatenated together without a space. {{'hello' "world"}} produces "helloworld" (no space).

I'm sure the branch has extra problems that are hiding, but all 37 tests pass with reasonable result, which is good enough for tonight. I'll work on testing against the official specs next week, reformatting code to look more presentable, splitting large functions for maintainability, cleaning up unused variables, making sure I declared all variables as local, etc. I've also got to update the documentation to include the rules above, information about how functions are now executed in strict mode, how to enable debugging with MO_DEBUG or -d, how to upgrade templates (the behavior for function arguments changed), and remove mentions about how function arguments use eval in an unsafe way. Lots of good changes on the way.

[Swivelgames]

This is looking great! I have to say, I appreciate the responsiveness and receptiveness that you've demonstrated throughout all this. I know I kind of came out of nowhere with quite a few ideas and contributions; not everyone is as receptive or intrigued!

The code is much cleaner, and I love how everything has been broken out. Definitely makes things easier to parse with the eyes (especially the double-colon: mo::)

As for the parser rewrite, this is looking very promising!

Array Lookup

One area that I noticed it struggled with was when interpolating the keyword itself. For instance:

i=2
k="foo"

declare -a repo
repo=( "resque" "hub" "rip" )

declare -A assoc
assoc[foo]="111"
assoc[bar]="222"
assoc[baz]="333"

template() {
    cat <<EOF
Specific Array Element: {{repo.{i}}}
Specific Array Element: {{repo.(i)}}
Specific Assoc Element: {{assoc.{k}}}
Specific Assoc Element: {{assoc.(k)}}
EOF
}

All of these resulted in either:

../mo: line 1557: {i: syntax error: operand expected (error token is "{i")
../mo: line 1557: {k: syntax error: operand expected (error token is "{k")

or:

../mo: line 1557: (i: missing `)' (error token is "i")
../mo: line 1557: (k: missing `)' (error token is "k")

Obscure tests that also failed

Additionally, in my attempts to finagle it a bit, the following gave an "Unbalanced parenthesis" error:

Specific Array Element: {{ {'repo.' (i)} }}

ERROR: Unbalanced parenthesis near (i)} }}

Just as well, these:

Specific Array Element: {{repo.2}}
Specific Array Element: {{{"repo." {i}}}}
Specific Array Element: {{("repo." {i})}}

Produced:

Specific Array Element: rip
Specific Array Element: repo.2
ERROR: Unbalanced parenthesis near ("repo." {i})}}

Back to array lookup

Granted, I don't think any of these more obscure scenarios are practically relevant, given that the initial attempts probably just need some minor attention to the parser to get working:

Specific Array Element: {{repo.{i}}}
Specific Array Element: {{repo.(i)}}
Specific Assoc Element: {{assoc.{k}}}
Specific Assoc Element: {{assoc.(k)}}

Not sure if I'll have the time tonight to poke at it much, but I've been studying the branch, and it already looks very intriguing!

(Also, love the new Test functionality!)

[Swivelgames]

Unfortunately, I've been quite swamped with a number of things that have prevented me from taking a look at this, but I'm hoping to give it a look if I can get some other items off my plate soon enough.

[Swivelgames]

I made an exploratory adjustment and pushed it up in #66 that allows for:

i=2
declare -a repo_arr
repo_arr=("resque" "hub" "rip")
template="Specific Array Element: {{repo_arr.{i}}}!"
expected="Specific Array Element: rip!"

However, the following syntax is not yet supported. This PR will likely be replaced with one that supports the below:

arr_prefix="repo"
i=2
declare -a repo_arr
repo_arr=("resque" "hub" "rip")
template="Specific Array Element: {{{arr_prefix}_arr.{i}}}!"
expected="Specific Array Element: rip!"

This example currently fails with the following error:

ERROR: Unable to index a scalar as an array: _arr.1

arr_prefix seems to get interpolated to repo, but then its discarded.

[fidian]

I have another rewrite of the internals coming soon. Right now it can evaluate tags within a discarded block, which is wrong and is causing tests to fail. I only have a little time each day, but I am not forgetting about this.

On Tue, Apr 18, 2023, 18:30 Joseph Dalrymple @.***> wrote:

I made a cursory adjustment and pushed it up in #66 https://github.com/tests-always-included/mo/pull/66 that allows for:

i=2declare -a repo_arr repo_arr=("resque" "hub" "rip") template="Specific Array Element: {{repo_arr.{i}}}!" expected="Specific Array Element: rip!"

However, the following syntax is not yet supported. This PR will likely be replaced with one that supports the below:

arr_prefix="repo" i=2declare -a repo_arr repo_arr=("resque" "hub" "rip") template="Specific Array Element: {{{arr_prefix}_arr.{i}}}!" expected="Specific Array Element: rip!"

This example currently fails with the following error:

ERROR: Unable to index a scalar as an array: _arr.1

arr_prefix seems to get interpolated to repo, but then its discarded.

— Reply to this email directly, view it on GitHub https://github.com/tests-always-included/mo/issues/63#issuecomment-1513909548, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADISPADYBIPNJTYQ7HLVQ3XB4P2BANCNFSM6AAAAAAWTLKWBA . You are receiving this because you were mentioned.Message ID: @.***>

[Swivelgames]

Hey, no problem at all! I totally understand. I'm kind of in the same boat. I had a good week or two that I had the bandwidth to throw a bunch of effort at this, but that's been reduced quite a bit as of late.

Right now, I'm trying to get it to cover the last three remaining scenarios from my second comment in this issue:

{{arr1.(x)}} => {{arr1.test}} => "bar"
{{(ref).test}} => {{arr1.test}} => "bar"
{{(ref).(x)}} => {{arr1.test}} => "bar"

Here's to hoping!

This has been a pretty fun puzzle from the beginning. Really curious to see what you come up with. I've enjoyed not only contributing what I have, but also the discussion and seeing your approach to these things.

[fidian]

This has been implemented in the latest version. Thanks for the great discussion and your patience with how long it took to implement.