Help wanted: implementing my own liquid filters

[sc0ttj]

Hi all,

Thanks for making mo, first of all.. I'm using it in a shell-based static site generator called mdsh.

I'd like some help using mo, if that is alright..

Specifically, for a PR on my own project, I have done some little hacks to make mo support a liquid-like syntax for filters, like this:

{{some_var | some_func}}

and a more complex one:

{{some_var | some_func param1 | another_func}}

As in

{{page_title | truncate_words 6 "..." | uppercase}}

However, whenever I use a custom iterator, as defined in your examples, I am really struggling to make mo respect these filters (I.E, evaluate "the var piped to the filter(s)" and return that).

In my project, outiside the mo script itself, I have the following custom iterator (derived from your demos), which I re-use in various places, by way of resetting and re-defining the ITEMS array it uses before running mo:

# ITEMS():  A custom mustache iterator called ITEMS:
#           Uses special keywords ITEMS and ITEM.
#
# Usage:
#
# Set an array of hashes, which must be called ITEMS:
#
#   declare -A archive=([url]="archive.html" [name]="Archive page")
#   declare -A contact=([url]="contact.html" [name]="Contact page")
#   ITEMS=(archive contact)
#
# You can then use nested array data in your mustache templates like so:
#
#   {{#ITEMS}}
#     {{ITEM.name}} is at {{ITEM.url}}
#   {{/ITEMS}}
#
# Finally, build the output like so:
#
#   cat path/to/some-file.mustache | mo > some-file.html
#
function ITEMS {
  # The block contents come in through standard input. Capture it here.
  local content="$(cat)"
  local length=${#ITEMS[@]}
  local i=0
  # Get list of items
  for ITEM in "${ITEMS[@]}"; do
    # String replace ITEM_ with the name
    # This changes everything in the content block of the template.
    # It rewrites {{ITEM.name}} into {{foo.name}}, for example - where
    # 'foo' is a hash with they key 'name'.
    # You can prefix your environment variables and do other things as well.
    echo -ne "$content" | sed "s/{{ITEM/{{${ITEM}/g"
    i=$(($i + 1))
    # if not last item in array, add comma
    if [ "$AUTO_APPEND" != '' ];then
      [ "$i" != "$length" ] && echo "$AUTO_APPEND" || echo ''
    fi
  done
}
# export the function so `mo` can use it
export -f ITEMS

^ that is the iterator in which my custom liquid-style filters don't work - they get ignored if I use something like this:

{{#ITEMS}}
  <p>{{ITEM.name | uppercase}}</p>
{{/ITEMS}}

^ the value of variable ITEM.name is not converted to upper case :(

...I have a few different custom iterators similar to {{#ITEMS}}, and so I would love to make the filters work inside these iterators by making changes inside my mo script - so that things stay DRY, and I don't need re-code filter support every time I make an iterator.

So far, these are the dumb little hacks I did to mo itself to make it support the filters stuff:

Around line 745, end of main case statement, when processing regular {{things}}, I set a liquid_filter boolean to true if a pipe was detected in the args:

            *)
                # Normal environment variable or function call
                moStandaloneDenied moContent "${moContent[@]}"
                moArgs=$moTag
                moTag=${moTag%% *}
                moTag=${moTag%%$'\t'*}
                moArgs=${moArgs:${#moTag}}
                moFullTagName moTag "$moCurrent" "$moTag"

+                # sc0ttj: detect liquid filters (detected by finding
+                # a pipe (|) character in $moArgs (the stuff after
+                # the var/func
+                liquid_filter=false
+                if [ "$(grep -m1 '| ' <<< $moArgs)" != "" ];then
+                  liquid_filter=true
+                fi
+
                # Quote moArgs here, do not quote it later.
                moShow "$moTag" "$moCurrent" "$moArgs"
                ;;
        esac

And later in the moShow function, we do the evaluation, to support filters for variables like {{var}}:

            # shellcheck disable=SC2031
            if moTestVarSet "$1"; then
-                echo -n "${!1}"
+                # if $liquid_filter=true, process it here
+                if [ "$liquid_filter" = true ] && [ ! -z "$3" ];then
+                  # $3 is our filter (the bit after the pipe)
+                  eval "echo -n '${!1}' $3"
+                else
+                  # no liquid filters being used
+                  echo -n "${!1}"
+                fi
            elif [[ -n "${MO_FAIL_ON_UNSET-}" ]]; then

Lastly, I added the following, so calling a function, like {{some_func}}, also supprots filters:

moCallFunction() {
    local moArgs

    moArgs=()

    # shellcheck disable=SC2031
    if [[ -n "${MO_ALLOW_FUNCTION_ARGUMENTS-}" ]]; then
        moArgs=$3
    fi

+    # if $liquid_filter=true, process it here
+    if [ "$liquid_filter" = true ];then
+      # $3 is our filter (the bit after the pipe),
+      # add it to the arguments after the function ($1)
+      moArgs="$3"
+    fi
    echo -n "$2" | eval "$1" "$moArgs"
}

I don't really understand mos internals.. but these changes seem to be enough to get mo to evaulate and return the piped filter stuff most of the time, but NOT with {{ITEMS.foo}}, inside a custom iterator, or with iterators themselves (see my comment below)...

I would love for that to be sorted, so that the experience (that "filters work in Mustache templates") is consistent in my project ..

Any advice would be greatly appreciated.

[sc0ttj]

BTW...

I would also ideally like my project to have more filters for arrays, like liquid does.. such as limit, sort_by, exclude, etc..

But aside form not knowing how to make filters work inside iterators, I also don't don't know how to implement filters on the iterators themselves:

For example.. something like this would be very, very useful to me:

{{#ITEMS | sort_by 'age' 'asc' | limit 10}}
  Name: {{ITEM.name}}
  Age: {{ITEM.age}}
{{/ITEMS}}

^ would show the 10 oldest people.. or whatever...

..Essentially, I guess I'd like to know how to run the array (in this case ITEMS, an array of hashes) through the filters, before actually running the iterator in mo..

Obviously I'm not asking for you guys how to implement the filters, but your thoughts or advice on passing the array to them would be greatly appreciated - you're obviously bigger shell experts than me, and it must be do-able.

EDIT: BTW, if anyone cares, I have most of this working fine now - not including custom iterators (see above) - except sort_by .. I have sort_array [asc|desc], so I just need to write some code that goes down into the hashes of the array, if that is what it contains, then re-orders the main array based on hash contents :+1:

Thanks.

[sc0ttj]

EDIT: I've moved this last comment to a new "feature request" issue

It asks for:

1. Nicer iterators
2. A global partial dir
3. Partials with params

[fidian]

I am lost in the amount of detail that was provided. It looks like you implemented pipes (shell terminology - I you call them liquid filters) and that you're asking about how to manipulate the array that's used in the template. The point of mustache templates is that the data is prepared ahead of time and the templates are logic-less, as seen by the first three words on mustache.github.io. When I came initially to the project, I also wanted to blend very simple logic into the templates. There's iteration and conditionals, so those also seem like logic to me. However, the point of mustache is that it's simple and intentionally provides only basic operations.

Ok, 'nuff of that. Let's get to two related problems. I've wanted to also implement arrays and pipes, but continually hit a snag. With the custom functions that are used by mo, there are two inputs. The first input is the environment variables and the second is stdin for the template fragment, if one was captured. So, when you run a series of piped commands in the shell, you would do something like this:

cat filename | grep -v "exclude_this_keyword" | tr a-z A-Z

With mo, I would expect the functions to work like that too, but they can also alter the environment. Here's my imaginary template.

{{ myList | sort }}

The part that always stumps me is how I parse the text inside the braces. Right now I assume it's a variable, which I'm sure you have noticed. I'd like to support the piped syntax, but then again I would also want to do something like this:

{{ myList | join " | " | uppercase }}

There should be a way to escape the pipe symbol, but that starts to open a whole can of worms because I'd have to handle quoted strings, escape sequences, and probably start to tokenize everything. And, of course, someone will want to use crazy syntax.

{{ myList | join " }} | {{" | uppercase }}
{{ myList param="one | two | three" }}
{{ functionName --name John\ Doe }}
{{ myList | join '"'\\"'" }}
{{ functionName --param1="{{some_var_or_func}}" }}

That last one is extra tricky because I'd have to build a recursive parser, which means I'd have to rewrite mo entirely. I have not yet decided that investing in a tokenizer for bash-like syntax would be beneficial, but your request is starting to make me reconsider. Having this would allow me to also offer the possibility for partials, as mentioned in #31.

I hope this ramble gives you more insight into the problem. Let me know your thoughts.

[sc0ttj]

Well, I am satisfied that I have implemented it OK now in my project, including the filters working on custom iterator functions/arrays.

My implementation is not something I would try to put into mo... Cos it doesn't take into account the "crazy syntax" examples, nor does it do any real error checking, nor does it limit people to using only the custom filters I defined (might do that next).. And I have no tests inlcluded ;)

But if you are interested, here is the PR in my project with the mo changes I made.

[fidian]

When I was talking about "crazy syntax", I was thinking of times I wanted that same feature in mo. :-)

Thanks for linking to the PR. I am going to leave this issue open. Maybe I will have an idea, build a tokenizer, or will find another way to solve the same issue.

[fidian]

As an update, the content within braces is now parsed and tokenized.

This doesn't mean I'm even thinking of starting the work to allow pipes. It's an intriguing idea but hasn't appeared to have significant demand over the past few years.