Iterating through arrays is slow

Unfortunately I have to open another performance issue.

I have six arrays with 130 values in each array. The values are only 10 characters long.

I use the following template:

      "xxxx" : [
         {
            "xxxx" : "xxxx",
            "xxxx" : [
               {{#VAR_JSON_CHARACTERSET}}
                  {{MO_COMMA_IF_NOT_FIRST}}
                  {
                     "xxxx" : {{.}},
                     "xxxx" : "{{.}}"
                  }
               {{/VAR_JSON_CHARACTERSET}}
            ]
         },
         {
            "xxxx" : "xxxx",
            "xxxx" : [
               {{#VAR_XML_CHARACTERSET}}
                  {{MO_COMMA_IF_NOT_FIRST}}
                  {
                     "xxxx" : {{.}},
                     "xxxx" : "{{.}}"
                  }
               {{/VAR_XML_CHARACTERSET}}
            ]
         },
         {
            "xxxx" : "xxxx",
            "xxxx" : [
               {{#VAR_HTTP_HEADER_CHARACTERSET}}
                  {{MO_COMMA_IF_NOT_FIRST}}
                  {
                     "xxxx" : {{.}},
                     "xxxx" : "{{.}}"
                  }
               {{/VAR_HTTP_HEADER_CHARACTERSET}}
            ]
         },
         {
            "xxxx" : "xxxx",
            "xxxx" : [
               {{#VAR_PARAMETERS_NAME_CHARACTERSET}}
                  {{MO_COMMA_IF_NOT_FIRST}}
                  {
                     "xxxx" : {{.}},
                     "xxxx" : "{{.}}"
                  }
               {{/VAR_PARAMETERS_NAME_CHARACTERSET}}
            ]
         },
         {
            "xxxx" : "xxxx",
            "xxxx" : [
               {{#VAR_URLS_CHARACTERSET}}
                  {{MO_COMMA_IF_NOT_FIRST}}
                  {
                     "xxxx" : {{.}},
                     "xxxx" : "{{.}}"
                  }
               {{/VAR_URLS_CHARACTERSET}}
            ]
         },
         {
            "xxxx" : "xxxx",
            "xxxx" : [
               {{#VAR_GWT_CHARACTERSET}}
                  {{MO_COMMA_IF_NOT_FIRST}}
                  {
                     "xxxx" : {{.}},
                     "xxxx" : "{{.}}"
                  }
               {{/VAR_GWT_CHARACTERSET}}
            ]
         }
      ]
   }
}

MO_COMMA_IF_NOT_FIRST is from your mo/demo/function-for-building-json.

time mo < ../tmp/test.json > t

real    0m48,688s
user    0m26,592s
sys 0m24,636s

Removing the MO_COMMA_IF_NOT_FIRST call reduces the processing time:

real    0m31,196s
user    0m18,953s
sys 0m13,247s

Removing the second {{.}} in each loop reduces the processing time further:

real    0m19,137s
user    0m12,301s
sys 0m7,368s

Is there a potential to optimize this? MO version is 3.0.5.

Thanks for your help and this great tool!

mo has a significant disadvantage to other, compiled tools. It's relying on Bash's internals to be somewhat fast. There's a point where I can't optimize it further and still have somewhat readable code. (I mention that because profiling shows that even calling functions takes a bit of time, but that makes me cringe to think I could inline functions.)

However, I did identify three areas of concern that impact this example. The first one was where mo checked to see if a function was defined. I added function name caches for hits and misses, speeding this up dramatically. That wasn't good enough, so I also optimized two sections of code where single characters were removed from strings in loops, trimming one end or the other repeatedly until none of the specific characters existed. These have been changed to check the character at the index and change the index to point at a new character, then do one string assignment at the end if necessary.

Originally, the code ran for 30.159 seconds (a single run; I did not run it multiple times). Now the code runs at about 12 seconds for your template, 6 arrays, each array containing 200 values and each value is 10 characters long. Not bad.

Keep up reporting problems, though I don't know if I can promise to find and fix major causes of slowdown. Perhaps there would be some that are triggered with specific inputs, but the general cases are about as optimized as possible without depending on Bash 4+ syntax.

Version 3.0.6 has been released.

Many thanks for this great enhancement! In my local testcase the speed improvement is also enormous. From 110 to 70 seconds. I love this project because I can easily extend the functionality with simple bash functions. This makes it unique in the templating ecosystem.

Just for curiosity: MAC support is the only requirement to keep compatibility with Bash version 3?

Yes, that's correct. I know that one can install Bash 4 on Macs, but it ships with a modified Bash 3.2. If one was going to install software to update a Mac (or a router or whatever), then they might as well install a compiled version of Handlebars as well.

There's zsh since 2019 (Catalina) on Mac, so I suppose whenever the 2018 Macs stop working then I could stop targeting Bash 3.2.

Also, I don't know your comfort level for programming, but there's a couple versions mustache parsers out there that allow you to extend them. 70 seconds seems like a long time to parse a template and I'm glad you're happy with taking over a minute to generate JSON. If you are familiar with other languages, I might recommend taking a look at the other engines and see which ones allow you to extend them because Bash is inherently slow.

If you still love Bash functions, you might get better use out of breaking up your templates, like this example.

#!/usr/bin/env bash

. ./mo

VAR_JSON_CHARACTERSET=( {0000000000..0000000199} )
VAR_XML_CHARACTERSET=( {0000000000..0000000199} )
VAR_HTTP_HEADER_CHARACTERSET=( {0000000000..0000000199} )
VAR_PARAMETERS_NAME_CHARACTERSET=( {0000000000..0000000199} )
VAR_URLS_CHARACTERSET=( {0000000000..0000000199} )
VAR_GWT_CHARACTERSET=( {0000000000..0000000199} )

cat <<EOF
      "xxxx" : [
         {
            "xxxx" : "xxxx",
            "xxxx" : [
EOF 

needComma=false
for characterSet in "${VAR_JSON_CHARACTERSET[@]}"; do
    if [[ "$needComma" == true ]]; then
        echo ","
    fi
    needComma=true
    cat <<EOF
                  {
                     "xxxx" : $characterSet,
                     "xxxx" : "$characterSet"
                  }
EOF
done

cat <<EOF
            ]
         },
         {
            "xxxx" : "xxxx",
            "xxxx" : [
EOF 

# Keep repeating the clauses for each section of JSON.

This is much uglier but significantly faster. In my testing, it completed in about 1/30 of the time (under 1 second). I don't know how often you need to generate the JSON files; if it's very rare then perhaps a minute is fine. However, if you do them frequently, then this approach has merit. Don't get me wrong - I love that you are using mo; I just want your solution to fit your needs.

In case you were curious, I tried splitting the template into chunks and running multiple mo commands for each chunk, but that didn't improve speed at all. In previous versions, that had worked well but I suppose that the later optimizations eliminated the slowness so now running one command or several makes negligible difference.

Thanks for your reply. I also thought about breaking up the templates, but it seems too ugly for me and speed is not the primary goal. It should be only not too slow.

I use MO in self written bash framework for interacting with the F5 REST-API. As the framework is written in bash, it is great that the templating engine is also written in bash because I can easily and spontaneously extend the templating engine with bash functions. With other templating engines I loose this flexibility.

In most cases the speed of MO is sufficient. In interactive mode there are only small templates and bigger templates are only processed by CI/CD pipelines and the MO run is not the longest task in this pipelines.

tests-always-included / mo

Iterating through arrays is slow #73