Inconsistent behavior when rendering JSON arrays in search templates

[Mikep86]

Elasticsearch Version

8.10.3

Installed Plugins

No response

Java Version

bundled

OS Version

Cloud

Problem Description

Search template rendering behavior is inconsistent when trying to render a JSON array in a template.

When using "generic" search templates, JSON arrays cannot be rendered without hacky workarounds due to a trailing comma appearing after the last item in the array.

However, when using search templates with search applications, you can render JSON arrays much more easily due to fixes applied only in the context of search templates used for search applications (see PRs #95200 & #96197).

This inconsistency is a) confusing and b) undocumented. From the perspective of an external user, JSON array rendering auto-magically works with only search applications.

Steps to Reproduce

1. Create a search template:

PUT _scripts/my-search-template
{
  "script": {
    "lang": "mustache",
    "source": """
    {
      "query": {
        "multi_match": {
          "query": "{{query_string}}",
          "fields": [{{#text_fields}}"{{name}}^{{boost}}",{{/text_fields}}]
        }
      }
    }
    """
  }
}

2. Try to render the template:

POST _render/template
{
  "id": "my-search-template",
  "params": {
    "query_string": "my first query",
    "text_fields": [
        {
            "name": "title",
            "boost": 10
        },
        {
            "name": "text",
            "boost": 1
        }
    ]
  }
}

Elasticsearch currently returns a JSON parsing exception due to the generated JSON array containing a trailing comma:

{
  "error": {
    "root_cause": [
      {
        "type": "json_parse_exception",
        "reason": """Unexpected character (']' (code 93)): expected a value
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 6, column: 43]"""
      }
    ],
    "type": "json_parse_exception",
    "reason": """Unexpected character (']' (code 93)): expected a value
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 6, column: 43]""",
    "suppressed": [
      {
        "type": "illegal_state_exception",
        "reason": "Failed to close the XContentBuilder",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unclosed object or array found"
        }
      }
    ]
  },
  "status": 500
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/ent-search-eng (Team:Enterprise Search)

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)