[FEATURE] - Add Iterative Mustache Support for Plain Text / JSON Payloads in Bucket-Level Monitors

[m00lav]

Problem:

Let's say for instance we have a simple aggregation where we group by IP address via this DSL Query:

{
  "size": 0,
  "query": {
    "range": {
      "event.ingested": { 
        "gte": "now-2h", 
        "lt":  "now" 
      } 
    } 
  },
  "aggs": {
    "topIPs": {
        "terms": {
          "field": "source.ip",
          "size": 10
        }
    }
  }
}

and the painless trigger returns True for a subset of the IPs via this condition:

{
    "buckets_path": {
        "count_var": "_count"
    },
    "parent_bucket_path": "topIps",
    "script": {
        "source": "params.count_var > 5",
        "lang": "painless"
    },
    "gap_policy": "skip"
}

Then, let's say the configured destination is a custom webhook like FastAPI, where the goal is to post some plain text or JSON payload and have it then preprocess it to perform some task. A simple example payload structure would be set as such:

{"source_ip": SOME_SOURCE_IP, **kwargs}

In mustache, the way to iterate through result within a bucket-level monitor to get the key, for both plain text and structured JSON payloads from what I have seen through testing is by the following:

{{#ctx.results.0.aggregations.topIPs.buckets}}
      {{key}} 
{{/ctx.results.0.aggregations.topIPs.buckets}}

or

{"sourceIP": {{#ctx.results.0.aggregations.topIPs.buckets}} {{key}} {{/ctx.results.0.aggregations.topIPs.buckets}}}

which will essentially return a new-line delimited list of IP addresses. It will not give the current value within the scope of the iteration, which has been frustrating.

What solution would you like? I would like to have N JSON / plaintext payloads being sent to the webhook destination, where N is the number of bucket results, where each text or key-value pair is a value within the scope of the buckets variable being iterated. I have not been able to reproduce that behavior as of yet, and not sure if it is some sort of limitation with mustache.

Alternatives Considered: Using an external destination to preprocess all the data from the bucket-level monitor in OpenSearch.

Other: Happy to help with any additional questions/concerns/clarifications! Thanks so much!

[praveensameneni]

Thank you for creating the request. Looking into it.