Add Transform layer to the Monitors

[qwinkler]

Is your feature request related to a problem? Please describe. The transform feature is useful when you want to modify your data before you will be able to send an alert. At the moment you can transform your data using the Trigger section, but this looks more like a bug than a feature.

Area of application:

• If you are using the thresholds and you provide an ability to override them. Let's say it will be a Hosts High Memory Usage Monitor. We can set the default thresholds in the params (it is supported as well via API), i.e. 80, and the overrides values for special hosts, i.e. {"overrides": {"my_host_01": {"value": 90}}}. We want to "drop" the hosts that missed a condition from a payload. For other hosts, we want to make some changes (next point).
• Change the "raw" data that we took from ES. Update the decimal format (using DecimalFormat from 1.00000009 to 1.00), generate dashboards/logs URLs;
• Modify data structure. If you are using PagerDuty/Opsgenie the payload has some format. It is much easier to manipulate the payload using painless instead of mustache.

The current workaround as I wrote earlier is to use the Trigger section. For this, add the Sample web logs data and create the following monitor:

POST _plugins/_alerting/monitors

```json { "name": "BytesOverThreshold12H", "type": "monitor", "monitor_type": "query_level_monitor", "enabled": false, "schedule": { "period": { "unit": "MINUTES", "interval": 1 } }, "inputs": [ { "search": { "indices": [ "opensearch_dashboards_sample_data_logs" ], "query": { "size": 0, "query": { "constant_score": { "filter": { "bool": { "must": [ { "range": { "@timestamp": { "from": "now-12h/m", "to": "now/m", "include_lower": true, "include_upper": false, "boost": 1 } } } ], "adjust_pure_negative": true, "boost": 1 } }, "boost": 1 } }, "aggregations": { "dest_country": { "terms": { "field": "geo.dest", "size": 100, "min_doc_count": 1, "shard_min_doc_count": 0, "show_term_doc_count_error": false, "order": [ { "_count": "desc" }, { "_key": "asc" } ] }, "aggregations": { "sum_total": { "sum": { "field": "bytes" } } } } } } } } ], "triggers": [ { "query_level_trigger": { "id": "oCke2n8BLXodUhHwOg-G", "name": "mediumThreshold", "severity": "3", "condition": { "script": { "source": "String kibanaUrl = \"https://localhost:5601/app/discover#/?_g=(filters:!(),query:(language:kuery,query:''),\"\n + \"refreshInterval:(pause:!t,value:0),time:(from:now-12h,to:now))&_a=(columns:!(bytes,message),\"\n + \"filters:!(),index:'90943e30-9a47-11e8-b64d-95841ca0b247',interval:auto,query:(language:kuery,\"\n + \"query:'geo.dest:%%20%%22%s%%22'),sort:!())\";\nList condition = new ArrayList();\nfor (Map i : ctx.results[0].aggregations.dest_country.buckets) {\n if (i.sum_total.value > 25000) {\n String url = String.format(kibanaUrl, new def[]{i.key});\n condition.add(['country': i.key, 'sum': i.sum_total.value, 'kibana_url': url]);\n }\n}\nMap conditionMap = new HashMap();\nconditionMap.put('size', condition.size());\nconditionMap.put('results', condition);\nctx.results[0].condition = conditionMap;\nreturn condition.size() > 0;", "lang": "painless" } }, "actions": [ { "id": "zCkp2n8BLXodUhHwSQ-R", "name": "errorsToSlack", "destination_id": "pyke2n8BLXodUhHwtQ8m", "message_template": { "source": "{{#ctx.results}}\nSize: {{condition.size}}\n{{#condition.results}}\n- Country: {{country}}, Sum bytes: {{sum}} Logs URL: <{{kibana_url}}|{{country}} Logs URL>\n{{/condition.results}}\n{{/ctx.results}}", "lang": "mustache" }, "throttle_enabled": false, "subject_template": { "source": "[P3] Bytes in 12H over threshold", "lang": "mustache" } } ] } } ], "ui_metadata": { "schedule": { "timezone": null, "frequency": "interval", "period": { "unit": "MINUTES", "interval": 1 }, "daily": 0, "weekly": { "tue": false, "wed": false, "thur": false, "sat": false, "fri": false, "mon": false, "sun": false }, "monthly": { "type": "day", "day": 1 }, "cronExpression": "0 */1 * * *" }, "search": { "searchType": "query", "timeField": "timestamp", "aggregations": [], "groupBy": [], "bucketValue": 1, "bucketUnitOfTime": "h", "where": { "fieldName": [], "fieldRangeEnd": 0, "fieldRangeStart": 0, "fieldValue": "", "operator": "is" } }, "monitor_type": "query_level_monitor" } } ```

Here is what the test message generated via UI looks like:

[P3] Bytes in 12H over the threshold
Size:

Here is what the real alert looks like:

[P3] Bytes in 12H over the threshold
Size: 3
- Country: CN, Sum bytes: 96085.0 Logs URL: [CN Logs URL](https://localhost:5601/app/discover#/?_g=(filters:!(),query:(language:kuery,query:''),refreshInterval:(pause:!t,value:0),time:(from:now-12h,to:now))&_a=(columns:!(bytes,message),filters:!(),index:'90943e30-9a47-11e8-b64d-95841ca0b247',interval:auto,query:(language:kuery,query:'geo.dest:%20%22CN%22'),sort:!()))
- Country: IN, Sum bytes: 28766.0 Logs URL: [IN Logs URL](https://localhost:5601/app/discover#/?_g=(filters:!(),query:(language:kuery,query:''),refreshInterval:(pause:!t,value:0),time:(from:now-12h,to:now))&_a=(columns:!(bytes,message),filters:!(),index:'90943e30-9a47-11e8-b64d-95841ca0b247',interval:auto,query:(language:kuery,query:'geo.dest:%20%22IN%22'),sort:!()))
- Country: US, Sum bytes: 28600.0 Logs URL: [US Logs URL](https://localhost:5601/app/discover#/?_g=(filters:!(),query:(language:kuery,query:''),refreshInterval:(pause:!t,value:0),time:(from:now-12h,to:now))&_a=(columns:!(bytes,message),filters:!(),index:'90943e30-9a47-11e8-b64d-95841ca0b247',interval:auto,query:(language:kuery,query:'geo.dest:%20%22US%22'),sort:!()))

Describe the solution you'd like The new Transform section that allows to update our payload before executing the Actions section. So, the final chain of the Monitor will look like: Schedule -> Query OpenSearch data -> Triggers[] -> Transform -> Actions[]

Describe alternatives you've considered

• Use the Trigger section without the ability to test your Monitor;
• ES Watcher Transform;

[rayangou]

+ 1

[guiguan]

Agree. Mustache is pretty useless and very frustrating to render an insightful alert message