[BUG] Notification messages to pagerduty fail for per document monitor

[AWSHurneyt]

What is the bug? When a monitor trigger action is configured with the following message body, the notification can be successfully sent to pagerduty for per query monitors, but it fails for per document monitors. Troubleshooting suggests that the custom_details field is not able to be sent for a per document monitor, but this will require further route causing to confirm; as described below, a workaround is to include the custom_details strings in the summary string. In addition, we should check whether the other monitor types are experiencing a similar problem.

{ 
    "event_action": "trigger",
    "payload": {   
        "summary": "{{ctx.trigger.name}}",
        "source": " {{ctx.monitor.name}}",
        "severity": "critical",
        "custom_details": {
            "-Severity" : "{{ctx.trigger.severity}}",
            "-Period start" : "{{ctx.periodStart}}",
            "-Period end": "{{ctx.periodEnd}}",
            “-Involved User": "{{ctx.Username}}"
        }
    }
}

How can one reproduce the bug? Steps to reproduce the behavior:

1. Create a notification channel to pagerduty; e.g.,

   POST /_plugins/_notifications/configs
   {
   "config_id": "pagerduty-test",
   "name": "pagerduty-test",
   "config": {
       "name": "pagerduty-test",
       "description": "",
       "config_type": "webhook",
       "is_enabled": true,
       "webhook": {
         "url": "https://events.pagerduty.com/v2/enqueue",
         "header_params": {
           "X-Routing-Key": "<ROUTING-ID>",
           "Content-Type": "application/json"
         },
         "method": "POST"
       }
     }
   }

2. Create a test index; e.g.,

   POST test-index/_doc 
   {
   "message": "this is a test document",
   "@timestamp": "2024-02-13T20:29:31.734Z"
   }

3. Create a test per document monitor; e.g.,

   /_plugins/_alerting/monitors
   {
   "name": "test-doc-monitor",
   "type": "monitor",
   "monitor_type": "doc_level_monitor",
   "enabled": true,
   "schedule": {
     "period": {
        "unit": "MINUTES",
        "interval": 1
     }
   },
   "inputs": [
     {
        "doc_level_input": {
           "description": "",
           "indices": [
              "test-index"
           ],
           "queries": [
              {
                 "name": "QueryOne",
                 "query": "message:\"this is a test document\"",
                 "tags": []
              }
           ]
        }
     }
   ],
   "triggers": [
     {
        "document_level_trigger": {
           "name": "test-trigger",
           "severity": "1",
           "condition": {
              "script": {
                 "source": "query[name=QueryOne]",
                 "lang": "painless"
              }
           },
           "actions": [
              {
                 "name": "test-action",
                 "destination_id": "pagerduty-test",
                 "message_template": {
                    "source": "\{ \n    "event_action": "trigger", \n    "payload": \{    \n        "summary": "{{ctx.trigger.name}}", \n        "source": " {{ctx.monitor.name}}", \n        "severity": "critical", \n        "custom_details": \{ \n            "-Severity" : "{{ctx.trigger.severity}}", \n            "-Period start" : "{{ctx.periodStart}}", \n            "-Period end": "{{ctx.periodEnd}}", \n            “-Involved User": "{{ctx.Username}}" \n        \} \n    \} \n\}
   ",
                    "lang": "mustache"
                 },
                 "throttle_enabled": false,
                 "subject_template": {
                    "source": "",
                    "lang": "mustache"
                 },
                 "action_execution_policy": {
                    "action_execution_scope": {
                       "per_alert": {
                          "actionable_alerts": []
                       }
                    }
                 }
              }
           ]
        }
     }
   ]
   }

4. Trigger the monitor by ingesting another document to the test index using the command in step2.
5. The notification will not be sent to pagerduty; view the ES logs to see the notification error.
6. The same trigger action should be able to successfully send notifications to pagerduty for a per query monitor.

What is the expected behavior? The different monitor types should support a consistent syntax for notification messages.

Do you have any additional context? A workaround is to include the custom_details strings in the summary string; e.g.,

{ 
    "event_action": "trigger",
    "payload": {   
        "summary": "{{ctx.trigger.name}} \n-Severity" : "{{ctx.trigger.severity}} \n-Period start" : "{{ctx.periodStart}} \n-Period end": "{{ctx.periodEnd}} \n-Involved User": "{{ctx.Username}}",
        "source": " {{ctx.monitor.name}}",
        "severity": "critical"
    }
}