[Alerting][Discuss] allow alerts to create default action parameters

[pmuellr]

Currently, when creating an alert and adding actions to it, a customer will have to fill in all the action parameters manually. We've tried making this a little easier with the index threshold alert, by having that alert create two context variables message and title, which can be used as the message and subject for an email action, and message for a slack action. The user can then enter {{context.message}} for the message body in the action form, or click on the menu item for it in the context variables associated with the field.

It would be nicer to make this even easier. Why require the customer to do anything, if they want to use a default message provided by the alert?

Some additional considerations:

• Different action groups should be able to have different default messages, especially since the "resolved" action group won't have any context variables, just the top-level variables.

• if we want the message to have formatting specific to the action, then somehow the message will have to know what formatting type the action takes - for email, the message should be in markdown, and for slack, the message should be in slack-flavored-markdown.

• some action parameters like documents in the index action aren't strings, but JSON-able objects; is that just another "format type" like markdown/slack?

• at some point, we'll probably want to have message itself be a set of messages - eg, you want to create a GH issue with one action, then send an email in a second action. The email should include a link to the GH issue created. Presumably the GH action will be able to create a formatted piece of text that can be added to the action body, presumably after the main message body for the alert. We've also talked about having "fixed" message snippets at the beginning and/or end of the messages - eg, allow for some kind of legal prefix message for compliance requirements, allow a suffix message that includes a link to the alert details page, etc. I don't think anything with this necessarily impacts having alerts create default action parameters, but would be good to keep in mind while developing this.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

My current thinking on this is to allow alerts (and eventually other objects) to provide a "action parameter provider" object ... somehow. Probably an object passed when scheduling actions, but ... not sure.

The basic idea is that the action parameter provider would look something like this:

interface ActionParameterProvider {
   getParameter(name: string, type: string): any;
}

The email action would call this object twice:

    message = paramProvider.getParameter('message', 'markdown')
    subject = paramProvider.getParameter('title', 'markdown')

The slack action would call this object once:

    message = paramProvider.getParameter('message', 'slack')

We'd have a list of "common parameter values" like message and title, and the action would be responsible for asking for the correct name for the parameter it was getting a value for.

Likewise, we'd have a set of "formats" like 'markdown', 'slack', which alerts could use for very specific formatting requirements.

It seems like this sort of interface would work, I worry about when we actually call it, to ensure the alert can provide the appropriate data - and I think that would be when it schedules actions.

Alternatively, instead of deal with callbacks, we could have an alert create the appropriate formatted objects for all formats ahead of time, before scheduling the actions. We could also have figure out all the types that would be needed, based on actions to be executed, and pass those formats to the alert, so it would know what formats it should generate data for.

Lastly, we'll need a way to allow customers to specify they want to use these default parameters, instead of customizing them, themselves. We'd probably like customers to use the defaults when available, so at a UI level we might not even provide an action parameter form, unless the user specified they wanted to "customize the action parameters" (for actions whose parameters can all be provided as defaults).

At an API level, we could probably get away with allowing null/undefined as the value, indicating that the alert default should be used, as I think currently these sorts of parameters do not allow null/undefined values.

[sorenlouv]

Right now, when a user selects an action they are greeted with a blank text area which in itself can be daunting. If the user just wants to setup a few alerts quickly they shouldn't be forced to spend time composing the perfect message.

Having an existing message will also make it much easier for users to discover the relevant variables. Currently it is hard to find the relevant variables in the dropdown, and afaict there are far too few useful variables, which leads me to my last point:

Having an existing message will force us to think through what a good message is, and what variables we need to make available. It looks like alerting developers haven't done that everywhere. For instance: how should a user compose a message which contains a link back to a given visualisation or solution? This is probably the primary use case for many users (something happened, go look at this) but we're forcing them to figure out how to build the correct links, with the correct variable substitutions and etc. This also opens up a related question: is there currently a variable for kibana hostname?

[pmuellr]

One thing I had forgotten about is the defaultActionMessage mustache template that alertTypes can specify in their definition. See https://github.com/elastic/kibana/issues/78148 for details on how we plan to add this for the index threshold alert type.

One idea riff'ing on that capability, is to expand it a bit more:

• allow more more "fields" than just a "message"; probably a "title" would be appropriate, which could be used as an email subject, and the "title" kind of field for incident management actions; but feels like we could keep this fairly open ended

• allow for different formats - currently the defaultActionMessage will be used filled in as the message body for both slack messages and email messages. But the formatting is different, so the text this is *bold* would render the word bold as bold in Slack, but italic in email (as we process the email body as plain old markdown where two asterisks are needed for "bold"ing, eg **bold**).

This makes it feel like you could perhaps provide an object with different properties, like:

defaultActionFields = {
   message: { markdown: '...', slackMarkdown: '...', plain: '...' },
   title: { markdown: '...', slackMarkdown: '...', plain: '...' },
}

Note that these message and title properties are intended to be general, they don't necessarily map to a specific action parameter. The code in the alert params editor for a particular alert would fetch the appropriate type based on their needs; eg, the email editor would fetch the title.plain property to use as the subject field.

There are some other considerations, especially looking forward, where we'll have things like "bulk" actions (when 10 instances are scheduled for an alert, send 1 email listing the 10 instances, instead of what happens today of sending 10 emails each with 1 instance), etc.

Rather than a big hairy object, we could also consider making this a function. Which makes this similar to what I described in https://github.com/elastic/kibana/issues/66095#issuecomment-626919382, however this would all be client-side, and not require interaction during action execution.

Last thing, we've had customers ask about "templates" for these, which means that they'd like to be able to create their own custom versions, presumably saved in Kibana somewhere. Completely separate issue with other concerns, but something to keep in mind as we develop this.

[pmuellr]

Re-reading this, I've of mind that we should go the "function" route, where the alerts provide a way of returning customized default mustache templates to be set in the UI when a new action form is created. Expanding on the version mentioned above, each alert type would provide an ActionParameterProvider object, which would be used in the action form to return the default value for some fields like message and subject:

interface ActionParameterOptions {
   type?: 'plain' | 'markdown' | 'slack' | 'json';
   actionGroup?: string;
   // later: bulk?: boolean
}

interface ActionParameterProvider {
   getParameter(name: string, options: ActionParameterOptions): string;
}

The email action form would call this object twice:

    message = paramProvider.getParameter('message', {type: 'markdown', actionGroup} )
    subject = paramProvider.getParameter('title', {type: 'plain', actionGroup} )

The slack action would call this object once:

    message = paramProvider.getParameter('message', {type: 'slack', actionGroup} )

The implementation for an alert that just handled message (but should check for the name!) would look like this:

function getParameterProvider(): ActionParameterProvider {
  return {
    getParameter(name: string, options: ActionParameterOptions): string {
      if (options.actionGroup === 'resolved') {
        switch(options.type) {
          case 'plain': return 'alert {{alertName}} resolved'
          case 'markdown': return 'alert **{{alertName}}** resolved'
          case 'slack': return 'alert *{{alertName}}* resolved'
        }
      }

      switch(options.type) {
        case 'markdown': return 'alert **{{alertName}}**  instance {{alertInstanceId}} ...'
        case 'slack': return 'alert *{{alertName}}*  instance {{alertInstanceId}} ...'
      }

      return 'alert {{alertName}} instance {{alertInstanceId}} ...'
    }
  }
}

As simple or as fancy as we want.

Eventually we'll want a way of allowing a user to overwrite something they may have customized, and then gotten into a bad state, with the default message again (via some UI gesture). We'll actually want to have this on the server eventually as well, so we can allow API usage of passing a null or empty string as a message (or other field) in an action param to indicate that the appropriate "default" value should be used. But we don't need to do that just yet, I don't think.

[mikecote]

Moving from 7.12 - Candidates to 7.x - Candidates.

[darnautov]

hey @pmuellr,

I'm currently working on the ML alert type and for our use-case it'd great to have a possibility to alternate the defaultActionMessage based on the current alertParams, e.g. depending on the selected "Result type" we have sligtly different contexts variables, and providing different templates will help the user to set up the action without much effort.

I know that having multiple alert types is the preferred way, but in our case having one type helps to guide the user with configuring the alert based on the job selection, i.e. when selected anomaly detection jobs don't contain influencers configuration, we don't suggest "Influencer" result type.

[pmuellr]

I hadn't thought about also allowing param values to also come into play here, but makes sense to me, I think. The ActionParameterOptions described in https://github.com/elastic/kibana/issues/66095#issuecomment-725203424 could certainly be extended to take into the params as well.

There are likely other properties that would be useful to add to that ActionParameterOptions - the small number in the example is illustrative, not complete.

[mikecote]

Moving from 7.x - Candidates to 8.x - Candidates (Backlog) after the latest 7.x planning session.