[RAC][Alert Triage][Timeline] Update the Alerts Table and Timeline APIs

[andrew-goldstein]

[RAC][Alert Triage][Timeline] Update the Alerts Table and Timeline APIs

Timeline-based tables provide APIs for configuring columns, customizing actions, and integrating with Kibana APIs for other concerns such as applying filters to the KQL bar at the top of the page, querying for alerts / events, and persisting investigations via saved objects.

The purpose of this issue is to update the public APIs for the Alerts Table and Timeline for consumers outside of the Security Solution.

The updated APIs must continue to provide the following functionality:

• Timeline-based tables are configured via an API to provide a custom set of default columns for different views, e.g. the Alerts table on the Detections page, Rule Details Pages, Events / External alerts tables, and Timeline itself

  • The API enables custom rendering of columns, that, for example, render an external link to an IP reputation site, or open a flyout to view details

• Static, page-level query filters are applied via the API to implement Rule Detail, Host Detail, and Network pages

• The common View details row-level action integrates with a flyout to display alert / event details

• The API allows each table to provide custom row-level actions

  • The Alerts view integrates with the Detections API to update the status of alerts and add rule exceptions

• Cases optionally link directly to Analyzer (Resolver) Graph Views via URLs

• Table-level filtering enables the alerts table to be filtered by alert status

• The API enables two-way integration with the KQL bar and Filters, which are located at the top of most pages

• All fields on a page, including chart legends and table content, are (optionally) rendered via a component that includes a hover menu that affords a consistent experience for:

  • pivoting
  • filtering
  • performing Top N aggregations
  • copying the field + value as KQL snytax to the clipboard

• Integration with the Detections API, enabling bulk-actions for changing the status of alerts (e.g. Open, Closed)

• The API accepts a unique identifier, enabling user-customized columns to persist across page loads via localStorage

• Timeline based tables combine query results from alert and non-alert indexes

• Investigations containing notes and pinned events are persisted via saved objects in the form of saved Timelines

  • An API enables Timelines to be imported / exported as ndjson files

Timeline-based tables are configured via an API to provide a custom set of default columns

Timeline-based tables are configured via an API to provide a custom set of default columns for the:

• Alerts table on the Detections page
• Alerts table on the Rule Details page
• Events and External alerts tables on the Host Details page
• Events and External alerts tables on the Host Page
• External alerts on the Network page
• Timeline itself

The API implemented today enables each page where a Timeline-based table is used to specify a unique set of default columns. The configuration includes:

• The field identifier, e.g. signal.rule.name
• An optional internationalized (i18n) custom column label, e.g. Rule name
• An optional hyperlink for the column value that may, for example, open a flyout containing the details of an alert, or link to an IP reputation site
• The default width of the field

For example, the following array defines the default columns used by the Alerts table on the Detections page:

export const alertsHeaders: ColumnHeaderOptions[] = [
  {
    columnHeaderType: defaultColumnHeaderType,
    id: '@timestamp',
    width: DEFAULT_DATE_COLUMN_MIN_WIDTH + 5,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'signal.rule.name',
    label: i18n.ALERTS_HEADERS_RULE,
    linkField: 'signal.rule.id',
    width: DEFAULT_COLUMN_MIN_WIDTH,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'signal.rule.version',
    label: i18n.ALERTS_HEADERS_VERSION,
    width: 95,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'signal.rule.type',
    label: i18n.ALERTS_HEADERS_METHOD,
    width: 100,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'signal.rule.severity',
    label: i18n.ALERTS_HEADERS_SEVERITY,
    width: 105,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'signal.rule.risk_score',
    label: i18n.ALERTS_HEADERS_RISK_SCORE,
    width: 115,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'event.module',
    linkField: 'rule.reference',
    width: DEFAULT_COLUMN_MIN_WIDTH,
  },
  {
    category: 'event',
    columnHeaderType: defaultColumnHeaderType,
    id: 'event.action',
    type: 'string',
    aggregatable: true,
    width: 140,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'event.category',
    width: 150,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'host.name',
    width: 120,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'user.name',
    width: 120,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'source.ip',
    width: 120,
  },
  {
    columnHeaderType: defaultColumnHeaderType,
    id: 'destination.ip',
    width: 140,
  },
];

The column configuration above is used in the Alerts table on the Detections page below:

The Alerts table on the Detections page

• [X] The API must accept a configuration that specifies the default set of columns
• [X] The API must enable custom rendering of columns, that, for example, render an external link to an IP reputation site, or open a flyout to view details

Static, page-level query filters

Some pages specify static, page-level query filters to some Timeline-based tables.

The following views in the Security Solution are rendered by Timeline-based tables, which pass page-level context to the tables query (via the API):

signal.rule.id filters the Rule Details page's Timeline-based table

A Rule Details page

host.name filters both the Events and External Alerts tables on Host Details pages

A Host Details page

event.kind": "alert" and source.ip: * and destination.ip: * filters the External Alerts table on the Network page

The Network page

• [X] The API must enable static, page-level filters, e.g. signal.rule.id, to be applied to the table's query

The common View details row-level flyout interaction

All Timeline-based tables provide a row level action that, when clicked, displays the details of the alert or event in a flyout:

The View details row-level action opens an Alert Details flyout

• [X] The API must enable consumers to override the default behavior of the View details row-level action

Custom row-level actions

Today, the API enables each Timeline-based table to display a custom set of row-level actions:

Alerts table actions

Events table actions

Timeline contains the superset of all table actions, including some actions that are exclusive to Timeline itself, i.e. the Add notes for this event and Pin event actions:

Timeline's table actions

Additional actions are programmatically provided to each table via an array of JSX.Element[]:

Row-level actions for updating alert status, and adding rule exceptions

The Alerts view integrates with the Detections API to update the status of alerts and add rule exceptions:

• [X] The API must allow table-specific row-level actions to be provided

Cases optionally link directly to Analyzer (Resolver) Graph Views via URLs

Timeline-based tables provide the Analyze Event row-level action shown below:

When users click the Analyze Event action, the (Resolver) process graph view is displayed as a table overlay:

Users may attach cases that directly link to Timeline's Analyzer view, shown below:

• [X] The API must generate URLs that link directly to a saved Timeline's process graph

Table-level filtering

The API enables Timeline-based tables to implement table-level filtering.

For example, the Alerts table provides filtering by alert status:

Table-level filtering

• [X] The API must enable (optional) table-level filtering by alert status

Two-Way integration with the page-level KQL bar and filters

Timeline-based tables use Kibana APIs for two-way integration with the KQL bar and it's Filters, which are located at the top of most pages:

Two-way filtering on the Detections page

An integration with Kibana APIs enables, in the screenshot above:

• The Alerts table to be filtered by the KQL bar at the top of the page

• The Alerts table to be filtered by Filters added to the KQL bar

• Users may, via the hover menu on fields, add filters to the KQL bar at the top of the page

• [X] The API must enable the table to be filtered by a page-level KQL bar and its Filters

• [X] The API must enable filters in the table to be added to page-level Filters

Hover menu actions

All fields on a page, including chart legends and table content, are (optionally) rendered via a component that includes a hover menu that affords a consistent experience for:

• pivoting
• filtering
• performing Top N aggregations
• copying the field + value as KQL snytax to the clipboard

Every field rendered in a Timeline-based table provides the following actions via a hover menu:

• Filter for value (applies a filter to the KQL bar)

• Filter out value (also via the KQL) bar

• Add to timeline investigation

Clicking the button above programmatically drag-and-drops the field to Timeline's query area:

• Show Top field

Clicking the button above displays an an interactive Top N popover, shown below:

The animated gif below shows the Top N popover appearing when user user clicks Show top event.action in the hover menu:

The legend items rendered in the popover also contain a hover menu proffering the same actions.

• [X] The API must enable consumers to specify the indexes included in Top N aggregations

• Copy to clipboard

The Copy to clipboard action formats the field and value using KQL syntax, e.g.:

user.name: "NETWORK SERVICE"

Any field on a page, whether it's rendered in a table or not, can be wrapped in a React component that provides the hover menu functionality. By default, the field is rendered / styled as a draggable.

Some applications may not wish to render fields as draggables. Thus:

• [X] The API must enable fields to be rendered with the common hover menu, with drag-and-drop disabled

Integration with the Detections API, enabling bulk-actions for changing the status of alerts (e.g. Open, Closed)

The alerts table enables integrates with the Detections API, enabling bulk-actions for changing the status of alerts (e.g. Open, Closed):

• [X] The API must provide consumers the option of enabling bulk status updates in the alerts table

User-customized columns are saved to localStorage to persist across page-loads

Users may customize the columns in a table via the Fields Browser, shown below:

The fields browser groups fields from multiple indexes / field mappings into a single view, organized by category, with descriptions. It also provides an affordance for resetting the table to it's specific set of default columns.

• [X] The API must allow consumers to specify the set of indexes that will be used to display fields in the Customize Columns view

Timeline-based tables persist user-customized columns to the browser's localStorage when:

• Columns are added to the table via the Customize Columns view

• Default columns are restored via the Reset fields action in the Customize Columns view

• Columns are re-arranged via drag-and drop:

• Columns are added via the Table view in the Alert / Event details flyout:

• [X] The API must enable a unique ID to be specified for each table instance, which will be used to persist user-customized columns to localStorage

Timeline based tables combine query results from alert and non-alert indexes

Timeline based tables combine query results from alert and non-alert indexes.

• [X] The API must allow consumers the specify the non-alert indexes to query

Today, Timeline-based tables directly query alert indexes, but a new abstraction layer for querying alerts may soon be available.

• [X] If available, Timeline-based tables should consume the new abstraction layer for querying alerts

Timelines persisted as saved objects

While performing an investigation, users may add inline notes to alerts and events in Timeline via Markdown:

Users may also pin events of interest:

When users are ready to share the notes and pinned events gathered during an investigation, they give the untitled timeline a name. The timeline is then in-turn persisted as a saved object in the Kibana space where it was created.

• [X] Timeline saved objects created in the Security Solution before the Timeline is moved to it's own plugin must still be readable, ideally without the need for a saved object migration

• [X] Timelines created in the same Kibana space must be readable by any app that embeds Timeline

• [X] An API must be provided for exporting Timelines as downlodable ndjson files

• [X] An API must be provided for importing Timeline ndjson files into a Kibana space

Acceptance Criteria

• [ ] The API must accept a configuration that specifies the default set of columns
• [ ] The API must enable custom rendering of columns, that, for example, render an external link to an IP reputation site, or open a flyout to view details
• [ ] The API must enable static, page-level filters, e.g. signal.rule.id, to be applied to the table's query
• [ ] The API must enable consumers to override the default behavior of the View details row-level action
• [ ] The API must allow table-specific row-level actions to be provided
• [ ] The API must generate URLs that link directly to a saved Timeline's process graph
• [ ] The API must enable (optional) table-level filtering by alert status
• [ ] The API must enable the table to be filtered by a page-level KQL bar and its Filters
• [ ] The API must enable filters in the table to be added to page-level Filters
• [ ] The API must enable consumers to specify the indexes included in Top N aggregations
• [ ] The API must enable fields to be rendered with the common hover menu, with drag-and-drop disabled
• [ ] The API must provide consumers the option of enabling bulk status updates in the alerts table
• [ ] The API must allow consumers to specify the set of indexes that will be used to display fields in the Customize Columns view
• [ ] The API must enable a unique ID to be specified for each table instance, which will be used to persist user-customized columns to localStorage
• [ ] The API must allow consumers the specify the non-alert indexes to query
• [ ] If available, Timeline-based tables should consume the new abstraction layer for querying alerts
• [ ] Timelines created in the same Kibana space must be readable by any app that embeds Timeline
• [ ] An API must be provided for exporting Timelines as downlodable ndjson files
• [ ] An API must be provided for importing Timeline ndjson files into a Kibana space

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[andrew-goldstein]

https://github.com/elastic/kibana/pull/98241 updates the TGrid API to implement a subset of the EuiDataGridColumn API to render the example column configuration shown in the screenshot below: