[Epic] [ServiceNow] New Fleet integration

[cpascale43]

Description

We are developing a new ServiceNow integration to replace the current Search connector. This integration should support both default ServiceNow tables and custom tables. As part of the integration, we need to create a custom UI that adds a list of tables with the ability to assign ingest pipelines to each table as needed.

Key Requirements

• One integration that can be configured to collect data from multiple SNOW solutions
• Collect the following types of data via the ServiceNOW REST APIs:
  • Incident data
  • Event data
  • Change data
  • User data
  • User group data
  • Location data
  • Configuration management database (CMDB) configuration item (CI) data
• Collect data from both default and custom ServiceNow tables and views
• The integration allows ingestion of custom table data and provides the option for users to build and assign their own ingest pipelines

Technical Considerations

Default vs. Custom Tables

ServiceNow has a concept of "base" or "core" tables. Base tables in ServiceNow are extendable and multiple related child tables can be created off of one parent table. For example, base tables can be extended to other default child tables:

Original table	Related tables extended from original table
Task [task]	Incident [incident]Problem [problem]Change Request [change_request]
Configuration Item [cmdb_ci]	Application [cmdb_ci_appl]Computer [cmdb_ci_computer]Database [cmdb_ci_database]

Custom tables are any non-SNOW provided table that's been created or installed by or on behalf of a customer. A custom table can be a child table if it extends another table. For example, the following default tables are extendable 1000 times before they are considered (from a billing perspective) custom:

• See: ServiceNow Custom Tables 101

Database views

Database views are table joins, and differ from a custom table:

• See Database Views: Database views are table joins to create a pseudo-table that combines data from each table in the view. You can create a report on this data by adding the database view as a table in a report. Accessing the database view does not require database access.

Required fields - ECS mappings

At a minimum, all fields with user data should be compliant. I.e fields like username, analyst name, or rule name. Custom tables may contain various types of data (e.g., serial numbers, hostnames, usernames) that cannot be mapped to ECS fields in advance.

Example fields that all tables contain regardless of default or custom appear to be as follows. More research is needed to determine if there are additional fields.

• See: ServiceNow Custom Tables 101
  • Class [sys_class_name]
  • Created [sys_created_on]
  • Created by [sys_created_by]
  • Sys_id [sys_id]
  • Updates [sys_mod_count]
  • Updated by [sys_updated_by]
  • Updated [sys_updated_on]

Next Steps

• Engage with the security design team to create custom UI
  • See: https://github.com/elastic/security-team/issues/10597
• Determine the best approach for configuring multiple tables within a single integration
• Explore the use of CEL for data mapping and transformation. Ie, configuration of one instance of the CEL input per table.
• Determine the default fields that require mapping to ECS
• Consider how to handle ECS mapping for custom tables
• Plan for migration from the current Elasticsearch connector to this new integration

Additional Context

• This new integration will replace a temporary solution using custom integration configurations. The temporary solution was implemented to bridge the gap until this new integration is developed.
  • See: https://github.com/elastic/security-team/issues/10513

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[jamiehynds]

Linking duplicate issue: https://github.com/elastic/integrations/issues/11122