[Security Solution] Handle specific fields in the upgrade workflow's UI

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), https://github.com/elastic/kibana/issues/174168 Depends on: https://github.com/elastic/kibana/issues/171520

Summary

Based on the discussions that took place in https://github.com/elastic/kibana/issues/147239, we need to treat different rule fields in different ways in the context of the upgrade workflow.

For each field we must decide if Should the field be manually hidden and never appear as a diff in the Per Field UI?: (only fields part of DiffableAllFields will display)

• ✏️ : should be manually hidden in the UI
  • ➖ : irrelevant, field not returned in the /upgrade/_review (now or after changes marked as needed in this ticket)
  • ❗ : needs to display in the diff UI, but read-only
  • ✅ : needs to display in the diff UI

Field list

Field name	Hidden in UI?
id	➖
rule_source	➖
immutable	➖
version	❗
revision	➖
enabled	➖
execution_summary	➖
alert_suppression*	➖
actions	➖
throttle	➖
response_actions	➖
meta	➖
output_index	➖
namespace	➖
alias_purpose	➖
alias_target_id	➖
outcome	➖
created_at	➖
created_by	➖
updated_at	➖
updated_by	➖
author	❗
license	❗
concurrent_searches (IM Rules)	➖
items_per_search (IM Rules)	➖
rule_id	✅
name	✅
tags	✅
description	✅
severity	✅
severity_mapping	✅
risk_score	✅
risk_score_mapping	✅
references	✅
false_positives	✅
threat	✅
note	✅
setup	✅
related_integrations	✅
required_fields	✅
max_signals	✅
building_block_type	✅
from (rule_schedule)	✅
interval (rule_schedule)	✅
exceptions_list*	✅
rule_name_override	✅
timestamp_override	✅
timestamp_override_fallback_disabled	✅
timeline_id (timeline_template)	✅
timeline_title (timeline_template)	✅
index (data_source)	✅
data_view_id (data_source)	✅
query	✅
language	✅
filters	✅
saved_id	✅
machine_learning_job_id (ML Rules)	✅
anomaly_threshold (ML Rules)	✅
threat_filters (IM Rules)	✅
threat_query (IM Rules)	✅
threat_mapping (IM Rules)	✅
threat_language (IM Rules)	✅
threat_index (IM Rules)	✅
threat_indicator_path (IM Rules)	✅
new_terms_fields (New Terms Rules)	✅
history_window_start (New Terms Rules)	✅

General notes

• All references to "displaying fields diffs in the UI" in the ticket description refer exclusively to the Per Field Diff UI, which will be expanded to display the Three Way Diff upgrade flow. The JSON Diff won't be used anymore for it's current use (an alternative to displaying -more or less- the same data), but the React components we created could be reused to create the Final Preview view that will be used as the final step of the Three-Way Update workflow.

Notes on fields

1. exceptions_list: The Endpoint Security rule includes an exception list value, so this update/customization case needs to be handled. (That's the only prebuilt rule with an exception list as of now)
2. enabled: must be part of Prebuilt Asset schema as some important rules have their default value set to true. But it's not part of the diffing logic anyways, so it will not appear in the UI.
3. The concurrent_searches and items_per_search are part of the diffing logic, but they will have their own specialized diff algorithms that will ensure that the UI never shows them. The /upgrade/_perform endpoint will update to the current version by default, unless specific values for them are passed in the endpoint payload.

Work left over from this ticket

• Handling fields in the /upgrade/_perform endpoint. All of this needs to be done after the refactoring of the endpoint handler is done. As of now, it always installs the full target version, so the changes needed are not possible now. Moving the work to a separate ticket.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

@jpdjere Thank you, I updated the label to 8.16 -- think it should be addessed right after https://github.com/elastic/kibana/issues/171520