[Security Solution] Implement array of scalar values diff algorithm

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), https://github.com/elastic/kibana/issues/174168

Summary

Implement an algorithm for diffing and merging changes in array of scalar values type of fields of detection rules.

Context from the Rule Customization RFC:

• Concrete field diff algorithms by type
• Array of scalar value (strings/numbers) fields

To do

• [x] Step 0
  • [x] Make a proposal which fields should use this algorithm. Get feedback and approval from the team.
• [x] Step 1
  • [x] Implement the algorithm and cover it with unit tests (https://github.com/elastic/kibana/pull/186323)
• [x] Step 2 (separate PR)
  • [x] Apply the algorithm to a few rule fields. It will become used in the upgrade/_review endpoint.
  • [x] Write a test plan covering rule upgrade scenarios for the fields to which the algorithm has been applied.
  • [x] https://github.com/elastic/kibana/pull/186325
  • [x] Add test coverage according to the test plan (integration tests, primarily?).
  • [x] https://github.com/elastic/kibana/pull/187778

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[dplumlee]

Here are the proposed fields that would utilize this diff algorithm:

Common fields

• tags
• references
• author

Threat match fields

• threat_index

New terms fields

• new_terms_fields

[dplumlee]

Resolved by:

• https://github.com/elastic/kibana/pull/186323
• https://github.com/elastic/kibana/pull/186325
• https://github.com/elastic/kibana/pull/187778