[CDR] Query data using global index pattern

[JordanSh]

Summary

Currently, our dashboard page pulls data exclusively from the latest findings index. With the introduction of third-party (3P) integrations, starting with Wiz, we need to enhance our data query capabilities to include a generic index pattern that fetches data from all 3P integrations, specifically logs-*_latest_misconfigurations_cdr*.

Please refer to the detailed guide in the RFC Combining 3rd party data with native Cloud Security Posture data for comprehensive instructions.

Additionally, the dashboard retrieves data from the scores index to create trend lines. We need to modify this index to incorporate data from 3P latest findings indices as well.

Lastly, all modifications should support a filter parameter passed from the API. This parameter should allow us to selectively fetch data from all latest findings indices (when no filter is passed) or from a specific one based on the passed value.

Definition of Done

• [ ] Update dashboard APIs to include findings from Wiz alongside current latest findings
• [ ] Adjust scores index API to incorporate findings from Wiz alongside current scores
• [ ] Verify the benchmark API querying only native findings.
• [ ] Ensure API supports a filter parameter and retrieves data based on the passed value
• [ ] Identify optional fields and handle potential missing data throughout the data pipeline
• [ ] Implement necessary tests to validate the new functionality
• [ ] Safeguard the dashboard UI to gracefully handle 3P findings, filtering them out if not supported by the UI to prevent crashes

Related Links

• https://github.com/elastic/security-team/issues/9666
• 🎬 Epic
• 🎨 Figma Designs
• 📘 RFC Document
• 🙋 Epic Review Q&A

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[JordanSh]

moving to blocked for now, no point to start working on data pulling until Investigate options to query findings to combine native and 3rd party data is done