Query data using dataview that includes also 3rd party indices for dashboard

[JordanSh]

Summary

Currently, our dashboard page pulls data exclusively from the latest findings index. With the introduction of third-party (3P) integrations, starting with Wiz, we need to enhance our data query capabilities to include a generic index pattern that fetches data from all 3P integrations, specifically logs-*_latest_misconfigurations_cdr.

Please refer to the detailed guide in the RFC Combining 3rd party data with native Cloud Security Posture data for comprehensive instructions.

Additionally, the dashboard retrieves data from the scores index to create trend lines. We need to modify this index to incorporate data from 3P latest findings indices as well.

Lastly, all modifications should support a filter parameter passed from the API. This parameter should allow us to selectively fetch data from all latest findings indices (when no filter is passed) or from a specific one based on the passed value.

Definition of Done

• [ ] Update dashboard APIs to include findings from Wiz alongside current latest findings
• [ ] Adjust scores index API to incorporate findings from Wiz alongside current scores
• [ ] Verify the benchmark API querying only native findings.
• [ ] Ensure API supports a filter parameter and retrieves data based on the passed value
• [ ] Identify optional fields and handle potential missing data throughout the data pipeline
• [ ] Implement necessary tests to validate the new functionality
• [ ] Safeguard the dashboard UI to gracefully handle 3P findings, filtering them out if not supported by the UI to prevent crashes

Related Links

• https://github.com/elastic/security-team/issues/9666
• 🎬 Epic
• 🎨 Figma Designs
• 📘 RFC Document
• 🙋 Epic Review Q&A

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[JordanSh]

moving to blocked for now, no point to start working on data pulling until Investigate options to query findings to combine native and 3rd party data is done

[kfirpeled]

I believe this ticket can be split into all different APIs The scope here to handle all APIs is pretty big wdyt? @JordanSh , @CohenIdo

[CohenIdo]

The scope here to handle all APIs is pretty big

I agree, I will split it to different tasks.

[JordanSh]

I think the main focus here is the dashboard apis which are scores and latest findings, they need to be merged together for serverless.

@CohenIdo i saw that you added benchmark api in one of the edits, do you think its big enough on its own to separate to a different ticket?

[CohenIdo]

@CohenIdo i saw that you added benchmark api in one of the edits, do you think its big enough on its own to separate to a different ticket?

I need to estimate it before I start working on this ticket.