[Milestone 1] Add datastream.namespace field to the findings scoring index

[opauloh]

Motivation

As part of adding namespace support in CSP integrations in Kibana, we need to update the CSP scoring index (logs-cloud_security_posture.scores-default) to include the datastream.namespace field.

This will enable the filtering of findings data based on user roles to the Findings Misconfigurations DataView

Definition of done

• [ ] datastream.namespace field is added to the CSP scoring index

Out of scope

• Scoring query will be performed in another ticket
• Vulnerabilities

Related tasks/epics

Namespace docs Epic

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[kfirpeled]

This makes me think maybe this index should be hidden from our users I don't see any value in exposing that index for them wdyt?

[opauloh]

This makes me think maybe this index should be hidden from our users I don't see any value in exposing that index for them wdyt?

I believe this can be a good option given the following conditions:

• The scoring query remains performed on the backend and protected by the checkings to access the Findings misconfigurations Dataview.
• A query is made to the Misconfigurations Dataview with the logged user's permission to know what namespaces the current user can access.
• Then the scores query can be performed using internal user permission as it will be passing as filters to the namespaces that the current user can access.

[maxcold]

@kfirpeled @opauloh here is the case when user had benefit from the scores index exposed to them to build custom dashboards https://discuss.elastic.co/t/azure-cspm-multiple-questions/355534 . The question is if we want to support that or not. While it's nice to have this option, we can't guarantee that we won't break these custom dashboards in future versions. I guess exposing it to users kind of says that we treat it as a public API, while internally we are not. Probably we need to align here: either continue treating it as an internal index and don't expose it or continue exposing it and start treating it as a public interface

[opauloh]

@kfirpeled @opauloh here is the case when user had benefit from the scores index exposed to them to build custom dashboards discuss.elastic.co/t/azure-cspm-multiple-questions/355534 . The question is if we want to support that or not. While it's nice to have this option, we can't guarantee that we won't break these custom dashboards in future versions. I guess exposing it to users kind of says that we treat it as a public API, while internally we are not. Probably we need to align here: either continue treating it as an internal index and don't expose it or continue exposing it and start treating it as a public interface

I am actually thinking we should make the scores index hidden from the users as if we have it exposed users will likely expect that's a public index and it won't break, when in fact we might have to introduce break changes in the future, also users will have very different needs.

Then I believe our recommendation for users who want to have the trendline score for custom dashboards should be to configure Watchers to run a query against the index pattern which is public-facing on the desired schedule.