While working on the AWS Security Hub integration Service Integration team pointed out a problem with the cloud_security_posture plugin relying on querying _source in Kibana, eg. for data grid queries. It has at least two consequences:
in integrations, it's common to remove constant_keyword fields from the _source to optimize storage. As a result we don't have this data in our data gird. It happened with observer.vendor for example.
in serverless and in ESS 9.0 querying _source has a performance penalty due to the need to recreate the _source from the fields.
We need to stop relying on the _source field for queries and use the fields directly
Definition of done
[ ] _source field is not queried in Kibana plugin cloud_security_posture
Motivation
While working on the AWS Security Hub integration Service Integration team pointed out a problem with the
cloud_security_posture
plugin relying on querying_source
in Kibana, eg. for data grid queries. It has at least two consequences:_source
to optimize storage. As a result we don't have this data in our data gird. It happened withobserver.vendor
for example._source
has a performance penalty due to the need to recreate the _source from the fields.We need to stop relying on the
_source
field for queries and use the fields directlyDefinition of done
_source
field is not queried in Kibana plugincloud_security_posture
Out of scope
Related tasks/epics
_source
docs https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-source-field.html#synthetic-sourceTeam tag
@elastic/kibana-cloud-security-posture