Closed opauloh closed 3 months ago
@opauloh should we send null
when the values are missing or not send the fields at all?
@opauloh should we send
null
when the values are missing or not send the fields at all?
@orouz It's better not to send the fields at all, also in favour of establishing a consistent behaviour with other fields (for example, Cloudbeat does not send cloud.Organization.id
if it doesn't have organization information).
When there's no cloud.account.id or cloud.account.name data, cloudbeat should send missing fields instead of empty.
This breaks the dashboards though, no?
it would make more sense if the field where missing rather than empty strings, as that makes difference on how the data is being handled.
In what way? What are we trying to fix here @opauloh?
When there's no cloud.account.id or cloud.account.name data, cloudbeat should send missing fields instead of empty.
This breaks the dashboards though, no?
There is a work in progress to handle missing fields in Kibana for 8.14 +
it would make more sense if the field where missing rather than empty strings, as that makes difference on how the data is being handled.
In what way? What are we trying to fix here @opauloh?
The presence of empty strings in these fields introduces ambiguity in what the data represents. To clarify, when a field is represented as an empty string, it implies that the field exists but holds no value (for example, having a cloud.account.name
equals an empty string, leaves room to interpret that in the organization there is really a Cloud Account where it's name was set to an empty string). On the other hand, the absence of the field altogether indicates that the information is not applicable or available.
Therefore, as part of the solution, we propose that when there is no cloud.account.id
or cloud.account.name
data available, we suggest implementing this adjustment in Cloudbeat's data transmission logic to enhance the integrity and usability of the data generated by our native integration.
@opauloh
in aws / azure we do this: https://github.com/elastic/cloudbeat/blob/392f969a2e812617f4c5fdbcf748c9709a336d9b/internal/dataprovider/providers/cloud/data_provider.go#L63-L64
if GCP did the same, would it solve the issue? (if so, then this would be fixed in https://github.com/elastic/cloudbeat/pull/2085)
i'm a bit unsure because you say we shouldn't send empty fields at all, but that code is from a PR that fixes crashing dashboards due to lack of fields by sending empty string fields if we have no value to send (as in ""
)
in anyway, whatever we decide here: send fields without data as an empty string or not send those fields at all, should be done for all cloud vendors
@opauloh
in aws / azure we do this:
if GCP did the same, would it solve the issue? (if so, then this would be fixed in #2085)
i'm a bit unsure because you say we shouldn't send empty fields at all, but that code is from a PR that fixes crashing dashboards due to lack of fields by sending empty string fields if we have no value to send (as in
""
)
Thanks for sharing the PR with the fix, that was true and necessary for Kibana on versions 8.13 and lower as we were not considering the use case of Findings with organization data that does not relate to a specific cloud account, we now fixed that for 8.14+ on this PR, so this behaviour can now be reverted to use insertIfNotEmpty
.
in anyway, whatever we decide here: send fields without data as an empty string or not send those fields at all, should be done for all cloud vendors
I agree, all cloud vendors can use insertIfNotEmpty
to ensure consistent behaviour.
Verified:
Starts with GCP filter - have findings
Search for cloud.account.name : ""
- no findings
https://github.com/user-attachments/assets/9af4cfcb-5c01-4f4b-9fd7-38a9e0c7ef74
Motivation
While investigating this kibana issue, we found out the GCP Benchmark Rule 2.3 introduced findings data with empty
cloud.account.name
andcloud.account.id
field, while it's expected behaviour to not havecloud.account.*
data since the data is related to a misconfiguration at an organization level not attached to an account, it would make more sense if the field where missing rather than empty strings, as that makes difference on how the data is being handled.Definition of done
cloud.account.id
orcloud.account.name
data, cloudbeat should send missing fields instead of empty.Related tasks/epics
Screenshots
Document example: