[Cloud Posture Integration Network Traffic Integration interfere with Network Traffic Capture integration]

[Danouchka]

Hello

Problem Some visualizations in Network Packet Capture Mongo DB dashboard displays errors such as:

Saved field "resource" of data view "logs-*" is invalid for use with the "Terms" aggregation. Please select a new field.

What I expected to see For instance, Packetbeat Mongo DB dashboard on same data source and same platform displays correct visualisations as you can see in attached screenshots below

Explanation of the Issue Actually, I have installed Cloud Security Posture Integration. resource field is defined

• as an object in cloud security posture integration mapping
• as a keyword in packetbeat and network packet capture integrations mappings

As both network packet capture and cloud security posture dashboards are using logs-* dataview and as resource field is mapped differently there, the conflict prevents the Network Packet Capture Mongo DB dashboard to be displayed properly

[Danouchka]

cc @efd6 @elastic/cloud-security-posture

[tehilashn]

Thank you @Danouchka .

We need to investigate why the logs-* data view is causing this collision. Is there a customer that experienced this problem, or did you encounter it while doing internal experiments?

cc @kfirpeled @eyalkraft

[Danouchka]

Hi @tehilashn, I am experimenting before having customer using it

[Danouchka]

Please get in touch with packetbeat and network packet capture teams to agree on a common solution

[efd6]

Neither of us should be using that. It looks like the NPC pipelines are leaving all the mongo-relevant fields at root; they should be hoisted to their own group. I've done a survey of the other datastreams and pretty much all also appear to be polluting root to a greater or lesser degree. I'll move the polluting fields out to their own ns, but CS should also do the same.

[eyalkraft]

Seems like the searches used for this dashboard could be scoped to logs-network_traffic.* instead of logs-* to avoid the problem. Possibly by including an index pattern in the integration like we do for cloud security.

To my understanding there's no action item from the cloud security side, As this problem will occur with every document that has a non-keyword resource field in any log-* index. This field isn't defined as keyword (or at all) in ECS.

Let me know if I got something wrong or you think otherwise.

Edit: Just saw the new comment. We'll consider moving resource under some namespace. https://github.com/elastic/security-team/issues/6412

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!