Create Packetbeat Integration

[jamiehynds]

Now that Packetbeat agent integration is complete, we need to create a package(s) that can leverage Packetbeat.

I propose that we create a single package that includes support for 'generic' protocols supported by Packetbeat. This approach ensures ease of use when monitoring several protocols on one host. This package will be called 'Network Packet Capture'.

For any protocol that has an existing integration which captures logs/metrics, we add an input to the relevant package. e.g. On the MySQL package configuration we add an option to 'Collect network packets from MySQL hosts'.

For any protocol that is tied to a particular technology/vendor - create a package for that technology, e.g. Apache Cassandra.

Network Packet Capture (note the name change - we won't be reffering to Packetbeat in Fleet)

• AMQP
• ICMP
• DHCP
• DNS
• HTTP
• NFS
• SIP
• Thrift-RPC
• TLS

Existing integrations

• Memcache
• MongoDB
• Mysql
• PostgreSQL
• Redis

New integrations

• Apache Cassandra

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[jamiehynds]

FYI @mukeshelastic & @sorantis - we're planning to create a new Packetbeat (name is TBD) package using the above approach. If you have any thoughts or concerns, please feel free to chime in.

[sorantis]

thanks for the heads up @jamiehynds. The proposed structure makes sense to me, maybe the new package can be called protocols. cc @ruflin, @urso

[ruflin]

Let me propose an alternative option. We recently had discussions around that a single package can contain multiple integrations. So in the case of the nginx package, it contains an integrations for nginx logs and one for nginx metrics. In addition, it could contain an integration for nginx uptime and taking the protocols into account, one to monitor the nginx http and tls traffic.

Now applying this idea to the network traffic package, it could contain an integration for each input type you have specified above. But as we don't support this yet, it is definitively also an option to just start with a single integration that contains each protocol as an input as you proposed.

Could this integration only configured once per policy or could there be multiple instances?

@mostlyjason @mukeshelastic for awareness as this ties into granularity of integrations I think.

[jamiehynds]

Thanks for the insight @ruflin. I'd lean towards a single integration with an input for each protocol for now. We can look at creating individual integrations for MySQL, Mongo, etc in the future, when a single package with multiple integrations is possible.

There could be multiple instances per policy, e.g. using http multiple times with difference settings. @andrewstucki am I correct in saying you included support for multiple instances in a policy in the Packetbeat config?

[mostlyjason]

Yeah this is a great example for integration granularity. The first problem we want to tackle is making these protocols discoverable on the integrations page. Today, if the user types "DNS" in the search box they will not see any results. Exposing each of these separately in the search results will make them easier to find. We had a similar issue with AWS where there is a single package, but we want to expose search results for multiple services inside. This capability is not available yet, so it seems reasonable to do a single integration in the near term.

CC @sorantis

[aarju]

As a security analyst I would like to have a way to deploy packetbeat via Fleet so that it captures N bytes of the packet payload for all packets that match a bpf filter. In Digital Forensics and Incident Response training one of the things they teach when responding to an alert for strange activity on host is if possible to get a full packet capture of traffic being created by a piece of malware.

In the future with this capability and some alerting and actions capabilities we could even get to a point where Security analysts can automate the whole process so that when a critical alert happens on a host we automatically deploy packetbeat to begin capturing traffic.

[andrewkroh]

The initial "Network Packet Capture" integration has been rolled out to 7.15 instances. It's one integration with multiple data streams for each protocol + network flows. As future enhancements we might consider taking advantage of the policy_templates to expose individual protocols in the integrations list (similar to how the AWS package works).