elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.14k stars 4.91k forks source link

Add support for active flow timeout for packetbeat #37781

Open kvalliyurnatt opened 7 months ago

kvalliyurnatt commented 7 months ago

Describe the enhancement: Currently packetbeat kills flow when the time.now - time_since_last_traffic > timeout. There are scenarios where we would want more frequent final flows reported vs intermediate flows. To support that we would want something like an active flow timeout that would kill the flow when we hit the timeout irrespective of when the last traffic was seen on the flow. We also need to ensure we the flow duration is at least equal to or greater than the timeout as well so that we don't have flows of very small durations.

Describe a specific use case for the enhancement or feature:

This feature will be useful mainly when we have two services that have an open connection and constantly exchange traffic. In such a scenario the current implementation would never kill the flow and we would only have intermediate flows. So if we want to say calculate the total bytes transferred for a specific time window between the services the calculation is much easier if we just had a bunch flow.final records. We would just have to sum the total bytes for all final flow records for that time period. With intermediate flow records, which have running totals of the total bytes, we will have to do some additional calculations to figure out the total bytes transferred during a time period, which adds some complexity.

elasticmachine commented 7 months ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh commented 7 months ago

With adding more reasons for flows to terminate it would be good if there was some tagging in the events (like an event.reason) that indicated the reason for a final==true flow. I think IPFIX sets a good example of this

https://www.iana.org/assignments/ipfix/ipfix.xhtml#ipfix-flow-end-reason

Name Description
idle timeout The Flow was terminated because it was considered to be idle.
active timeout The Flow was terminated for reporting purposes while it was still active, for example, after the maximum lifetime of unreported Flows was reached.
end of Flow detected The Flow was terminated because the Metering Process detected signals indicating the end of the Flow, for example, the TCP FIN flag.
forced end The Flow was terminated because of some external event, for example, a shutdown of the Metering Process initiated by a network management application.
lack of resources The Flow was terminated because of lack of resources available to the Metering Process and/or the Exporting Process.