Packetbeat flows enhancements

urso commented 7 years ago

Flows support in 5.0 is quite rudimentary, not really taking any network layers (arbitrary timeout) into account and only counting packets and bytes. Plus, there is a parent-child relation between flows and transactions, not yet represented in packetbeat events being published.

List of flow proposed enhancements:

[ ] pass active flow to packet analyzer:
- [ ] add protocol specific stats to flows (e.g. number of transactions with success/failure/dropped due to packet loss/timed out)
- [ ] add flow id to protocol transaction events to establish some relationship between flows and transaction
[ ] add support to tie flow lifetime to connection status:
- [ ] Flows for TCP/SCTP streams do not timeout while TCP connection is active
- [ ] Protocols on top of UDP managing connection state should be able to disable flow timeout
- [ ] Stop flow if connection is closed or connection attempt failed:
- [ ] for TCP stop flow on RST or normal TCP shutdown
- [ ] consider ICMP messages if TCP/UDP port is not reachable
- [ ] timeout flow in case of handshake being incomplete and no data being send in either direction
- [ ] add indicator (string?) to indicate the reason a flow has ended
- [ ] add indicator if connection has been good (established TCP connection), in case it ended forcefully
- [ ] add indicator for flow starting to capture an older TCP connection (not having seen the connection attempt itself, as packetbeat was started after)
[ ] report TCP level stats:
- [ ] count flags usage (SYN, RST, FIN, PUSH)
- [ ] report bytes lost due to packet-loss
- [ ] report packets fully/partially resend (number of packets and bytes)

tbragin commented 6 years ago

@urso @adriansr Does this pull also partially address this issue? https://github.com/elastic/beats/pull/5476

urso commented 6 years ago

@tbragin No. Flows support and integrating flows into application protocols is about collecting metrics only.

ghost commented 6 years ago

If using Packetbeat and not interested in any of the supported protocols can we have a generic "TCP" option where we can specify a list of ports to be able to filter on specific traffic flows before the analysis stage.

urso commented 6 years ago

@london2016 this can already be done by configuring your custom packet filter in the device configs. Please checkout the forums if you need any help.

q2dg commented 5 years ago

Well...version 7.0 is here and nothing has been resolved about this issue yet...

Umarhayat3 commented 5 years ago

PB version 7.3.1 has been released but this issue is still there and many are facing problems due to lack of information in debugging mode. This flow enhancement is necessary for troubleshooting.

elasticmachine commented 4 years ago

Pinging @elastic/integrations-services (Team:Services)

faec commented 4 years ago

Adding this to triage because we're still seeing requests for ways to measure tcp connection drops in particular and it looks like this fell off the radar for a while.

botelastic[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

zez3 commented 3 years ago

would be a nice addition

botelastic[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

zez3 commented 2 years ago

Part of the proposed enhancements would still be useful to have. @jamiehynds Would it be possible to take some enhancements on the roadmap?

wangxin688 commented 2 years ago

It would be a very nice feature to support translation layer for TCP/UDP, which will help measure network performance, especially for forwording devices(use linux as a gateway, running routing, NAT, and firewall function)

elasticmachine commented 2 years ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

botelastic[bot] commented 1 year ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

willemdh commented 1 year ago

.

elasticmachine commented 10 months ago

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

gaby commented 7 months ago

This issue still work in progress after 7 years?

elastic / beats

Packetbeat flows enhancements #3444