[Fortinet Fortigate Traffic]: Wrong calculation of network.bytes

sbehrends83 commented 3 months ago

Integration Name

Fortinet FortiGate Firewall Logs [fortinet_fortigate]

Dataset Name

fortinet-fortigate.log

Integration Version

1.25.5

Agent Version

8.15.0

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.0

OS Version and Architecture

Ubuntu 22.04

Software/API Version

7.x

Error Message

Hi @all!

We investigated some issue in counting / sum some traffic data over the fields sentbyte and rcvdbyte. We investigated that the fields are more statistics than metrics. The real metrics are written to the fields sentdelta and rcvddelta, which are more usable than the *bytes fields.

Event Original

timestamp="1724322947" devname="myfgt" devid="FG5H0E5899999999" vd="root" date="2024-08-22" time="12:35:47" eventtime="1724322947595266411" tz="+0200" logid="0000000013" type="traffic" subtype="forward" level="notice" srcip="10.100.208.12" srcport="51396" srcintf="srv" srcintfrole="dmz" dstip="10.200.100.112" dstport="47106" dstintf="lan" dstintfrole="lan" srccountry="Reserved" dstcountry="Reserved" sessionid="1660293071" proto="6" action="success" policyid="15" policytype="policy" poluuid="476097ca-7fe5-51ee-1867-9a120be36b89" policyname="push-to-client" service="tcp/47106" trandisp="noop" duration="5" sentbyte="487" rcvdbyte="310" sentpkt="9" rcvdpkt="1" appcat="unscanned" dsthwvendor="Brocade" masterdstmac="ff:ff:ff:ff:ff:aa" dstmac="ff:ff:ff:ff:ff:ff" dstserver="1"

What did you do?

It a snipped from agent-policy

id: udp-fortinet_fortigate-86972f9d-2345-4867-9dcd-ae0b8f5eb9ae name: test-fgt-port-9004 revision: 5 type: udp use_output: default meta: package: name: fortinet_fortigate version: 1.25.5 data_stream: namespace: test package_policy_id: 86972f9d-2345-4867-9dcd-ae0b8f5eb9ae streams:
- id: udp-fortinet_fortigate.log-86972f9d-2345-4867-9dcd-ae0b8f5eb9ae data_stream: dataset: fortinet_fortigate.log type: logs host: '0.0.0.0:9004' tags:
  - preserve_original_event
  - fortinet-fortigate
  - fortinet-firewall
  - forwarded publisher_pipeline.disable_host: true processors:
  - add_fields: target: _temp fields: internal_networks:
    - private

What did you see?

Hte prob

What did you expect to see?

The calculated sum in network.bytes (sendbyte (=source.bytes) + rcvdbyte (=(destination.bytes)) which is done over the ingest pipeline logs-fortinet_fortigate.log-1.25.5 and a script processor:

ctx.network.bytes = ctx.source.bytes + ctx.destination.bytes

is using only sendbyte and rcvdbyte, but should use under conditions senddelta and rcvddelta instead of the byte fields.

Maybe something like this:

[ { "convert": { "field": "fortinet.firewall.sentdelta", "type": "long", "if": "ctx?.fortinet?.firewall?.sentdelta != null", "ignore_failure": true } }, { "convert": { "field": "fortinet.firewall.rcvddelta", "type": "long", "if": "ctx?.fortinet?.firewall?.rcvddelta != null", "ignore_failure": true } }, { "script": { "lang": "painless", "source": "ctx.network.bytesdelta = ctx.fortinet.firewall.sentdelta + ctx.fortinet.firewall.rcvddelta", "if": "ctx?.fortinet?.firewall?.sentdelta != null && ctx?.fortinet?.firewall?.rcvddelta != null", "ignore_failure": true } }, ]

This is just a workaround over the custom pipeline from the integration. Is it possible to integrate this somehow?

Anything else?

No response

elasticmachine commented 3 months ago

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

lillesvin commented 1 month ago

Yes, it appears that sentbyte and rcvdbyte (and consequently ´network.bytes`) are indeed the total amount of bytes exchanged in that session so far. This is hard to do traffic graphs on, since you can't just sum them but instead have to group them by session ID first, take only the last/largest value per session and sum them, which results in some ridiculously heavy queries.

Using sentdelta and rcvddelta, perhaps in a separate field (e.g. network.deltabytes) would make things a lot easier.

elastic / integrations