search

logstash-plugins / logstash-codec-netflow

Apache License 2.0
78 stars 87 forks source link

Implement RFC6759 (Was: Fortigate application_id not processed correctly) #116

Closed Sjaak01 closed 6 years ago

Sjaak01 commented 6 years ago

Hi,

I've noticed that on FortiOS 5.6.x the application_id field is not processed correctly.

The output shows something like 20:53068615909376 while the correct output should be something like 15921.

This is the information I received from our reseller with regards to a pcap I sent to them:

The HEX in the packet capture that represents the application_id is 00 00 30 44 00 00 3e 31. 3e 31 is the actual application ID and 30 44 is the Fortigate enterprise ID.

Elastiflow, on which the netflow codec is apparently based (?) had the same issue but apparently got fixed. https://github.com/robcowart/elastiflow/releases/tag/v1.2.0 https://github.com/robcowart/elastiflow/issues/14

From my understanding the application_id should be processed as a keyword. However changing the mapping type in /usr/share/logstash/modules/netflow/configuration/elasticsearch/netflow.json to keyword did not solve the problem.

Netflow codec version: 3.7.0 and 3.9.0.

jorritfolmer commented 6 years ago

No, applicationId is not a keyword, but a single field composed of two parts.

The IANA information elements list refers to RFC6759 for the decoding of element 95 (applicationId"). Relevant quote:

This document specifies the applicationId Information Element, which is a single field composed of two parts:

  1. 8 bits of Classification Engine ID. The Classification Engine can be considered as a specific registry for application assignments.
  2. n bits of Selector ID. The Selector ID length varies depending on the Classification Engine ID.
Sjaak01 commented 6 years ago

So what does that effectively mean with regards to the netflow codec and processing the Fortigate application_id?

If required I can provide a packet capture or requires further information from Fortinet.

Sjaak01 commented 6 years ago

Any comment on where the problem might be?

jorritfolmer commented 6 years ago

I'm not sure there is a problem.

You're expecting application id to be 15921. However, according to the RFC standard application_id will always be in "n:m" format, like you're seeing in ES.

Please provide a pcap if I'm misunderstanding.

Sjaak01 commented 6 years ago

Where can I send the pcap files to? Would prefer not uploading them online.

I think I found the problem. On FortiOS 5.2 the application id appears to work, it shows up as 0:xxxx in elastic and the number after 0: matches the fortigate application id database. The packets can also be decoded using wiresharks CFLOW.

However I think something changes with FortiOS 5.6. In elastic the application id now shows up as 20:xxxx (way too long number) that don't match the application id database. the CFLOW filter no longer decodes the application id correctly either, just shows up as 20:48 etc.

I can send you pcaps for both OS versions. Please let me know how I can get the files to you.

jorritfolmer commented 6 years ago

Sure, my email is in my profile.

anacelto commented 6 years ago

What about this issue? I'm interested

jorritfolmer commented 6 years ago

Closing this issue as I haven't yet received any pcaps to look at.

Sjaak01 commented 6 years ago

Apologies for the late reply. I've sent you an email including pcap last Friday.

tabrezgoilkar commented 6 years ago

i am facing the same issues, i have fortigate v5.4.2,

jorritfolmer commented 6 years ago

I'm reopening this issue. I've discovered some minutes ago that logstash-codec-netflow doesn't fully implement rfc6759 to properly decode application_id. See https://github.com/robcowart/elastiflow/issues/100#issuecomment-390624041

jorritfolmer commented 6 years ago

I've implemented support for application_id decoding in branch rfc6759.

jorritfolmer commented 6 years ago

Just released v4.0.0 with support for this.