RIverbed Netflow support (Was: Can’t yet decode flow set id)

venki522 commented 6 years ago

Version:6.3
Operating System:red hat7.1
Config File (if you have sensitive info, please remove it):
Sample Data:

[2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 300 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute. [2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 300 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute. [2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 300 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute. [2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 302 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute. [2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 301 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute. [2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 300 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute.

Steps to Reproduce:

venki522 commented 6 years ago

We r running elastiflow on elk 6.3 and the logstash-plain.log has all the above messages and it is filling our disk space..could you help us to figure out what is the issue and why we are seeing these messages

joriws commented 6 years ago

Your router is not sending template information (at all or often enough) required to decode flow (v9 or v10) packet at elastiflow-side.

On elasticflow's config file 10_input...conf enable template cache to see what you receive and also to get faster by elastiflow restart to normal decoding.

But. ELK's netflow template decoding beneath elastiflow is working against netflow standard which will affect you if you have multiple sources of flows so I recommend going directly with other pkgs like vflow or pmacct to check if those suit your needs.

ti 3. heinäk. 2018 klo 9.27 venki522 notifications@github.com kirjoitti:

We r running elastiflow on elk 6.3 and the logstash-plain.log has all the above messages and it is filling our disk space..could you help us to figure out what is the issue and why we are seeing these messages

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/logstash-plugins/logstash-codec-netflow/issues/147#issuecomment-402039761, or mute the thread https://github.com/notifications/unsubscribe-auth/AFV-9HpxUakfog65kj0ZhHuJ-RIXM-itks5uCxzEgaJpZM4VAbF1 .

jorritfolmer commented 6 years ago

Can you check the changelog to see if perhaps this was already fixed in a newer version of the codec? If not: can you post a pcap of your netflow traffic so I can look at it?

robcowart commented 6 years ago

@jorritfolmer I have the PCAP in question here. It looks like there are unknown fields...

I will check all of the templates in the PCAP and workout what needs to be added to netflow.yml.

robcowart commented 6 years ago

These are the IDs that need to be identified, and what I could find online (lengths are from templates).

100 - DEPRECATED (was className) - 1 Byte 102 - layer2packetSectionOffset - 4 Bytes 103 - layer2packetSectionSize - 4 Bytes 104 - layer2packetSectionData - 2 Bytes 105 - Reserved for future use by cisco - 2 Bytes 106 - Reserved for future use by cisco - 4 Bytes 107 - Reserved for future use by cisco - 2 Bytes 108 - Reserved for future use by cisco - 4 Bytes 109 - Reserved for future use by cisco - 4 Bytes 110 - Reserved for future use by cisco - 4 Bytes 111 - Reserved for future use by cisco - 1 Byte 51130 - I couldn't find anything for this ID - 1 Byte

@jorritfolmer if you have any info on these fields it would be helpful. I assume you don't or it would already be in the codec, but I wanted to double check.

robcowart commented 6 years ago

BTW, as I have researched this, I also searched through the implementations of other netflow projects here on GitHub, and I am convinced that the logstash netflow codec has the most comprehensive support for various vendor-specific fields.

I did find that nfdump had a few NProbe fields that aren't supported here. I was going to submit a PR to include them, but I noticed that they don't seem consistent with the NProbe docs, so I will need to do some testing first.

Other than that one exception, there is nothing supported by other projects (regardless of language) that isn't supported here. Great work @jorritfolmer !

robcowart commented 6 years ago

I figured out these are from Riverbed Cascade Sensor. More info when I get it.

venki522 commented 6 years ago

Hi James..

If u see in the above logs the we don’t see the IP address of source for which template is not getting decoded..is it possible for future release to add that feature where we can see the IP address if source in the log

jorritfolmer commented 6 years ago

Current status as I understand it:

Rob has a pcap (needed for our rspec tests)
but we lack documentation to properly name the Netflow fields

robcowart commented 6 years ago

Actually I now have the flows decoded properly by adding the needed fields to netflow.yml.

The question I have is whether you want to include them in the codec, as they overlap with an area that is used by other fields, or is "reserved"? I just worry about conflicts here. Of course this is alway a challenge with Netflow, which IPFIX handles with enterprises.

Because of concerns related to conflicting IDs, I have added a new directory in ElastiFlow that contains an alternate field definition files, which the user can optionally use by uncommenting a line in the input. The commit with all of the ElastiFlow changes is here: https://github.com/robcowart/elastiflow/commit/1399df87682ef4037c3fafef91bd9f28a472e09e

This is what I have added to netflow.yml:

100:
- :uint8
- :rb_passthru_reason
101:
- :uint8
- :rb_wan_visibility
102:
- :ip4_addr
- :rb_cfe_inpath_addr
103:
- :ip4_addr
- :rb_sfe_inpath_addr
104:
- :uint16
- :rb_cfe_tcp_port
105:
- :uint16
- :rb_sfe_tcp_port
106:
- :ip4_addr
- :rb_outer_inpath_addr
107:
- :uint16
- :rb_outer_tcp_port
108:
- :uint32
- :rb_retrans_pkts
109:
- :uint32
- :rb_retrans_bytes
110:
- :uint32
- :rb_tcp_conn_rtt
111:
- :uint8
- :rb_fe_type

51130:
- :uint8
- :rb_unk_51130

Note the name of that last one. It still remains a mystery, but at least it is generically handled this way.

If you are not concerned about the potential ID conflicts, I can submit a PR with the needed changes.

jorritfolmer commented 6 years ago

I like your definitions directory solution! For the time being I'll mark Riverbed unsupported in the codec documentation, pointing to Elastiflow for a workaround.

jorritfolmer commented 6 years ago

Docs updated

lucabelluccini commented 6 years ago

Hello @jorritfolmer, I'm facing some issue with a Riverbed appliance (a WAN Optimizer) sending a template as the following:

FlowSet 5 [id=0] (Data Template): 304
    FlowSet Id: Data Template (V9) (0)
    FlowSet Length: 116
    Template (Id = 304, Count = 27)
        Template Id: 304
        Field Count: 27
        Field (1/27): IP_SRC_ADDR
        Field (2/27): IP_DST_ADDR
        Field (3/27): IP_NEXT_HOP
        Field (4/27): PKTS
        Field (5/27): BYTES
        Field (6/27): FIRST_SWITCHED
        Field (7/27): LAST_SWITCHED
        Field (8/27): L4_SRC_PORT
        Field (9/27): L4_DST_PORT
        Field (10/27): INPUT_SNMP
        Field (11/27): OUTPUT_SNMP
        Field (12/27): TCP_FLAGS
        Field (13/27): PROTOCOL
        Field (14/27): IP_TOS
        Field (15/27): DIRECTION
        Field (16/27): SRC_VLAN
        Field (17/27): IP TTL MINIMUM
        Field (18/27): IP TTL MAXIMUM
        Field (19/27): Unknown(51103)
            Type: Unknown (51103)
            Length: 4
        Field (20/27): Unknown(51102)
            Type: Unknown (51102)
            Length: 4
        Field (21/27): Unknown(51105)
            Type: Unknown (51105)
            Length: 2
        Field (22/27): Unknown(51104)
            Type: Unknown (51104)
            Length: 2
        Field (23/27): Unknown(51110)
            Type: Unknown (51110)
            Length: 4
        Field (24/27): Unknown(51108)
            Type: Unknown (51108)
            Length: 4
        Field (25/27): Unknown(51109)
            Type: Unknown (51109)
            Length: 4
        Field (26/27): Unknown(51111)
            Type: Unknown (51111)
            Length: 1
        Field (27/27): Unknown(51101)
            Type: Unknown (51101)
            Length: 1

Do we expect to add those definitions in the codec or we should apply the Elastiflow workaround?

Best regards, Luca

jorritfolmer commented 6 years ago

There are a number of issues with the Riverbed Netflow:

they use fields that are already taken or reserved (id 100-110)
they use fields that we lack field names for (51102-51111 in your wireshark )

If you can provide documentation or other clues as to how we should name the 51102-51111 fields, I'd be happy to include them in the codec.

jorritfolmer commented 5 years ago

Any update on the fields (51102-51111)?

hyberdk commented 5 years ago

Any update on the fields (51102-51111)?

Hi Jorrit,

I think I have the same problem with my Riverbed Steelhead appliances. I took at look at the Riverbed knowlege base, and found this:

Type	Description	Length	Values
51100	Pass-through reason	1	Unknown
51101	WAN visibility	1	0: correct addressing1: port transparency2: full transparency
51102	CFE in-path IP address	4
51103	SFE in-path IP address	4
51104	CFE TCP port	2
51105	SFE TCP port	2	Typically 7800,or7810
51106	Outer in-path IP address	4
51107	Outer TCP port number	2
51108	Retransmitted packets	4	0 or greater
51109	Retransmitted bytes	4	0 or greater
51110	TCP connection RTT	4
51111	FE_TYPE	1	1: CFE, 2: SFE

Now a little explanation for non-Riverbed geeks.

SFE = server file engine (aka server-side Steelhead)
CFE = client file engine (aka client-side Steelhead)
Riverbeds NAT optimized traffic so when they talk about "outer" they are referring to the traffic "as-is" over the WAN (natted), typically SFE and CFE inpath-IP address.
Inner is the "original" client/server IP addresses
TCP connection RTT, is the round-trip-time between the two Riverbeds

Let me know if there are questions, and I will try and see if I can answer them..

Ps. Im pretty new to logstash and elastic search.. So, any pointers on how I can make this work with my installation is much appreciated.. :-)

Esben

logstash-plugins / logstash-codec-netflow

RIverbed Netflow support (Was: Can’t yet decode flow set id) #147