Can't decode flowset - IPFIX from IXIA packet broker

[janniten]

For all general issues, please provide the following details for fast resolution:

• Version: 6.3.2
• Operating System: Redhat
• Config File (if you have sensitive info, please remove it): input { udp { port => "4001" codec => netflow { versions => 10 cache_save_path => "/tmp/netflow_template" include_flowset_id => true cache_ttl => 999999999 } type => "ipfix" } }

Hi, I'm trying to ingest netflow data from IXIA packet broker into elastic, but it seems that the plugin cannot find the template in the traffic. I've aldready verify using a wireshark capture that the template is arriving. Continously I get:

Can't (yet) decode flowset id 256 from observation domain id 0, because no template to decode it with has been received. This message will usually go away after 1 minute.

Netflow is generated with IXIA packet broker.

Regards Ana

[robcowart]

Can you share a PCAP of the flows and template? Most like there are field definitions that need to be added to the definitions file used by the decoder.

[janniten]

Hi Rob, Here a capture including a template Thannk you ixflow.zip

[eijaniee]

I have the same issue. The collected flow stored into elasticsearch are not the same captured using wireshark.

[robcowart]

The fix to decode IXIA IPFIX records is in commit https://github.com/logstash-plugins/logstash-codec-netflow/pull/164/commits/5f3377eb068e4431bfd6deda07b19ba7f6485de3

[eijaniee]

So we just need to upload the file (ipfix.yaml) into the correct path, replacing original file right?

[eijaniee]

I already upload the updated file but still get the same result. Netflow decode error also keep appearing on the screen non stop.

Please advice if I miss any other action.

[WARN ] 2018-10-25 18:36:34.605 [<udp.0] netflow - Can't (yet) decode flowset id 271 from observation domain id 1, because no template to decode it with has been received. This message will usually go away after 1 minute. [WARN ] 2018-10-25 18:36:34.609 [<udp.0] netflow - Can't (yet) decode flowset id 259 from observation domain id 5, because no template to decode it with has been received. This message will usually go away after 1 minute. [WARN ] 2018-10-25 18:36:34.609 [<udp.1] netflow - Can't (yet) decode flowset id 256 from observation domain id 0, because no template to decode it with has been received. This message will usually go away after 1 minute.

[robcowart]

Where did you "upload" it to?

The IXIA PCAP that I used did not have flowsets 259 and 271 that you are seeing. You will need to provide a PCAP.

[robcowart]

Also, please make sure you are using the ipfix.yaml fill from the commit...

https://github.com/logstash-plugins/logstash-codec-netflow/blob/5f3377eb068e4431bfd6deda07b19ba7f6485de3/lib/logstash/codecs/netflow/ipfix.yaml

[eijaniee]

Hi,

Here is my pcacp and ipfix.yaml file.

ixia_20181025.pcap.zip

ipfix.yaml.zip

[robcowart]

@eijaniee there are two fields in your flows that were not in the previous sample provided by @janniten. The field IDs are 192 and 193. Do you have any IXIA documentation that describes these (and any other) fields that they might send?

[robcowart]

I figured out those two fields. Please try... https://github.com/logstash-plugins/logstash-codec-netflow/raw/872ba39b99cde0b04b499449ab036a970b2fa727/lib/logstash/codecs/netflow/ipfix.yaml

[eijaniee]

I'm still waiting feedback on the reference from ixia guy.

Beside updating the ipfix codec file, do i need to change anything in my logstash config file? Currently here is my logstash config:

input { udp { port => 1990 codec => netflow { versions => [10] target => ipfix } tags => "ixia" type => ipfix } }

output { if "ixia" in [tags]{ elasticsearch { hosts => ["x.x.x.x:9200", "x.x.x.x:9200", "x.x.x.x:9200", "x.x.x.x:9200"] index => "netflow-ixia-%{+YYYY.MM.dd}" } } }

[robcowart]

@eijaniee as your very simple Logstash pipeline is doing no additional processing of the data, there are no changes required.

If you are interested in more advanced enrichment of the IPFIX data, including out-of-the-box dashboards, you might want to try ElastiFlow.

[jorritfolmer]

Closing, fixed in 4.2.0