search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.17k stars 3.49k forks source link

Netflow codec does not work with netflow v9 logs #4165

Open akshayu opened 8 years ago

akshayu commented 8 years ago

Please see this thread for details - https://discuss.elastic.co/t/logstash-and-elasticsearch-setup-not-working-for-netflow-logs/27698

maechi commented 8 years ago

Hi I'm trying to install ELK on a FreeBSD to collect Netflow Data from an ASR Router and get the same message as akshayu

{:timestamp=>"2016-01-06T09:05:24.215000+0100", :message=>"Unsupported field", :type=>89, :length=>1, :level=>:warn} {:timestamp=>"2016-01-06T09:05:30.285000+0100", :message=>"No matching template for flow id 260", :level=>:warn} {:timestamp=>"2016-01-06T09:05:33.289000+0100", :message=>"No matching template for flow id 260", :level=>:warn} {:timestamp=>"2016-01-06T09:05:33.290000+0100", :message=>"No matching template for flow id 260", :level=>:warn} {:timestamp=>"2016-01-06T09:05:34.289000+0100", :message=>"No matching template for flow id 260", :level=>:warn}

The Software in use is:

I tried a few different How-To's for setting up the elk-stack. In general this is working. If I add the NGINX logs into logstash, i see them in ES and can add an index pattern and brows the data

As soon is I enable the udp listener in logstash, I get the errors above :(

logstash.conf input { udp { port => 9000 codec => netflow { definitions => "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-2.0.2/lib/logstash/codecs/netflow/netflow.yaml" versions => [9] } } } #filter { #} output { stdout { codec => rubydebug } elasticsearch { index => "logstash-netflow9-%{+YYYY.MM.DD}" hosts => "localhost:9200" } }

ASR Config flow exporter-map Flow-ExporterMap version v9 options interface-table template timeout 30 template data timeout 30 template options timeout 30 ! transport udp 9000 source BVI1 destination 10.20.0.3 vrf NMS ! flow monitor-map Flow-MonitorMap-IPv4 record ipv4 exporter Flow-ExporterMap cache entries 16384 cache timeout active 300 ! sampler-map Flow-SamplerMap random 1 out-of 1 !

Does anybody have a tip for me on how to solve this? If you need more data e.b. a wireshark dump, please let me know.

jdambly commented 8 years ago

I am having similar issues are there any updates to this issue?

andrewvc commented 8 years ago

@maechi @jdambly the problem here is this https://github.com/logstash-plugins/logstash-codec-netflow/blob/master/lib/logstash/codecs/netflow.rb#L173 . I'm not familiar at all with netflow but would be glad to work on a patch. Does anyone in this thread have an idea as to what's wrong with the logic in this code: https://github.com/logstash-plugins/logstash-codec-netflow/blob/master/lib/logstash/codecs/netflow.rb#L130 ?

jdambly commented 8 years ago

here is a pcap of my netflow traffic netflow.pcap.zip

ssi0202 commented 8 years ago

having the exact same issue, is there any movement on this or has a solution been found?

abdalians commented 8 years ago

same issue here..any one found a solution yet?

artemd93 commented 8 years ago

It seems that the only solution for the moment is to collect netflow from asr devices using third party collector, and then convert stuff into readable format. I'm using CSV. Not convenient at all, extra files use extra space, but works though.

Did anyone manage to create a patch?

niempy commented 7 years ago

+1 if this was solved. We are having big demand for this in our company. Netflow version 9 works on CISCO3925 devices. But not on the ASR9K Series which also use v9. Same error message No matching template for flow id 260

regards, mark.

P.S. sometimes these packets due come into ELK. But then only netflow information is this (not the interesting part like in_bytes and such):

netflow.flow_sampler_id 1

netflow.flow_sampler_mode 2

netflow.flow_sampler_random_interval 1,000

netflow.flow_seq_num 81,387,975

netflow.flowset_id 257

tnetflow.sampler_name sm-xyz-vlan tnetflow.scope_system 181969085

netflow.version 9

t severity informational t tags
t type netflow

debugloop commented 7 years ago

Im seeing the same issue with asr9k boxes on Cisco IOS XR Software, Version 6.0.2, exactly as described by @niempy

jorritfolmer commented 7 years ago

Thanks @analogbyte for creating an issue in logstash-plugins/logstash-codec-netflow#57, otherwise I would have (still) missed this one.

I just finished replaying @jdambly's pcap file to my logstash-codec-netflow v3.1.2. It looks good to me, with bytes_in etc, see below.

Perhaps the issue is only with sampled netflow output, instead of full netflow? Could you both send me a .pcap privately with some of your netflow traffic @analogbyte @niempy? I'll take a look at it, my email is in profile.

{
       "netflow" => {
            "output_snmp" => 1,
                 "dst_as" => 36180,
               "dst_mask" => 0,
                "in_pkts" => 1,
          "ipv4_dst_addr" => "10.1.1.14",
                "src_tos" => 0,
         "first_switched" => "2016-02-02T20:54:07.999Z",
             "flowset_id" => 256,
            "l4_src_port" => 161,
          "ipv4_next_hop" => "192.168.199.253",
               "src_mask" => 0,
                "version" => 9,
           "flow_seq_num" => 70603,
          "ipv4_src_addr" => "10.34.7.12",
               "in_bytes" => 105,
               "protocol" => 17,
          "last_switched" => "2016-02-02T20:54:07.999Z",
             "input_snmp" => 25,
              "tcp_flags" => 0,
        "flow_sampler_id" => 0,
            "l4_dst_port" => 33756,
              "direction" => 1,
                 "src_as" => 0
    },
    "@timestamp" => 2016-02-02T20:54:22.000Z,
      "@version" => "1",
          "host" => "172.16.32.201"
}
jorritfolmer commented 7 years ago

This issue can be closed. It has been fixed with commit:

Also closing issues: