No Output / no index' Multiple Outputs

[luis-nagaki]

Using ELK 6.4.x I have elastiflow installed and inputs up, but elasticsearch is not receiving anything.

I have the follow output

output {
  if [type] == "netflow" {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "elastiflow-3.3.0-%{+YYYY.MM.dd}"
    template_name => "elastiflow-3.3.0"
    template_overwrite => true
    manage_template => true
  }
  } else if [type] == "sflow" {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "elastiflow-3.3.0-%{+YYYY.MM.dd}"
    template_name => "elastiflow-3.3.0"
    template_overwrite => true
    manage_template => true
  }
  } else if [type] == "ipfix" {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "elastiflow-3.3.0-%{+YYYY.MM.dd}"
    template_name => "elastiflow-3.3.0"
    template_overwrite => true
    manage_template => true
  }
  } else {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
}

I also tried just if [type] == "netflow" or [type] == "ipfix" etc and nothing. no error msgs so far but not data. If it matters, its data from a Juniper device

All on single node ELK

I am going to try and do [event][type] if statement. I did a stdout on that section for IF netflow and see nothing out. so I dont think its picking up [netflow]

[robcowart]

What results do you get if use ElastiFlow unmodified?

[umarizulkifli]

Please check firewall-cmd allow incoming port for netflow

[luis-nagaki]

I will try unmodified output and report back @robcowart , firewall is off not even installed @xUmaRix

[luis-nagaki]

@robcowart used this which works w/ other output integrations. and still no index i also tried adding a tag to netflow and did an if condition on output for it no index either.

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "elastiflow-3.3.0-%{+YYYY.MM.dd}"
    template_name => "elastiflow-3.3.0"
    template_overwrite => true
    manage_template => true
  }
}

also used id => "output_elasticsearch_single" Not change

[luis-nagaki]

[2018-11-15T11:11:44,735][WARN ][logstash.codecs.netflow  ] Invalid netflow packet received (value '0' not as expected for obj.records[2].flowset_length)
[2018-11-15T11:11:44,854][WARN ][logstash.codecs.netflow  ] Reduced-size encoding for uint32 is larger than uint32 {:field=>[:uint32, :conn_id], :length=>8}

[2018-11-15T11:12:27,151][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 512 from observation domain id 524288, because no template to decode it with has been received. This message will usually go away after 1 minute.

[robcowart]

Basically the Logstash Netflow Codec has trouble with the records it is being sent. You may need to open issue here... https://github.com/logstash-plugins/logstash-codec-netflow

A PCAP of the flow records will likely be necessary to troubleshoot.

[luis-nagaki]

Not sure its a codec issue, as I got it working with this before http://www.routereflector.com/2017/07/elk-as-a-free-netflow-ipfix-collector-and-visualizer/

But then I saw ElastiFlow and fell in love, just trying to get it working. I removed all previous netflow configs

[luis-nagaki]

Any thoughts? Tried out sflow, getting this and no index yet [2018-11-16T13:13:41,140][WARN ][logstash.codecs.sflow ] Unknown record entreprise 0, format 7

[luis-nagaki]

Worked FYI if it was a fresh install