search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.48k stars 592 forks source link

Issue with Logstash 6.7 #298

Closed fkocharli closed 5 years ago

fkocharli commented 5 years ago

I have Elastic Stack 6.7 running on the same host. It runs several pipelines to get logs from cisco devices (syslog), vmware (syslog) and windows (winlogbeat). I've installed Elastiflow following installation guide to receive Netflow data. When I start service I have not any errors in logs. It listens 2055 port. With tcpdump I can see that I receive data on that port. But my Kibana has not any data. Can this happen because I have 6.7 version instead of 6.6?

It listens 2055:

[root@elk ~]# ss -tulpna | grep 2055
udp    UNCONN     0      0         *:2055                  *:*                   users:(("java",pid=20427,fd=112))

My /var/log/logstash/logstash-plain.log

[2019-04-25T11:12:54,046][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.1"}
[2019-04-25T11:12:58,395][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"vmware", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-04-25T11:13:01,004][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-25T11:13:02,286][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-25T11:13:02,819][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-25T11:13:02,823][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-04-25T11:13:03,082][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-25T11:13:04,741][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:1514"}
[2019-04-25T11:13:05,022][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"vmware", :thread=>"#<Thread:0x54627e7d run>"}
[2019-04-25T11:13:05,707][INFO ][org.logstash.beats.Server] Starting server on port: 1514
[2019-04-25T11:13:15,589][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"cisco", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-04-25T11:13:16,203][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-25T11:13:16,234][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-25T11:13:16,254][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-25T11:13:16,256][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-04-25T11:13:16,406][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-25T11:13:18,068][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:8514", :ssl_enable=>"false"}
[2019-04-25T11:13:18,090][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"cisco", :thread=>"#<Thread:0x1d1175dd run>"}
[2019-04-25T11:13:18,344][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:8514"}
[2019-04-25T11:13:18,450][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:8514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2019-04-25T11:15:59,744][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"elastiflow", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-04-25T11:16:00,002][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-25T11:16:00,111][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-25T11:16:00,183][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-25T11:16:00,184][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-04-25T11:16:00,256][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"windows", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-04-25T11:16:00,259][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elastiflow/templates/elastiflow.template.json"}
[2019-04-25T11:16:00,286][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-25T11:16:00,342][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-25T11:16:00,487][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-25T11:16:00,553][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-25T11:16:00,581][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-04-25T11:16:00,705][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-25T11:16:00,840][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-04-25T11:16:00,937][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"windows", :thread=>"#<Thread:0x79bc97e3 run>"}
[2019-04-25T11:16:00,957][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-04-25T11:16:01,349][INFO ][logstash.outputs.elasticsearch] type"=>"long"}}}, {"sflow.frame_length_times_sampling_rate"=>{"path_match"=>"sflow.frame_length_times_sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.header_size"=>{"path_match"=>"sflow.header_size", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface"=>{"path_match"=>"sflow.input_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface_format"=>{"path_match"=>"sflow.input_interface_format", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface_value"=>{"path_match"=>"sflow.input_interface_value", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_address_next_hop_router"=>{"path_match"=>"sflow.ip_address_next_hop_router", "mapping"=>{"type"=>"ip"}}}, {"sflow.ip_checksum"=>{"path_match"=>"sflow.ip_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_dscp"=>{"path_match"=>"sflow.ip_dscp", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_ecn"=>{"path_match"=>"sflow.ip_ecn", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_flags"=>{"path_match"=>"sflow.ip_flags", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_fragment_offset"=>{"path_match"=>"sflow.ip_fragment_offset", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_header_length"=>{"path_match"=>"sflow.ip_header_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_identification"=>{"path_match"=>"sflow.ip_identification", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_next_header"=>{"path_match"=>"sflow.ip_next_header", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_options"=>{"path_match"=>"sflow.ip_options", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_packet_length"=>{"path_match"=>"sflow.ip_packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_priority"=>{"path_match"=>"sflow.ip_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_protocol"=>{"path_match"=>"sflow.ip_protocol", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_total_length"=>{"path_match"=>"sflow.ip_total_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_ttl"=>{"path_match"=>"sflow.ip_ttl", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_type"=>{"path_match"=>"sflow.ip_type", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_version"=>{"path_match"=>"sflow.ip_version", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface"=>{"path_match"=>"sflow.output_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_format"=>{"path_match"=>"sflow.output_interface_format", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_value"=>{"path_match"=>"sflow.output_interface_value", "mapping"=>{"type"=>"long"}}}, {"sflow.packet_length"=>{"path_match"=>"sflow.packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.padded"=>{"path_match"=>"sflow.padded", "mapping"=>{"type"=>"long"}}}, {"sflow.protocol"=>{"path_match"=>"sflow.protocol", "mapping"=>{"type"=>"keyword"}}}, {"sflow.protocol_name"=>{"path_match"=>"sflow.protocol_name", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sample_length"=>{"path_match"=>"sflow.sample_length", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_pool"=>{"path_match"=>"sflow.sample_pool", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_seq_number"=>{"path_match"=>"sflow.sample_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.sampling_rate"=>{"path_match"=>"sflow.sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.sequence_number"=>{"path_match"=>"sflow.sequence_number", "mapping"=>{"type"=>"long"}}}, {"sflow.sflow_type"=>{"path_match"=>"sflow.sflow_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sflow_version"=>{"path_match"=>"sflow.sflow_version", "mapping"=>{"type"=>"long"}}}, {"sflow.size_header"=>{"path_match"=>"sflow.size_header", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index"=>{"path_match"=>"sflow.source_id_index", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index_name"=>{"path_match"=>"sflow.source_id_index_name", "mapping"=>{"type"=>"keyword"}}}, {"sflow.source_id_type"=>{"path_match"=>"sflow.source_id_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_ip"=>{"path_match"=>"sflow.src_ip", "mapping"=>{"type"=>"ip"}}}, {"sflow.src_mac"=>{"path_match"=>"sflow.src_mac", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_mask_len"=>{"path_match"=>"sflow.src_mask_len", "mapping"=>{"type"=>"long"}}}, {"sflow.src_port"=>{"path_match"=>"sflow.src_port", "mapping"=>{"type"=>"long"}}}, {"sflow.src_priority"=>{"path_match"=>"sflow.src_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.src_vlan"=>{"path_match"=>"sflow.src_vlan", "mapping"=>{"type"=>"long"}}}, {"sflow.stripped"=>{"path_match"=>"sflow.stripped", "mapping"=>{"type"=>"long"}}}, {"sflow.sub_agent_id"=>{"path_match"=>"sflow.sub_agent_id", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_ack_number"=>{"path_match"=>"sflow.tcp_ack_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_checksum"=>{"path_match"=>"sflow.tcp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_flags"=>{"path_match"=>"sflow.tcp_flags", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_header_length"=>{"path_match"=>"sflow.tcp_header_length", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_is_ack"=>{"path_match"=>"sflow.tcp_is_ack", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_cwr"=>{"path_match"=>"sflow.tcp_is_cwr", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_ecn_echo"=>{"path_match"=>"sflow.tcp_is_ecn_echo", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_fin"=>{"path_match"=>"sflow.tcp_is_fin", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_nonce"=>{"path_match"=>"sflow.tcp_is_nonce", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_push"=>{"path_match"=>"sflow.tcp_is_push", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_reset"=>{"path_match"=>"sflow.tcp_is_reset", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_syn"=>{"path_match"=>"sflow.tcp_is_syn", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_urgent"=>{"path_match"=>"sflow.tcp_is_urgent", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_options"=>{"path_match"=>"sflow.tcp_options", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_reserved"=>{"path_match"=>"sflow.tcp_reserved", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_seq_number"=>{"path_match"=>"sflow.tcp_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_urgent_pointer"=>{"path_match"=>"sflow.tcp_urgent_pointer", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_window_size"=>{"path_match"=>"sflow.tcp_window_size", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_checksum"=>{"path_match"=>"sflow.udp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_length"=>{"path_match"=>"sflow.udp_length", "mapping"=>{"type"=>"long"}}}, {"sflow.uptime_in_ms"=>{"path_match"=>"sflow.uptime_in_ms", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_cfi"=>{"path_match"=>"sflow.vlan_cfi", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_id"=>{"path_match"=>"sflow.vlan_id", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_priority"=>{"path_match"=>"sflow.vlan_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_type"=>{"path_match"=>"sflow.vlan_type", "mapping"=>{"type"=>"long"}}}, {"string_fields"=>{"mapping"=>{"type"=>"keyword"}, "match_mapping_type"=>"string", "match"=>"*"}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "event"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"host"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}, "flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"application"=>{"type"=>"keyword"}, "autonomous_system"=>{"type"=>"keyword"}, "bgp_next_hop"=>{"type"=>"ip"}, "bgp_valid_state"=>{"type"=>"long"}, "bytes"=>{"type"=>"long"}, "city"=>{"type"=>"keyword"}, "client_addr"=>{"type"=>"ip"}, "client_asn"=>{"type"=>"long"}, "client_autonomous_system"=>{"type"=>"keyword"}, "client_city"=>{"type"=>"keyword"}, "client_country"=>{"type"=>"keyword"}, "client_country_code"=>{"type"=>"keyword"}, "client_geo_location"=>{"type"=>"geo_point"}, "client_hostname"=>{"type"=>"keyword"}, "client_rep_tags"=>{"type"=>"keyword"}, "country"=>{"type"=>"keyword"}, "country_code"=>{"type"=>"keyword"}, "direction"=>{"type"=>"keyword"}, "dst_addr"=>{"type"=>"ip"}, "dst_addr_trans"=>{"type"=>"ip"}, "dst_asn"=>{"type"=>"long"}, "dst_autonomous_system"=>{"type"=>"keyword"}, "dst_city"=>{"type"=>"keyword"}, "dst_country"=>{"type"=>"keyword"}, "dst_country_code"=>{"type"=>"keyword"}, "dst_geo_location"=>{"type"=>"geo_point"}, "dst_hostname"=>{"type"=>"keyword"}, "dst_mac"=>{"type"=>"keyword"}, "dst_mask_len"=>{"type"=>"long"}, "dst_port"=>{"type"=>"long"}, "dst_port_trans"=>{"type"=>"long"}, "dst_port_name"=>{"type"=>"keyword"}, "dst_rep_tags"=>{"type"=>"keyword"}, "input_ifname"=>{"type"=>"keyword"}, "input_snmp"=>{"type"=>"keyword"}, "ip_protocol"=>{"type"=>"keyword"}, "ip_version"=>{"type"=>"keyword"}, "next_hop"=>{"type"=>"ip"}, "output_ifname"=>{"type"=>"keyword"}, "output_snmp"=>{"type"=>"keyword"}, "packets"=>{"type"=>"long"}, "rep_tags"=>{"type"=>"keyword"}, "sampling_interval"=>{"type"=>"long"}, "server_addr"=>{"type"=>"ip"}, "server_asn"=>{"type"=>"long"}, "server_autonomous_system"=>{"type"=>"keyword"}, "server_city"=>{"type"=>"keyword"}, "server_country"=>{"type"=>"keyword"}, "server_country_code"=>{"type"=>"keyword"}, "server_geo_location"=>{"type"=>"geo_point"}, "server_hostname"=>{"type"=>"keyword"}, "server_rep_tags"=>{"type"=>"keyword"}, "service_name"=>{"type"=>"keyword"}, "service_port"=>{"type"=>"long"}, "src_addr"=>{"type"=>"ip"}, "src_addr_trans"=>{"type"=>"ip"}, "src_asn"=>{"type"=>"long"}, "src_autonomous_system"=>{"type"=>"keyword"}, "src_city"=>{"type"=>"keyword"}, "src_country"=>{"type"=>"keyword"}, "src_country_code"=>{"type"=>"keyword"}, "src_geo_location"=>{"type"=>"geo_point"}, "src_hostname"=>{"type"=>"keyword"}, "src_mac"=>{"type"=>"keyword"}, "src_mask_len"=>{"type"=>"long"}, "src_port"=>{"type"=>"long"}, "src_port_trans"=>{"type"=>"long"}, "src_port_name"=>{"type"=>"keyword"}, "src_rep_tags"=>{"type"=>"keyword"}, "tcp_flags"=>{"type"=>"keyword"}, "tos"=>{"type"=>"long"}, "traffic_direction"=>{"type"=>"keyword"}, "traffic_locality"=>{"type"=>"keyword"}, "vlan"=>{"type"=>"long"}}}, "node"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ipaddr"=>{"type"=>"ip"}, "hostname"=>{"type"=>"keyword"}}}, "tags"=>{"type"=>"keyword"}}}}}}
[2019-04-25T11:16:01,803][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/elastiflow-3.4.1
[2019-04-25T11:16:04,340][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
[2019-04-25T11:16:04,575][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
[2019-04-25T11:16:22,959][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
[2019-04-25T11:16:22,961][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
[2019-04-25T11:16:46,226][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"elastiflow", :thread=>"#<Thread:0x166fce59 run>"}
[2019-04-25T11:16:46,296][INFO ][logstash.agent           ] Pipelines running {:count=>4, :running_pipelines=>[:cisco, :vmware, :elastiflow, :windows], :non_running_pipelines=>[]}
[2019-04-25T11:16:46,341][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:2055"}
[2019-04-25T11:16:46,353][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"33554432", :queue_size=>"4096"}
[2019-04-25T11:16:46,759][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

My pipelines.yml:

`[root@elk ~]# vim /etc/logstash/pipelines.yml
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

#- pipeline.id: main
#  path.config: "/etc/logstash/conf.d/*.conf"

- pipeline.id: cisco
  path.config: "/etc/logstash/conf.d/cisco.conf"

- pipeline.id: vmware
  path.config: "/etc/logstash/conf.d/vmware.conf"

- pipeline.id: windows
  path.config: "/etc/logstash/conf.d/windows.conf"

- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

My elastiflow/conf.d directory (I have disabled all files for sflow and ipfix):

[root@elk ~]# ll /etc/logstash/elastiflow/conf.d/
total 140
-rw-r--r--. 1 root root  2139 Apr 23 15:11 10_input_ipfix_ipv4.logstash.conf.disabled
-rw-rw-r--. 1 root root  2134 Apr 23 15:11 10_input_ipfix_ipv6.logstash.conf.disabled
-rw-rw-r--. 1 root root  1580 Apr 24 15:32 10_input_netflow_ipv4.logstash.conf
-rw-rw-r--. 1 root root  1578 Apr 23 15:11 10_input_netflow_ipv6.logstash.conf.disabled
-rw-r--r--. 1 root root  1699 Apr 23 15:11 10_input_sflow_ipv4.logstash.conf.disabled
-rw-rw-r--. 1 root root  1697 Apr 23 15:11 10_input_sflow_ipv6.logstash.conf.disabled
-rw-rw-r--. 1 root root  2593 Apr 23 15:11 20_filter_10_begin.logstash.conf.disabled
-rw-rw-r--. 1 root root 27848 Apr 23 15:11 20_filter_20_netflow.logstash.conf
-rw-r--r--. 1 root root 19802 Apr 23 15:11 20_filter_30_ipfix.logstash.conf.disabled
-rw-r--r--. 1 root root 12542 Apr 23 15:11 20_filter_40_sflow.logstash.conf.disabled
-rw-rw-r--. 1 root root 39571 Apr 23 15:11 20_filter_90_post_process.logstash.conf
-rw-rw-r--. 1 root root  1621 Apr 24 17:33 30_output_10_single.logstash.conf
-rw-rw-r--. 1 root root  1674 Apr 23 15:11 30_output_20_multi.logstash.conf.disabled
robcowart commented 5 years ago

The likely issue is that you have disabled a required file:

-rw-rw-r--. 1 root root  2593 Apr 23 15:11 20_filter_10_begin.logstash.conf.disabled
fkocharli commented 5 years ago

No. I have disabled it for troubleshooting purposes. It is the same when file is enabled. I have all indexes except elasti.

[root@elk ~]# curl 'localhost:9200/_cat/indices?v'
health status index                                                   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   network-2019.05.01                                      oJl1CshqR3a2itLxxue9kw   5   1       2804            0      859kb          859kb
green  open   .kibana_task_manager                                    no-P1rc_Rg6-8iCupsxHIQ   1   0          2            0     13.2kb         13.2kb
yellow open   winlogbeat-6.7.1-2019.05.02                             bUEfIzahSVSOkvELJJOjow   5   1       5819            0      5.2mb          5.2mb
yellow open   network-2019.04.30                                      SyX6oGH9RV6JKhi9UdfTuA   5   1       3566            0    817.9kb        817.9kb
yellow open   winlogbeat-6.7.1-2019.04.26                             cAh-B2d9Sz6mDjU5zj0qTg   5   1      21565            0     17.2mb         17.2mb
yellow open   winlogbeat-6.7.1-2019.04.29                             avPTL1xKT1qPVVuuAkS3QA   5   1      21252            0       17mb           17mb
green  open   .kibana_1                                               HJseSurIQjipIoKkpJC_hA   1   0        350            1    326.6kb        326.6kb
yellow open   network-2019.04.27                                      Arge_TjeTryD87wSGACM8g   5   1       1986            0    658.5kb        658.5kb
yellow open   network-2019.04.24                                      -DEWU5cAS_GVRXvnFrlmAw   5   1        850            0    588.4kb        588.4kb
yellow open   logstash-vsphere-syslog-esxi-2019.04.25                 1vz4VaAcS3-S6KKwqZEisQ   5   1       1203            0      2.3mb          2.3mb
yellow open   %{[@metadata][beat]}-%{[@metadata][version]}-2019.04.25 y1kXBwmqShC_e7yIuE4eiA   5   1         58            0    261.4kb        261.4kb
yellow open   winlogbeat-6.7.1-2019.04.27                             wCdeaUHiS-eZlLQJfaEp2Q   5   1      19814            0     15.6mb         15.6mb
yellow open   winlogbeat-6.7.1-2019.04.30                             pJKxCwPzQB2uGWkB7yqqYw   5   1      22070            0     17.9mb         17.9mb
yellow open   winlogbeat-6.7.1-2019.04.23                             kmh0pomMS8OCS2Xn9DZMSQ   5   1      18987            0     15.5mb         15.5mb
yellow open   winlogbeat-6.7.1-2019.04.28                             Fs4awzxOSOSpiwP4F4nx3Q   5   1      19835            0     15.6mb         15.6mb
yellow open   network-2019.04.26                                      RSB4Uj1cQQiz9RNNCH-6xA   5   1       2038            0    775.7kb        775.7kb
yellow open   winlogbeat-6.7.1-2019.05.01                             jHNBLQMaR7yZ7IqbJ27Isw   5   1      22014            0     17.7mb         17.7mb
yellow open   winlogbeat-6.7.1-2019.04.24                             ASXdS9eOQjysEL1HICYd9g   5   1      21391            0       17mb           17mb
yellow open   winlogbeat-6.7.1-2019.04.25                             tn7azpQDTwiCkMcQ-HOM-Q   5   1      22472            0     18.2mb         18.2mb
yellow open   network-2019.04.25                                      dhrGCq0zTlSvr8c2eCtK6w   5   1       2450            0    862.5kb        862.5kb
yellow open   network-2019.04.28                                      -e219JgFQx6Ul_toh3kuFw   5   1       1974            0    771.7kb        771.7kb
yellow open   network-2019.05.02                                      t5i66l7aSRKLg9E4T9FCag   5   1       1412            0    948.3kb        948.3kb
yellow open   network-2019.04.29                                      J-F1WasOSbiBUovvkheI2A   5   1       2044            0    677.5kb        677.5kb
robcowart commented 5 years ago

Can you send me a PCAP of some of your flows so I can investigate?

fkocharli commented 5 years ago

How can I get this?

robcowart commented 5 years ago

Use tcpdump, wireshark or similar.

robcowart commented 5 years ago

Closing this issue as it is a duplicate of #311. Please focus all comments on #311.