search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.48k stars 595 forks source link

Elastiflow cant put data in ElasticsearchLo #311

Closed fkocharli closed 4 years ago

fkocharli commented 5 years ago

Hello Rob.

I am already 1 month cant fix this issue. Think only you can help. Will be really thankful if you can advise sollution.

I am running Elastickseach, Logstash and Kibana on the same host. I have reinstalled Logstash 6.7.2 and Elastiflow 3.4.2. The only difference I made that I have commented out 'username' and 'password' lines and replaced '127.0.0.1' with 'localhost' in my variables and 30_output_10_single.logstash.conf. Without this changes Logstash hangs and does not start at all.

From tcpdump output I can confirm that I receive data on 2055 port:

tcpdump -Xni ens160 port 2055
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
17:38:02.369725 IP 192.168.40.1.51232 > 192.168.40.161.iop: UDP, length 1388
    0x0000:  4500 0588 cc15 0000 ff11 185c c0a8 2801  E..........\..(.
    0x0010:  c0a8 28a1 c820 0807 0574 1e54 0009 0010  ..(......t.T....
    0x0020:  538f 141b 5cd0 60fd 0000 84a1 0000 0000  S...\.`.........
    0x0030:  0000 0558 0100 0015 0094 0004 0008 0004  ...X............
    0x0040:  0007 0002 000a 0002 000c 0004 000b 0002  ................
    0x0050:  000e 0002 0004 0001 00b0 0001 00b1 0001  ................
    0x0060:  00e1 0004 00e2 0004 00e3 0002 00e4 0002  ................
    0x0070:  00e9 0001 80ea 0002 0143 0008 0098 0008  .........C......

Below output shows that logstash listens 2055 port:

ss -tulpna | grep 2055
udp    UNCONN     0      0         *:2055                  *:*                   users:(("java",pid=8598,fd=74))

My log file has not any errors:

[2019-05-06T17:27:18,967][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.2"}
[2019-05-06T17:27:20,220][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2019-05-06T17:29:46,646][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"elastiflow", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-05-06T17:29:47,411][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-05-06T17:29:47,779][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-05-06T17:29:47,851][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-05-06T17:29:47,855][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-05-06T17:29:47,898][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-05-06T17:29:47,991][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elastiflow/templates/elastiflow.template.json"}
[2019-05-06T17:29:48,313][INFO ][logstash.outputs.elasticsearch] type"=>"long"}}}, {"sflow.frame_length_times_sampling_rate"=>{"path_match"=>"sflow.frame_length_times_sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.header_size"=>{"path_match"=>"sflow.header_size", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface"=>{"path_match"=>"sflow.input_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface_format"=>{"path_match"=>"sflow.input_interface_format", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface_value"=>{"path_match"=>"sflow.input_interface_value", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_address_next_hop_router"=>{"path_match"=>"sflow.ip_address_next_hop_router", "mapping"=>{"type"=>"ip"}}}, {"sflow.ip_checksum"=>{"path_match"=>"sflow.ip_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_dscp"=>{"path_match"=>"sflow.ip_dscp", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_ecn"=>{"path_match"=>"sflow.ip_ecn", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_flags"=>{"path_match"=>"sflow.ip_flags", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_fragment_offset"=>{"path_match"=>"sflow.ip_fragment_offset", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_header_length"=>{"path_match"=>"sflow.ip_header_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_identification"=>{"path_match"=>"sflow.ip_identification", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_next_header"=>{"path_match"=>"sflow.ip_next_header", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_options"=>{"path_match"=>"sflow.ip_options", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_packet_length"=>{"path_match"=>"sflow.ip_packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_priority"=>{"path_match"=>"sflow.ip_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_protocol"=>{"path_match"=>"sflow.ip_protocol", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_total_length"=>{"path_match"=>"sflow.ip_total_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_ttl"=>{"path_match"=>"sflow.ip_ttl", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_type"=>{"path_match"=>"sflow.ip_type", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_version"=>{"path_match"=>"sflow.ip_version", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface"=>{"path_match"=>"sflow.output_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_format"=>{"path_match"=>"sflow.output_interface_format", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_value"=>{"path_match"=>"sflow.output_interface_value", "mapping"=>{"type"=>"long"}}}, {"sflow.packet_length"=>{"path_match"=>"sflow.packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.padded"=>{"path_match"=>"sflow.padded", "mapping"=>{"type"=>"long"}}}, {"sflow.protocol"=>{"path_match"=>"sflow.protocol", "mapping"=>{"type"=>"keyword"}}}, {"sflow.protocol_name"=>{"path_match"=>"sflow.protocol_name", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sample_length"=>{"path_match"=>"sflow.sample_length", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_pool"=>{"path_match"=>"sflow.sample_pool", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_seq_number"=>{"path_match"=>"sflow.sample_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.sampling_rate"=>{"path_match"=>"sflow.sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.sequence_number"=>{"path_match"=>"sflow.sequence_number", "mapping"=>{"type"=>"long"}}}, {"sflow.sflow_type"=>{"path_match"=>"sflow.sflow_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sflow_version"=>{"path_match"=>"sflow.sflow_version", "mapping"=>{"type"=>"long"}}}, {"sflow.size_header"=>{"path_match"=>"sflow.size_header", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index"=>{"path_match"=>"sflow.source_id_index", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index_name"=>{"path_match"=>"sflow.source_id_index_name", "mapping"=>{"type"=>"keyword"}}}, {"sflow.source_id_type"=>{"path_match"=>"sflow.source_id_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_ip"=>{"path_match"=>"sflow.src_ip", "mapping"=>{"type"=>"ip"}}}, {"sflow.src_mac"=>{"path_match"=>"sflow.src_mac", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_mask_len"=>{"path_match"=>"sflow.src_mask_len", "mapping"=>{"type"=>"long"}}}, {"sflow.src_port"=>{"path_match"=>"sflow.src_port", "mapping"=>{"type"=>"long"}}}, {"sflow.src_priority"=>{"path_match"=>"sflow.src_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.src_vlan"=>{"path_match"=>"sflow.src_vlan", "mapping"=>{"type"=>"long"}}}, {"sflow.stripped"=>{"path_match"=>"sflow.stripped", "mapping"=>{"type"=>"long"}}}, {"sflow.sub_agent_id"=>{"path_match"=>"sflow.sub_agent_id", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_ack_number"=>{"path_match"=>"sflow.tcp_ack_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_checksum"=>{"path_match"=>"sflow.tcp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_flags"=>{"path_match"=>"sflow.tcp_flags", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_header_length"=>{"path_match"=>"sflow.tcp_header_length", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_is_ack"=>{"path_match"=>"sflow.tcp_is_ack", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_cwr"=>{"path_match"=>"sflow.tcp_is_cwr", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_ecn_echo"=>{"path_match"=>"sflow.tcp_is_ecn_echo", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_fin"=>{"path_match"=>"sflow.tcp_is_fin", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_nonce"=>{"path_match"=>"sflow.tcp_is_nonce", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_push"=>{"path_match"=>"sflow.tcp_is_push", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_reset"=>{"path_match"=>"sflow.tcp_is_reset", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_syn"=>{"path_match"=>"sflow.tcp_is_syn", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_is_urgent"=>{"path_match"=>"sflow.tcp_is_urgent", "mapping"=>{"type"=>"integer"}}}, {"sflow.tcp_options"=>{"path_match"=>"sflow.tcp_options", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_reserved"=>{"path_match"=>"sflow.tcp_reserved", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_seq_number"=>{"path_match"=>"sflow.tcp_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_urgent_pointer"=>{"path_match"=>"sflow.tcp_urgent_pointer", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_window_size"=>{"path_match"=>"sflow.tcp_window_size", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_checksum"=>{"path_match"=>"sflow.udp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_length"=>{"path_match"=>"sflow.udp_length", "mapping"=>{"type"=>"long"}}}, {"sflow.uptime_in_ms"=>{"path_match"=>"sflow.uptime_in_ms", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_cfi"=>{"path_match"=>"sflow.vlan_cfi", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_id"=>{"path_match"=>"sflow.vlan_id", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_priority"=>{"path_match"=>"sflow.vlan_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_type"=>{"path_match"=>"sflow.vlan_type", "mapping"=>{"type"=>"long"}}}, {"string_fields"=>{"mapping"=>{"type"=>"keyword"}, "match_mapping_type"=>"string", "match"=>"*"}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "event"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"host"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}, "flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"application"=>{"type"=>"keyword"}, "autonomous_system"=>{"type"=>"keyword"}, "bgp_next_hop"=>{"type"=>"ip"}, "bgp_valid_state"=>{"type"=>"long"}, "bytes"=>{"type"=>"long"}, "city"=>{"type"=>"keyword"}, "client_addr"=>{"type"=>"ip"}, "client_asn"=>{"type"=>"long"}, "client_autonomous_system"=>{"type"=>"keyword"}, "client_city"=>{"type"=>"keyword"}, "client_country"=>{"type"=>"keyword"}, "client_country_code"=>{"type"=>"keyword"}, "client_geo_location"=>{"type"=>"geo_point"}, "client_hostname"=>{"type"=>"keyword"}, "client_rep_tags"=>{"type"=>"keyword"}, "country"=>{"type"=>"keyword"}, "country_code"=>{"type"=>"keyword"}, "direction"=>{"type"=>"keyword"}, "dst_addr"=>{"type"=>"ip"}, "dst_addr_trans"=>{"type"=>"ip"}, "dst_asn"=>{"type"=>"long"}, "dst_autonomous_system"=>{"type"=>"keyword"}, "dst_city"=>{"type"=>"keyword"}, "dst_country"=>{"type"=>"keyword"}, "dst_country_code"=>{"type"=>"keyword"}, "dst_geo_location"=>{"type"=>"geo_point"}, "dst_hostname"=>{"type"=>"keyword"}, "dst_mac"=>{"type"=>"keyword"}, "dst_mask_len"=>{"type"=>"long"}, "dst_port"=>{"type"=>"long"}, "dst_port_trans"=>{"type"=>"long"}, "dst_port_name"=>{"type"=>"keyword"}, "dst_rep_tags"=>{"type"=>"keyword"}, "input_ifname"=>{"type"=>"keyword"}, "input_snmp"=>{"type"=>"keyword"}, "ip_protocol"=>{"type"=>"keyword"}, "ip_version"=>{"type"=>"keyword"}, "next_hop"=>{"type"=>"ip"}, "output_ifname"=>{"type"=>"keyword"}, "output_snmp"=>{"type"=>"keyword"}, "packets"=>{"type"=>"long"}, "rep_tags"=>{"type"=>"keyword"}, "sampling_interval"=>{"type"=>"long"}, "server_addr"=>{"type"=>"ip"}, "server_asn"=>{"type"=>"long"}, "server_autonomous_system"=>{"type"=>"keyword"}, "server_city"=>{"type"=>"keyword"}, "server_country"=>{"type"=>"keyword"}, "server_country_code"=>{"type"=>"keyword"}, "server_geo_location"=>{"type"=>"geo_point"}, "server_hostname"=>{"type"=>"keyword"}, "server_rep_tags"=>{"type"=>"keyword"}, "service_name"=>{"type"=>"keyword"}, "service_port"=>{"type"=>"long"}, "src_addr"=>{"type"=>"ip"}, "src_addr_trans"=>{"type"=>"ip"}, "src_asn"=>{"type"=>"long"}, "src_autonomous_system"=>{"type"=>"keyword"}, "src_city"=>{"type"=>"keyword"}, "src_country"=>{"type"=>"keyword"}, "src_country_code"=>{"type"=>"keyword"}, "src_geo_location"=>{"type"=>"geo_point"}, "src_hostname"=>{"type"=>"keyword"}, "src_mac"=>{"type"=>"keyword"}, "src_mask_len"=>{"type"=>"long"}, "src_port"=>{"type"=>"long"}, "src_port_trans"=>{"type"=>"long"}, "src_port_name"=>{"type"=>"keyword"}, "src_rep_tags"=>{"type"=>"keyword"}, "tcp_flags"=>{"type"=>"keyword"}, "tos"=>{"type"=>"long"}, "traffic_direction"=>{"type"=>"keyword"}, "traffic_locality"=>{"type"=>"keyword"}, "vlan"=>{"type"=>"long"}}}, "node"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ipaddr"=>{"type"=>"ip"}, "hostname"=>{"type"=>"keyword"}}}, "tags"=>{"type"=>"keyword"}}}}}}
[2019-05-06T17:29:49,100][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/elastiflow-3.4.2
[2019-05-06T17:29:50,173][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
[2019-05-06T17:29:50,205][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
[2019-05-06T17:30:03,856][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
[2019-05-06T17:30:03,858][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
[2019-05-06T17:30:24,184][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"elastiflow", :thread=>"#<Thread:0x50961ef3 run>"}
[2019-05-06T17:30:24,248][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:elastiflow], :non_running_pipelines=>[]}
[2019-05-06T17:30:24,302][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:2055"}
[2019-05-06T17:30:24,537][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"33554432", :queue_size=>"4096"}
[2019-05-06T17:30:24,956][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

List of my variables. I have disabled most of them. 

[root@elk logstash]# cat /etc/systemd/system/logstash.service.d/elastiflow.conf 
#------------------------------------------------------------------------------
# Copyright (C)2019 Robert Cowart
# 
# The contents of this file and/or repository are subject to the Robert Cowart
# Public License (the "License") and may not be used or distributed except in
# compliance with the License. You may obtain a copy of the License at:
# 
# http://www.koiossian.com/public/robert_cowart_public_license.txt
# 
# Software distributed under the License is distributed on an "AS IS" basis,
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
# the specific language governing rights and limitations under the License.
# 
# The Original Source Code was developed by Robert Cowart. Portions created by
# Robert Cowart are Copyright (C)2019 Robert Cowart. All Rights Reserved.
#------------------------------------------------------------------------------

[Service]
# ElastiFlow global configuration
Environment="ELASTIFLOW_DICT_PATH=/etc/logstash/elastiflow/dictionaries"
Environment="ELASTIFLOW_DEFINITION_PATH=/etc/logstash/elastiflow/definitions"
Environment="ELASTIFLOW_TEMPLATE_PATH=/etc/logstash/elastiflow/templates"
Environment="ELASTIFLOW_GEOIP_DB_PATH=/etc/logstash/elastiflow/geoipdbs"
Environment="ELASTIFLOW_GEOIP_CACHE_SIZE=8192"
Environment="ELASTIFLOW_GEOIP_LOOKUP=true"
Environment="ELASTIFLOW_ASN_LOOKUP=true"
Environment="ELASTIFLOW_KEEP_ORIG_DATA=true"
Environment="ELASTIFLOW_DEFAULT_APPID_SRCTYPE=__UNKNOWN"

# Name resolution option
#Environment="ELASTIFLOW_RESOLVE_IP2HOST=false"
#Environment="ELASTIFLOW_NAMESERVER=127.0.0.1"
#Environment="ELASTIFLOW_DNS_HIT_CACHE_SIZE=25000"
#Environment="ELASTIFLOW_DNS_HIT_CACHE_TTL=900"
#Environment="ELASTIFLOW_DNS_FAILED_CACHE_SIZE=75000"
#Environment="ELASTIFLOW_DNS_FAILED_CACHE_TTL=3600"

# Elasticsearch connection settings
#Environment="ELASTIFLOW_ES_USER=elastic"
#Environment="ELASTIFLOW_ES_PASSWD=changeme"

# If you need Logstash to connect to only one Elasticsearch server, use the following environment variable.
Environment="ELASTIFLOW_ES_HOST=localhost:9200"

# If you need Logstash to connect to one of an array of three Elasticsearch servers, use the following environment variables.
# It is also necessary to rename the output files to disable single node output, and enable multi-node.
#Environment="ELASTIFLOW_ES_HOST_1=127.0.0.1:9200"
#Environment="ELASTIFLOW_ES_HOST_2=127.0.0.2:9200"
#Environment="ELASTIFLOW_ES_HOST_3=127.0.0.3:9200"

# If ELASTIFLOW_ES_SSL_VERIFY is true then you must edit the output and set the path where the cacert can be found.
Environment="ELASTIFLOW_ES_SSL_ENABLE=false"
Environment="ELASTIFLOW_ES_SSL_VERIFY=false"

# Netflow - IPv4
Environment="ELASTIFLOW_NETFLOW_IPV4_HOST=0.0.0.0"
Environment="ELASTIFLOW_NETFLOW_IPV4_PORT=2055"
# Netflow - IPv6
#Environment="ELASTIFLOW_NETFLOW_IPV6_HOST=[::]"
#Environment="ELASTIFLOW_NETFLOW_IPV6_PORT=52055"
# Netflow - UDP input options
Environment="ELASTIFLOW_NETFLOW_UDP_WORKERS=4"
Environment="ELASTIFLOW_NETFLOW_UDP_QUEUE_SIZE=4096"
Environment="ELASTIFLOW_NETFLOW_UDP_RCV_BUFF=33554432"
# Netflow timestamp options
Environment="ELASTIFLOW_NETFLOW_LASTSW_TIMESTAMP=false"
Environment="ELASTIFLOW_NETFLOW_TZ=UTC"

# sFlow - IPv4
#Environment="ELASTIFLOW_SFLOW_IPV4_HOST=0.0.0.0"
#Environment="ELASTIFLOW_SFLOW_IPV4_PORT=6343"
# sFlow - IPv6
#Environment="ELASTIFLOW_SFLOW_IPV6_HOST=[::]"
#Environment="ELASTIFLOW_SFLOW_IPV6_PORT=56343"
# sFlow - UDP input options
#Environment="ELASTIFLOW_SFLOW_UDP_WORKERS=4"
#Environment="ELASTIFLOW_SFLOW_UDP_QUEUE_SIZE=4096"
#Environment="ELASTIFLOW_SFLOW_UDP_RCV_BUFF=33554432"

# IPFIX - IPv4
#Environment="ELASTIFLOW_IPFIX_TCP_IPV4_HOST=0.0.0.0"
#Environment="ELASTIFLOW_IPFIX_TCP_IPV4_PORT=4739"
#Environment="ELASTIFLOW_IPFIX_UDP_IPV4_HOST=0.0.0.0"
#Environment="ELASTIFLOW_IPFIX_UDP_IPV4_PORT=4739"
# IPFIX - IPv6
#Environment="ELASTIFLOW_IPFIX_TCP_IPV6_HOST=[::]"
#Environment="ELASTIFLOW_IPFIX_TCP_IPV6_PORT=54739"
#Environment="ELASTIFLOW_IPFIX_UDP_IPV6_HOST=[::]"
#Environment="ELASTIFLOW_IPFIX_UDP_IPV6_PORT=54739"
# IPFIX - UDP input options
#Environment="ELASTIFLOW_IPFIX_UDP_WORKERS=4"
#Environment="ELASTIFLOW_IPFIX_UDP_QUEUE_SIZE=4096"
#Environment="ELASTIFLOW_IPFIX_UDP_RCV_BUFF=33554432"

My pipelines.yml:

# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

My elastiflow/conf.d directory:

[root@elk logstash]# ll elastiflow/conf.d/
total 144
-rw-rw-r-- 1 root root  2139 May  6 15:47 10_input_ipfix_ipv4.logstash.conf.disabled
-rw-rw-r-- 1 root root  2134 May  6 15:47 10_input_ipfix_ipv6.logstash.conf.disabled
-rw-rw-r-- 1 root root  1580 May  6 15:47 10_input_netflow_ipv4.logstash.conf
-rw-rw-r-- 1 root root  1578 May  6 15:47 10_input_netflow_ipv6.logstash.conf.disabled
-rw-rw-r-- 1 root root  1699 May  6 15:47 10_input_sflow_ipv4.logstash.conf.disabled
-rw-rw-r-- 1 root root  1697 May  6 15:47 10_input_sflow_ipv6.logstash.conf.disabled
-rw-rw-r-- 1 root root  2593 May  6 15:47 20_filter_10_begin.logstash.conf
-rw-rw-r-- 1 root root 31360 May  6 15:47 20_filter_20_netflow.logstash.conf
-rw-rw-r-- 1 root root 19802 May  6 15:47 20_filter_30_ipfix.logstash.conf.disabled
-rw-rw-r-- 1 root root 12542 May  6 15:47 20_filter_40_sflow.logstash.conf.disabled
-rw-rw-r-- 1 root root 39709 May  6 15:47 20_filter_90_post_process.logstash.conf
-rw-rw-r-- 1 root root  1591 May  6 15:47 30_output_10_single.logstash.conf
-rw-rw-r-- 1 root root  1674 May  6 15:47 30_output_20_multi.logstash.conf.disabled

My 10_input_netflow_ipv4.logstash.conf file:

[root@elk conf.d]# cat 10_input_netflow_ipv4.logstash.conf
#------------------------------------------------------------------------------
# Copyright (C)2019 Robert Cowart
# 
# The contents of this file and/or repository are subject to the Robert Cowart
# Public License (the "License") and may not be used or distributed except in
# compliance with the License. You may obtain a copy of the License at:
# 
# http://www.koiossian.com/public/robert_cowart_public_license.txt
# 
# Software distributed under the License is distributed on an "AS IS" basis,
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
# the specific language governing rights and limitations under the License.
# 
# The Original Source Code was developed by Robert Cowart. Portions created by
# Robert Cowart are Copyright (C)2019 Robert Cowart. All Rights Reserved.
#------------------------------------------------------------------------------

input {
  # Netflow
  udp {
    id => "input_udp_netflow_ipv4"
    host => "${ELASTIFLOW_NETFLOW_IPV4_HOST:0.0.0.0}"
    port => "${ELASTIFLOW_NETFLOW_IPV4_PORT:2055}"
    workers => "${ELASTIFLOW_NETFLOW_UDP_WORKERS:4}"
    queue_size => "${ELASTIFLOW_NETFLOW_UDP_QUEUE_SIZE:2048}"
    receive_buffer_bytes => "${ELASTIFLOW_NETFLOW_UDP_RCV_BUFF:33554432}"
    codec => netflow {
      versions => [5,9,10]
      include_flowset_id => "true"
      netflow_definitions => "${ELASTIFLOW_DEFINITION_PATH:/etc/logstash/elastiflow/definitions}/netflow.yml"
      ipfix_definitions => "${ELASTIFLOW_DEFINITION_PATH:/etc/logstash/elastiflow/definitions}/ipfix.yml"
    }
    type => "netflow"
  }
}

My 30_output_10_single.logstash.conf file:

[root@elk conf.d]# cat 30_output_10_single.logstash.conf 
#------------------------------------------------------------------------------
# Copyright (C)2019 Robert Cowart
# 
# The contents of this file and/or repository are subject to the Robert Cowart
# Public License (the "License") and may not be used or distributed except in
# compliance with the License. You may obtain a copy of the License at:
# 
# http://www.koiossian.com/public/robert_cowart_public_license.txt
# 
# Software distributed under the License is distributed on an "AS IS" basis,
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
# the specific language governing rights and limitations under the License.
# 
# The Original Source Code was developed by Robert Cowart. Portions created by
# Robert Cowart are Copyright (C)2019 Robert Cowart. All Rights Reserved.
#------------------------------------------------------------------------------

output {
  elasticsearch {
    id => "output_elasticsearch_single"
    hosts => [ "${ELASTIFLOW_ES_HOST:localhost:9200}" ]
    ssl => "${ELASTIFLOW_ES_SSL_ENABLE:false}"
    ssl_certificate_verification => "${ELASTIFLOW_ES_SSL_VERIFY:false}"
    # If ssl_certificate_verification is true, uncomment cacert and set the path to the certificate.
    #cacert => "/PATH/TO/CERT"
    #user => "${ELASTIFLOW_ES_USER:elastic}"
    #password => "${ELASTIFLOW_ES_PASSWD:changeme}"
    index => "elastiflow-3.4.2-%{+YYYY.MM.dd}"
    template => "${ELASTIFLOW_TEMPLATE_PATH:/etc/logstash/elastiflow/templates}/elastiflow.template.json"
    template_name => "elastiflow-3.4.2"
    template_overwrite => "true"
  }
}
fkocharli commented 5 years ago

Hello Rob.

I have installed elk 7 on new machine. The same issue. Can you advise something?

robcowart commented 5 years ago

I don't see any obvious issue. Which OS are you using?

Commenting out the user and password of the elasticsearch output should make no difference. It would good to know what logs you are seeing if you start an unmodified version of the logstash pipeline.

fkocharli commented 5 years ago

I am using CentOS 7.

There is no any errors in logs, which you can see on my other topic regarding the same issue on Logstash 6.7: https://github.com/robcowart/elastiflow/issues/298

robcowart commented 5 years ago

If you are using Netflow v9 or IPFIX you should see logs telling you that it can't decode the flows. These should appear until the devices send a template record. The fact that you don't get any such logs leads me to believe that the packets aren't making it to Logstash. This would lead me to suspect an OS-level issue.

To determine if there is some other issue I would need a PCAP. You can capture this using tcpdump. You will need to ensure that it collects long enough to also capture template records from the devices sending data.

robcowart commented 4 years ago

The only other thing I can think of is the local firewall. You can try disabling this with...

sudo systemctl stop firewalld

Otherwise I am closing as I have no additional information to troubleshoot further.