search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.49k stars 595 forks source link

Set fielddata=true on [event.type] #341

Closed edielson closed 5 years ago

edielson commented 5 years ago

The request for this panel failed Fielddata is disabled on text fields by default. Set fielddata=true on [event.type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

robcowart commented 5 years ago

Please provide additional details. OS? and version? Elastic Stack version? ElastiFlow release?

Also... this error always indicated that the index template was not loaded properly into elasticsearch by the Logstash’s elasticsearch output.

edielson commented 5 years ago

Ubuntu 18.04.2 LTS Elasticsearch 6.8.0 ElastiFlow release

robcowart commented 5 years ago

Which ElastiFlow release???

edielson commented 5 years ago

ElastiFlow v3.5.0

edielson commented 5 years ago

I used this installation manual http://blogs.catapultsystems.com/mdowst/archive/2018/06/18/install-elastiflow-on-ubuntu-18-04-part-3/

edielson commented 5 years ago

I downloaded the latest version of ElastiFlow

robcowart commented 5 years ago

3.5 is for Elastic Stack 7. Please refer to the compatibility chart in INSTALL.md. You will need the last 3.4.x release. Index template changes mandated by the complete removal of multi-type indices in 7.0. Are the reason you are seeing this problem.

Note however that I have not tested 6.8. It came out after 7.0. Hopefully it is similar enough to 6.7 that a 3.4 release will work. You will also have to delete the indices that were created with the wrong version.

edielson commented 5 years ago

thank you so much, then you are installing Elastic 7.x any error I report here

robcowart commented 5 years ago

Where you successful with Elastic Stack 7.x and ElastiFlow 3.5.0?

edielson commented 5 years ago

It worked perfectly now with the Elastic Stack 7.X version

m0zart89 commented 4 years ago

Is it difficult to get elastiflow.kibana.6.8.json ? How can I create it ?

robcowart commented 4 years ago

@m0zart89 just use the 6.7 file. However, if you are getting the error mentioned in this issue, the Kibana dashboard import is not the problem. You issue is the that index template didn't get loaded into Elasticsearch when Logstash started.

nickzxcv commented 4 years ago

I'm having trouble with logstash loading the index template also I think. I'm running 6.8.5 of logstash, elasticsearch, and kibana. Elastiflow version 3.5.2. [2019-12-17T16:20:17,594][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/elastiflow-3.5.2 [2019-12-17T16:20:17,933][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://127.0.0.1:9200/_template/elastiflow-3.5.2'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:inwith_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:inblock in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:352:in template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:intemplate_install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:28:in install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:16:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:129:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:50:inblock in setup_after_successful_connection'"]}

robcowart commented 4 years ago

3.5.x won't work with ES verions pre-7.x. See this note in the README.md.

https://github.com/robcowart/elastiflow#getting-started

nickzxcv commented 4 years ago

Ah, thanks I missed that and was following this guide https://www.catapultsystems.com/blogs/install-elastiflow-on-ubuntu-18-04-part-3/ sorry about that! Thank you!!!

fsantulli commented 4 years ago

Is there any setup for Elasticsearch/Kibana 7.9 and LogStash 7.1.1 ? I have the same issue and i have the same error (maybe index related).

robcowart commented 4 years ago

@fsantulli this occurs when the index template was not loaded successfully. I have not yet tested with Elasticsearch 7.9. However I can tell you that if anything is expected to work it would be ElastiFlow 4.0.1, which was tested with Elastic Stack 7.8.1. It should not matter that you are using Logstash 7.1.1 as long as you have updated plugins as described in the installation instructions.

fsantulli commented 4 years ago

@robcowart I have installed Elastiflow 4.0.1 today with ES7.8.1 and even if i've updated the plugins it stucks with the fielddata=true issue. The installation was above Centos 7 with the guide at https://medium.com/@ronaldbartels/a-guide-to-installing-elastiflow-53c915250df8. Also, when importing the objects i have 3 errors over the maps.

robcowart commented 4 years ago

@fsantulli Can you provide the logs produced when Logstash starts?

fsantulli commented 4 years ago

@robcowart Here they are. I do confirm that elastiflow is listening on 127.0.0.1 at port 9200.

[2020-08-25T16:00:07,956][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"} [2020-08-25T16:02:36,602][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@127.0.0.1:9200/]}} [2020-08-25T16:02:37,176][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@127.0.0.1:9200/"} [2020-08-25T16:02:37,274][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7} [2020-08-25T16:02:37,278][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7} [2020-08-25T16:02:37,315][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]} [2020-08-25T16:02:37,366][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elastiflow/templates/elastiflow.template.json"} [2020-08-25T16:02:37,842][INFO ][logstash.outputs.elasticsearch] Index Lifecycle Management is set to 'auto', but will be disabled - Index Lifecycle management is not installed on your Elasticsearch cluster [2020-08-25T16:02:37,977][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"} [2020-08-25T16:02:37,851][INFO ][logstash.outputs.elasticsearch] ns"=>{"path_match"=>"sflow.tcp_options", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_reserved"=>{"path_match"=>"sflow.tcp_reserved", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_seq_number"=>{"path_match"=>"sflow.tcp_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_urgent_pointer"=>{"path_match"=>"sflow.tcp_urgent_pointer", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_window_size"=>{"path_match"=>"sflow.tcp_window_size", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_checksum"=>{"path_match"=>"sflow.udp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_length"=>{"path_match"=>"sflow.udp_length", "mapping"=>{"type"=>"long"}}}, {"sflow.uptime_in_ms"=>{"path_match"=>"sflow.uptime_in_ms", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_cfi"=>{"path_match"=>"sflow.vlan_cfi", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_id"=>{"path_match"=>"sflow.vlan_id", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_priority"=>{"path_match"=>"sflow.vlan_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_type"=>{"path_match"=>"sflow.vlan_type", "mapping"=>{"type"=>"long"}}}, {"string_fields"=>{"mapping"=>{"type"=>"keyword"}, "match_mapping_type"=>"string", "match"=>"*"}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "agent"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ephemeral_id"=>{"type"=>"keyword"}, "hostname"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}, "version"=>{"type"=>"keyword"}}}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "client"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"number"=>{"type"=>"keyword"}, "organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "bytes"=>{"type"=>"long"}, "domain"=>{"type"=>"keyword"}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}, "location"=>{"type"=>"geo_point"}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}, "nat"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ip"=>{"type"=>"ip"}, "port"=>{"type"=>"long"}}}, "packets"=>{"type"=>"long"}, "port"=>{"type"=>"long"}, "registered_domain"=>{"type"=>"keyword"}, "top_level_domain"=>{"type"=>"keyword"}, "user"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "email"=>{"type"=>"keyword"}, "full_name"=>{"type"=>"keyword"}, "group"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "hash"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "destination"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"number"=>{"type"=>"keyword"}, "organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "bytes"=>{"type"=>"long"}, "domain"=>{"type"=>"keyword"}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}, "location"=>{"type"=>"geo_point"}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}, "nat"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ip"=>{"type"=>"ip"}, "port"=>{"type"=>"long"}}}, "packets"=>{"type"=>"long"}, "port"=>{"type"=>"long"}, "registered_domain"=>{"type"=>"keyword"}, "top_level_domain"=>{"type"=>"keyword"}, "user"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "email"=>{"type"=>"keyword"}, "full_name"=>{"type"=>"keyword"}, "group"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "hash"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "ecs"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"version"=>{"type"=>"keyword"}}}, "event"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"category"=>{"type"=>"keyword"}, "dataset"=>{"type"=>"keyword"}, "duration"=>{"type"=>"long"}, "end"=>{"type"=>"date"}, "kind"=>{"type"=>"keyword"}, "module"=>{"type"=>"keyword"}, "severity"=>{"type"=>"long"}, "start"=>{"type"=>"date"}, "type"=>{"type"=>"keyword"}}}, "flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"bgp_next_hop"=>{"type"=>"ip"}, "bgp_valid_state"=>{"type"=>"long"}, "client_rep_tags"=>{"type"=>"keyword"}, "direction"=>{"type"=>"keyword"}, "dst_mac_oui"=>{"type"=>"keyword"}, "dst_mask_len"=>{"type"=>"long"}, "dst_port_name"=>{"type"=>"keyword"}, "dst_rep_tags"=>{"type"=>"keyword"}, "input_ifname"=>{"type"=>"keyword"}, "input_snmp"=>{"type"=>"keyword"}, "next_hop"=>{"type"=>"ip"}, "output_ifname"=>{"type"=>"keyword"}, "output_snmp"=>{"type"=>"keyword"}, "rep_tags"=>{"type"=>"keyword"}, "sampling_interval"=>{"type"=>"long"}, "server_rep_tags"=>{"type"=>"keyword"}, "service_name"=>{"type"=>"keyword"}, "service_port"=>{"type"=>"long"}, "src_mac_oui"=>{"type"=>"keyword"}, "src_mask_len"=>{"type"=>"long"}, "src_port_name"=>{"type"=>"keyword"}, "src_rep_tags"=>{"type"=>"keyword"}, "tcp_flags"=>{"type"=>"keyword"}, "tos"=>{"type"=>"long"}, "traffic_direction"=>{"type"=>"keyword"}, "traffic_locality"=>{"type"=>"keyword"}, "vlan"=>{"type"=>"long"}, "wifi_sta_mac"=>{"type"=>"keyword"}, "wifi_ssid"=>{"type"=>"keyword"}, "wifi_wtp_mac"=>{"type"=>"keyword"}}}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}}}, "host"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"architecture"=>{"type"=>"keyword"}, "hostname"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "ip"=>{"type"=>"ip"}, "os"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"family"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}, "platform"=>{"type"=>"keyword"}, "version"=>{"type"=>"keyword"}}}}}, "log"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"level"=>{"type"=>"keyword"}}}, "message"=>{"type"=>"text", "norms"=>false}, "network"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"application"=>{"type"=>"keyword"}, "bytes"=>{"type"=>"long"}, "iana_number"=>{"type"=>"long"}, "packets"=>{"type"=>"long"}, "transport"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}, "observer"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "egress"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"interface"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"alias"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "vlan"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "hostname"=>{"type"=>"keyword"}, "ingress"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"interface"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"alias"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "vlan"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}}}, "server"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"number"=>{"type"=>"keyword"}, "organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "bytes"=>{"type"=>"long"}, "domain"=>{"type"=>"keyword"}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}, "location"=>{"type"=>"geo_point"}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}, "nat"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ip"=>{"type"=>"ip"}, "port"=>{"type"=>"long"}}}, "packets"=>{"type"=>"long"}, "port"=>{"type"=>"long"}, "registered_domain"=>{"type"=>"keyword"}, "top_level_domain"=>{"type"=>"keyword"}, "user"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "email"=>{"type"=>"keyword"}, "full_name"=>{"type"=>"keyword"}, "group"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "hash"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "source"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"number"=>{"type"=>"keyword"}, "organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "bytes"=>{"type"=>"long"}, "domain"=>{"type"=>"keyword"}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}, "location"=>{"type"=>"geo_point"}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}, "nat"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ip"=>{"type"=>"ip"}, "port"=>{"type"=>"long"}}}, "packets"=>{"type"=>"long"}, "port"=>{"type"=>"long"}, "registered_domain"=>{"type"=>"keyword"}, "top_level_domain"=>{"type"=>"keyword"}, "user"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "email"=>{"type"=>"keyword"}, "full_name"=>{"type"=>"keyword"}, "group"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "hash"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "tags"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}}} [2020-08-25T16:02:38,195][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/elastiflow-4.0.1 [2020-08-25T16:02:38,722][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://127.0.0.1:9200/_template/elastiflow-4.0.1'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:inwith_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:inblock in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:352:in template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:intemplate_install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:28:in install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:16:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:130:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:51:inblock in setup_after_successful_connection'"]} [2020-08-25T16:02:51,477][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"} [2020-08-25T16:03:02,525][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"} [2020-08-25T16:03:03,148][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"} [2020-08-25T16:03:03,196][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"elastiflow", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#"} [2020-08-25T16:03:03,571][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"elastiflow"} [2020-08-25T16:03:04,028][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:4739", :ssl_enable=>"false"} [2020-08-25T16:03:04,039][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:2055"} [2020-08-25T16:03:04,203][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:4739"} [2020-08-25T16:03:04,205][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:6343"} [2020-08-25T16:03:04,582][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"33554432", :queue_size=>"4096"} [2020-08-25T16:03:04,684][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:4739", :receive_buffer_bytes=>"33554432", :queue_size=>"4096"} [2020-08-25T16:03:04,684][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:6343", :receive_buffer_bytes=>"33554432", :queue_size=>"4096"} [2020-08-25T16:03:04,857][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:elastiflow], :non_running_pipelines=>[]} [2020-08-25T16:03:05,982][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

fsantulli commented 4 years ago

@robcowart this is the error detail:

[2020-08-26T10:16:26,964][INFO ][logstash.outputs.elasticsearch] Bad Request http://127.0.0.1:9200/_template/elastiflow-4.0.1 {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"}],"type":"illegal_argument_exception","reason":"unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"},"status":400}

fsantulli commented 4 years ago

The issue was the elasticsearch-oss release installed in place of just elasticsearch... poor me! Now template has been installed. Just waiting for data.