robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.48k stars 592 forks source link

What happened in the last few months?! #336

Closed nsolimando closed 5 years ago

nsolimando commented 5 years ago

Back in March or early April I followed this guide to setting up Elastiflow: https://www.catapultsystems.com/blogs/install-elastiflow-on-ubuntu-18-04-part-3/

Everything worked perfectly and I am shipping flow data from my pfsense CARP pair via softflowd. Everything is working GREAT on that instance.

I am now looking to install another instance of this for another data center and I am having problems.

Aside from the oracle-java8-installer package being decommissioned (which I've found a workaround for) I am running into problem after problem after problem and starting to lose my mind. What am I doing wrong here? I am so close. The server is accepting and indexing flow data because I can see it in Kibana's "discover" tab. But not 1 dashboard is working. They all throw this error:

Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.server_hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"elastiflow-3.5.0-2019.05.22","node":"BF2hOm7ISeqJj-ye2SFZjg","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.server_hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.server_hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.server_hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}},"status":400}

PLEASE HELP. WHAT DO I HAVE TO DO TO MAKE THIS WORK?

I even tried manually installing the exact same ELK versions (6.6.2) as my working instance and I am still getting these errors.

robcowart commented 5 years ago

That error occurs when the index template was not installed into Elasticsearch before the index was created. The Logstash elasticsearch output should upload the index template when it starts.

I notice from the error message that you are using ElasticSearch 3.5.0. That release supports ONLY Elastic Stack 7.x. as is indicating in the install instructions.

Please always follow the instructions here in the repository. These are the only instructions that I maintain for each release.

nsolimando commented 5 years ago

So do you suggest installing ELK 7.x instead?

robcowart commented 5 years ago

That choice is yours. The only thing I recommend is that you use either Logstash 7.x or 6.1.3. I have seen a bunch of stability issues with 6.2 thru 6.7. The recent 7.x releases seem to have addressed those issues. In the version 6.x line, 6.1.3 was the last release that was mostly rock solid. Apparently there were issues as they phased in the new Java Execution engine and plugins.

nsolimando commented 5 years ago

Installed ELK 7.0 then followed INSTALL.md exactly:

[2019-05-23T20:30:41,700][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:elastiflow, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 18, column 1 (byte 889) after #------------------------------------------------------------------------------\n# Copyright (C)2019 Robert Cowart\n# \n# The contents of this file and/or repository are subject to the Robert Cowart\n# Public License (the \"License\") and may not be used or distributed except in\n# compliance with the License. You may obtain a copy of the License at:\n# \n# http://www.koiossian.com/public/robert_cowart_public_license.txt\n# \n# Software distributed under the License is distributed on an \"AS IS\" basis,\n# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for\n# the specific language governing rights and limitations under the License.\n# \n# The Original Source Code was developed by Robert Cowart. Portions created by\n# Robert Cowart are Copyright (C)2019 Robert Cowart. All Rights Reserved.\n#------------------------------------------------------------------------------\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]} [2019-05-23T20:30:41,700][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 18, column 1 (byte 889) after #------------------------------------------------------------------------------\n# Copyright (C)2019 Robert Cowart\n# \n# The contents of this file and/or repository are subject to the Robert Cowart\n# Public License (the \"License\") and may not be used or distributed except in\n# compliance with the License. You may obtain a copy of the License at:\n# \n# http://www.koiossian.com/public/robert_cowart_public_license.txt\n# \n# Software distributed under the License is distributed on an \"AS IS\" basis,\n# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for\n# the specific language governing rights and limitations under the License.\n# \n# The Original Source Code was developed by Robert Cowart. Portions created by\n# Robert Cowart are Copyright (C)2019 Robert Cowart. All Rights Reserved.\n#------------------------------------------------------------------------------\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]} [2019-05-23T20:30:42,136][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2019-05-23T20:30:47,114][INFO ][logstash.runner ] Logstash shut down.

robcowart commented 5 years ago

Please provide your pipelines.yml and logstash.yml file.

nsolimando commented 5 years ago

Sure, see attached.

pipelines.txt logstash.txt

robcowart commented 5 years ago

Can you do an ls -l on the /etc/logstash/elastiflow/conf.d directory?

nsolimando commented 5 years ago

total 144 -rw-r--r-- 1 root root 2139 May 23 21:27 10_input_ipfix_ipv4.logstash.conf -rw-r--r-- 1 root root 2134 May 23 21:27 10_input_ipfix_ipv6.logstash.conf.disabled -rw-r--r-- 1 root root 1580 May 23 21:27 10_input_netflow_ipv4.logstash.conf -rw-r--r-- 1 root root 1578 May 23 21:27 10_input_netflow_ipv6.logstash.conf.disabled -rw-r--r-- 1 root root 1699 May 23 21:27 10_input_sflow_ipv4.logstash.conf -rw-r--r-- 1 root root 1697 May 23 21:27 10_input_sflow_ipv6.logstash.conf.disabled -rw-r--r-- 1 root root 2593 May 23 21:27 20_filter_10_begin.logstash.conf -rw-r--r-- 1 root root 31360 May 23 21:27 20_filter_20_netflow.logstash.conf -rw-r--r-- 1 root root 19802 May 23 21:27 20_filter_30_ipfix.logstash.conf -rw-r--r-- 1 root root 12542 May 23 21:27 20_filter_40_sflow.logstash.conf -rw-r--r-- 1 root root 39709 May 23 21:27 20_filter_90_post_process.logstash.conf -rw-r--r-- 1 root root 1589 May 23 21:27 30_output_10_single.logstash.conf -rw-r--r-- 1 root root 1674 May 23 21:27 30_output_20_multi.logstash.conf.disabled

nsolimando commented 5 years ago

Rob,

What dashboards JSON file should I be using with ELK7.x? I used to use this but it no longer exists: https://github.com/robcowart/elastiflow/raw/master/kibana/elastiflow.dashboards.json

nsolimando commented 5 years ago

I am close. I am back to seeing legit flow data in the Discover tab- all with ELK7.1

robcowart commented 5 years ago

You must use elastiflow.kibana.7.0.x.json

nsolimando commented 5 years ago

It is working now. Thank you for the replies.

I must have had a folder structure problem or a .conf in the wrong place. It took several rebuilds but I finally have my 7.x dashboards working now.

Thanks again, best flow analytics product I've used.

robcowart commented 5 years ago

Glad you got it working.