LOGSTASH ERROR WHEN ENABLING NETFLOW

[dwizel]

This is the error I am getting when I try to enable Netflow in Logstash. Curretly I have Filebeat sending logs to Logstash. I there a way to resolve this please?

./logstash --modules netflow --setup -M netflow.var.input.udp.port=9995 Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2019-08-23 12:41:27.222 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2019-08-23 12:41:27.237 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.2.0"} [INFO ] 2019-08-23 12:41:27.951 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] modulescommon - Setting up the netflow module [ERROR] 2019-08-23 12:41:28.482 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>} [ERROR] 2019-08-23 12:41:28.619 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>} [ERROR] 2019-08-23 12:41:28.818 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] sourceloader - Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:108:in block in pipeline_configs'", "org/jruby/RubyArray.java:1792:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:54:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:14:inpipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in block in fetch'", "org/jruby/RubyArray.java:2572:incollect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:367:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in block in initialize'"]} [ERROR] 2019-08-23 12:41:28.826 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:367:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]} [INFO ] 2019-08-23 12:41:29.083 [LogStash::Runner] runner - Logstash shut down.

[jesusgn90]

Hi @dwizel ,

I think the module is trying to reach Elasticsearch using the wrong address.

Is your Elasticsearch listening on localhost or is it listening on a certain IP address?

Is your Elasticsearch listening on HTTPS or is it listening on HTTP?

Is your Elasticsearch using security (credentials)?

Per the docs https://www.elastic.co/guide/en/logstash/current/netflow-module.html#configuring-netflow I can see the module must be configured properly, see Common options section where they explain how to configure those parameters.

Regards, Jesús

[dwizel]

It is listening on localhost because when I try to change it to 0.0.0.0 or the IP of my server it breaks when I restart the service. No it's not using security credentials.

[jesusgn90]

Yes, that's right @dwizel , for listening on wildcards (0.0.0.0) or specific addresses (such as 10.0.0.2) you need to edit the elasticsearch.yml as follow (I'm assuming you are using 7.x):

...
# Arbitrary name, choose it at your own
node.name: node-1 

# 0.0.0.0 is valid for all network interfaces, but you can use <instance_ip> instead
network.host: 0.0.0.0 

# Include here the node.name value
cluster.initial_master_nodes: ["node-1"] 
...

Now, restart the service so changes take effect:

$ sudo systemctl restart elasticsearch

Regards, Jesús

[dwizel]

Hi jesusgn90, as I said I have already tried that twice and it breaks (starts for a bit then fails).

[jesusgn90]

Hello again @dwizel ,

As I can see in https://github.com/wazuh/wazuh-kibana-app/issues/1683#issuecomment-527033366 this seems to be solved for you.

Are you still in need of help or can we close this ticket too?

Regards, Jesús

[dwizel]

Yes still need help, The elasticsearch bit is solved but this is the error I get when I try to setup Netflow in Logstash. Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2019-09-03 12:40:57.460 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2019-09-03 12:40:57.476 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.2.0"} [INFO ] 2019-09-03 12:40:58.118 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] modulescommon - Setting up the netflow module [ERROR] 2019-09-03 12:40:58.667 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>} [ERROR] 2019-09-03 12:40:58.810 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>} [ERROR] 2019-09-03 12:40:58.946 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] sourceloader - Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:108:in block in pipeline_configs'", "org/jruby/RubyArray.java:1792:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:54:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:14:inpipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in block in fetch'", "org/jruby/RubyArray.java:2572:incollect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:367:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in block in initialize'"]} [ERROR] 2019-09-03 12:40:58.953 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:367:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]} [INFO ] 2019-09-03 12:40:59.228 [LogStash::Runner] runner - Logstash shut down.

Does it matter if I already have Nfsen listening on that port as well?

[jesusgn90]

Hi @dwizel ,

Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

For those errors, please let us know about the Logstash directories:

ls -lh /etc/logstash/conf.d/
ls -lh /etc/logstash/
ls -lh /usr/share/logstash

All errors come from Logstash not reading its own configuration properly.

Does it matters if I already have Nfsen listening on that port as well?

Definitively yes, every service in your system should use different ports, not only for the Elastic stack but all services.

Regards, Jesús

[dwizel]

Not a problem, here you go the info you requested.

ls -lh /etc/logstash/conf.d/

root@CLAY-EU:~# ls -lh /etc/logstash/conf.d/
total 4.0K
-rw-r--r-- 1 root root 459 Aug 22 09:22 01-wazuh.conf

ls -lh /etc/logstash/

root@CLAY-EU:~# ls -lh /etc/logstash/
total 44K
drwxrwxr-x 2 root root 4.0K Sep  3 12:39 conf.d
-rw-r----- 1 root root 2.8K Jul 25 18:47 elastic.bak
-rw-r--r-- 1 root root 1.8K Jul 26 08:49 jvm.options
-rw-r--r-- 1 root root 4.9K Jun 21 03:26 log4j2.properties
-rw-r--r-- 1 root root  342 Jun 21 03:26 logstash-sample.conf
-rw-r--r-- 1 root root 8.1K Aug 23 12:39 logstash.yml
-rw-r--r-- 1 root root  285 Jun 21 03:26 pipelines.yml
-rw------- 1 root root 1.7K Jun 21 03:26 startup.options

ls -lh /usr/share/logstash

root@CLAY-EU:~# ls -lh /usr/share/logstash
total 880K
drwxrwxr-x 2 logstash logstash 4.0K Jul 17 11:27 bin
-rw-r--r-- 1 logstash logstash 2.3K Jun 21 03:26 CONTRIBUTORS
drwxrwxr-x 4 logstash logstash 4.0K Jul 25 13:44 data
-rw-r--r-- 1 logstash logstash 4.1K Jul 26 08:53 Gemfile
-rw-r--r-- 1 logstash logstash  22K Jul 26 08:52 Gemfile.lock
drwxrwxr-x 6 logstash logstash 4.0K Jul 17 11:27 lib
-rw-r--r-- 1 logstash logstash  14K Jun 21 03:26 LICENSE.txt
drwxrwxr-x 4 logstash logstash 4.0K Jul 17 11:27 logstash-core
drwxrwxr-x 3 logstash logstash 4.0K Jul 17 11:27 logstash-core-plugin-api
drwxrwxr-x 4 logstash logstash 4.0K Jul 17 11:27 modules
-rw-r--r-- 1 logstash logstash 790K Jun 21 03:26 NOTICE.TXT
drwxrwxr-x 3 logstash logstash 4.0K Jul 17 11:27 tools
drwxrwxr-x 4 logstash logstash 4.0K Jul 17 11:27 vendor
drwxrwxr-x 9 logstash logstash 4.0K Jul 17 11:27 x-pack

[jesusgn90]

Hi @dwizel ,

Sorry for the late response. Can you try to change the owner and group for the whole /etc/logstash directory?

$ sudo chown logstash:logstash -R /etc/logstash

Then, restart the Logstash service and look at the logs again.

$ sudo systemctl restart logstash

Regards, Jesús

[juankaromo]

Hi @dwizel

Looks like your problem's solved, so I'm closing the issue. If not, please do not hesitate to open another issue.

Regards.