search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.48k stars 596 forks source link

ERROR] [org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit #155

Closed andressalesa closed 6 years ago

andressalesa commented 6 years ago

I have followed its instructions and following others links to install elk, kibana and logstash version 6.3.2.

https://zassoul.blogspot.com/2018/05/netflowelastiflow.html http://blogs.catapultsystems.com/mdowst/archive/2018/06/18/install-elastiflow-on-ubuntu-18-04-part-3/#

I have made a clone: git clone https://github.com/robcowart/elastiflow.git

The logstash service does not work.

tail -f /var/log/logstash/logstash-plain.log

2018-08-02T12: 26: 44,129] [INFO] [logstash.setting.writabledirectory] Creating directory {: setting => "path.queue",: path => "/ var / lib / logstash / queue"} [2018-08-02T12: 26: 44,175] [INFO] [logstash.setting.writabledirectory] Creating directory {: setting => "path.dead_letter_queue",: path => "/ var / lib / logstash / dead_letter_queue"} [2018-08-02T12: 26: 45,724] [ERROR] [org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2018-08-02T12: 28: 04,883] [ERROR] [org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2018-08-02T12: 29: 19,247] [ERROR] [org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2018-08-02T12: 33: 26,672] [ERROR] [org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I don't load indexes, dashboard or send netflow traffic yet.

The problem occurs both for Ubuntu 18.04 and Centos 7-x86_64

Could somebody help me?

Thank you

robcowart commented 6 years ago

How much JVM heap did you give to Logstash?

andressalesa commented 6 years ago

Hi Rob,

I edited the file "nano /etc/logstash/jvm.options" and configured:

-Xms4g -Xmx4g

The virtual machine has 4 processors and 6GB of ram.

The problem is the same

robcowart commented 6 years ago

Are you running the whole stack on that VM, or only Logstash? It saves a lot of time if you provide complete details of the environment.

andressalesa commented 6 years ago

I have installed an mv in Esxi 6.5. On it I installed elk, kibana and logstash. Subsequently I installed elastiflow. I do not use docker or anything like that.

If you tell me what you need to know and how to get it, I'll give it to you.

I have already done the same installation 2 times in Centos and 2 in Ubuntu.

I can tell you that if I do not configure elastiflow and configure a simple configuration to read mysql data with jdbc logstash it works. It is from applying the entire configuration article

andressalesa commented 6 years ago

nano /etc/logstash/jvm.options

-Xms4g -Xmx4g

Install plugins:

sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-netflow sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow sudo /usr/share/logstash/bin/logstash-plugin install logstash-input-udp sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp sudo /usr/share/logstash/bin/logstash-plugin install logstash-filter-dns sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns

Clone repository:

apt-get install git

cd /tmp

git clone https://github.com/robcowart/elastiflow.git

Copy ElastiFlow configuration files to the Logstash directory

sudo cp -a elastiflow/logstash/elastiflow/. /etc/logstash/elastiflow/

Copy the elastiflow.conf to systemd

sudo cp -a elastiflow/logstash.service.d/. /etc/systemd/system/logstash.service.d/

nano /etc/systemd/system/logstash.service.d/elastiflow.conf

Environment="ELASTIFLOW_NAMESERVER=10.0.1.122" Environment="ELASTIFLOW_ES_HOST=127.0.0.1" Environment="ELASTIFLOW_ES_PASSWD=elastic" Environment="ELASTIFLOW_NETFLOW_IPV4_HOST=0.0.0.0" Environment="ELASTIFLOW_NETFLOW_IPV4_PORT=2055"

Add the ElastiFlow pipeline to pipelines.yml

sudo nano /etc/logstash/pipelines.yml

Disabled others configurations

cd /etc/logstash/elastiflow/conf.d mv 10_input_ipfix_ipv4.logstash.conf 10_input_ipfix_ipv4.logstash.conf.disabled mv 10_input_sflow_ipv4.logstash.conf 10_input_sflow_ipv4.logstash.conf.disabled mv 20_filter_30_ipfix.logstash.conf 20_filter_30_ipfix.logstash.conf.disabled mv 20_filter_40_sflow.logstash.conf 20_filter_40_sflow.logstash.conf.disabled

Edit configurations:

nano 10_input_netflow_ipv4.logstash.conf host => "${ELASTIFLOW_NETFLOW_IPV4_HOST:0.0.0.0}" pyort => "${ELASTIFLOW_NETFLOW_IPV4_PORT:2055}"

nano 30_output_10_single.logstash.conf hosts => [ "${ELASTIFLOW_ES_HOST:127.0.0.1:9200}" ] user => "${ELASTIFLOW_ES_USER:admin}" password => "${ELASTIFLOW_ES_PASSWD:elastic}"

robcowart commented 6 years ago

ElastiFlow is already a very tight squeeze with 8GB of RAM, when the whole stack is on a single machine. 12GB would be a better starting point. Odds are that you are running out of memory.

andressalesa commented 6 years ago

Now it has 13gb ram with same error.

1 2 3

robcowart commented 6 years ago

What version of java are you runnning?

pyort is misspelled. You shouldn't have to edit the pipeline. ElastiFlow is configured via environment variables.

If you are only starting ElastiFlow, you don't need the main pipeline. It should be removed from pipelines.yml.

How are you starting Logstash?

andressalesa commented 6 years ago

I wrote bad "port". It's correctly in configuration "10_input_netflow_ipv4.logstash.conf" host => "${ELASTIFLOW_NETFLOW_IPV4_HOST:0.0.0.0}" port => "${ELASTIFLOW_NETFLOW_IPV4_PORT:2055}"

I comment from pipelines.yml these lines

- pipeline.id: main

path.config: "/etc/logstash/conf.d/*.conf"

Only are theres

I followed the standard installation process of elk, kibana and logstash. In the case of logstash:

nano /etc/yum.repos.d/logstash.repo

[logstash-6.x] name=Logstash repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md

sudo yum install logstash

sudo /bin/systemctl daemon-reload sudo /bin/systemctl enable logstash.service

Logstash can be started and stopped as follows:

sudo systemctl start logstash.service sudo systemctl stop logstash.service

sudo systemctl start logstash.service

andressalesa commented 6 years ago

4 5

robcowart commented 6 years ago

Any word on the java version you are using?

andressalesa commented 6 years ago

Hi Rob,

OpenJDK Runtime Environment (build 1.8.0_181-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

robcowart commented 6 years ago

How much memory did you give to Elasticsearch? Your symptoms are similar to when Logstash is running out of memory. Or when Linux is running out of memory and the Logstash process if being killed by the OS because of it. You could put Logstash in debug logging mode and see if it provides any additional insights. I usually don't get much value from the debug logs, but it could be worth a try at this point.

andressalesa commented 6 years ago

In Ubuntu 18.04.1 TLS (stelk01), java version is

openjdk version "1.8.0_171" OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.18.04.1-b11) OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

In file logstash.yml I uncommented: log.level: trace

I ran command "/usr/share/logstash/bin/logstash --path.settings /etc/logstash --debug". But it shows:

[2018-08-06T09:07:07,810][DEBUG][logstash.runner ] --------------- Logstash Settings ------------------- [2018-08-06T09:07:07,935][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"} ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/pipelines.yml usage: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG] bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG] bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG] bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace] bin/logstash -V [--log.level fatal|error|warn|info|debug|trace] bin/logstash --help [2018-08-06T09:07:08,034][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

After I tried to ran

/usr/share/logstash/bin/logstash -f /etc/logstash -log.level=debug

and it shows:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2018-08-06 09:10:15.692 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2018-08-06 09:10:17.689 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.2"} [ERROR] 2018-08-06 09:10:19.029 [Converge PipelineAction::Create

] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]} [INFO ] 2018-08-06 09:10:19.738 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}

In Centos 7 (stelk02):

I ran:

[root@stelk02 logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash -log.level=debug WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO ] 2018-08-06 09:19:17.259 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} [INFO ] 2018-08-06 09:19:17.276 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} [WARN ] 2018-08-06 09:19:18.023 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2018-08-06 09:19:18.091 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"63e854fb-823e-4c5e-8308-56a52be62e5f", :path=>"/usr/share/logstash/data/uuid"} [INFO ] 2018-08-06 09:19:19.447 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.2"} [ERROR] 2018-08-06 09:19:20.204 [Converge PipelineAction::Create

] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]} [INFO ] 2018-08-06 09:19:20.920 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}

After I ran "/usr/share/logstash/bin/logstash -f /etc/logstash -log.level=debug". It shows

[2018-08-06T09:20:30,608][DEBUG][logstash.runner ] --------------- Logstash Settings ------------------- [2018-08-06T09:20:30,823][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"} ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/pipelines.yml usage: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG] bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG] bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG] bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace] bin/logstash -V [--log.level fatal|error|warn|info|debug|trace] bin/logstash --help [2018-08-06T09:20:30,989][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I have had attached log files and configurations.

Centos and Ubuntu have same configuration. logstash-plain_ubuntu.log

logstash-plain_centos.log

files.zip

robcowart commented 6 years ago

Don't start Logstash from the command line. You need to start it as it will be run normally. Just set the log level in logstash.yml and start Logstash with systemd.

Have you been starting from the command line the whole time? When you run from the command line with any arguments, it will ignore pipelines.yml.

andressalesa commented 6 years ago

The start of the services are in automatic mode in all services (elastic, logstash and kibana). I followed manual:

sudo /bin/systemctl daemon-reload sudo /bin/systemctl enable logstash.service

Only this time to see more information about logstash boot (debug). I have executed it in manual mode.

Is that, I removed the comment to the "line log.level: trace" of the file logstash.yml and did not see that it will get more information. That's why I thought you had to launch it in manual mode. However, I do not know if you have to modify something else to get more information. Could you tell me what I have to do to get more debug information?

I have increased the memory of the virtual machine to 15 GB. I have configured in files /etc/elasticsearch/jvm.options and /etc/logstash/jvm.options the parameters

-Xms4g -Xmx4g

Centos:

[2018-08-06T10:46:24,174][DEBUG][logstash.runner ] xpack.monitoring.collection.timeout_interval: 600000000000 [2018-08-06T10:46:24,174][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.username: "logstash_system" [2018-08-06T10:46:24,175][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate" [2018-08-06T10:46:24,175][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.sniffing: false [2018-08-06T10:46:24,175][DEBUG][logstash.runner ] xpack.monitoring.collection.pipeline.details.enabled: true [2018-08-06T10:46:24,175][DEBUG][logstash.runner ] xpack.monitoring.collection.config.enabled: true [2018-08-06T10:46:24,175][DEBUG][logstash.runner ] node.uuid: "" [2018-08-06T10:46:24,176][DEBUG][logstash.runner ] --------------- Logstash Settings ------------------- [2018-08-06T10:46:24,309][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"} [2018-08-06T10:46:24,408][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Ubuntu:

[2018-08-06T10:57:22,322][DEBUG][logstash.runner ] xpack.monitoring.collection.timeout_interval: 600000000000 [2018-08-06T10:57:22,323][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.username: "logstash_system" [2018-08-06T10:57:22,323][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate" [2018-08-06T10:57:22,323][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.sniffing: false [2018-08-06T10:57:22,324][DEBUG][logstash.runner ] xpack.monitoring.collection.pipeline.details.enabled: true [2018-08-06T10:57:22,324][DEBUG][logstash.runner ] xpack.monitoring.collection.config.enabled: true [2018-08-06T10:57:22,324][DEBUG][logstash.runner ] node.uuid: "" [2018-08-06T10:57:22,324][DEBUG][logstash.runner ] --------------- Logstash Settings ------------------- [2018-08-06T10:57:22,414][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"} [2018-08-06T10:57:22,882][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

However, I'm still not launching the traces of netflow, nor do I have anything else configured in the environment of ELK except "elastiflow" and if something from memory (we still do not know) and with 15gb does not start. 4gb for elk and 4gb for logstash ...

I have attached files /etc/systemd/logstash.service.d and etc/systemd/ and logfiles.

files_centos.zip files_ubuntu.zip

andressalesa commented 6 years ago

I have attached the logstash.zip logstash.yml

robcowart commented 6 years ago

Set log.level to debug.

https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html

andressalesa commented 6 years ago

I read it. And enabled

config.debug = true log.level=trace

I ran again then in Centos: logstash-plain.log

[2018-08-06T11:24:59,539][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [2018-08-06T11:24:59,553][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x20c3548c @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>} [2018-08-06T11:24:59,556][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [2018-08-06T11:24:59,558][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x7d83f099 @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>} [2018-08-06T11:25:00,443][DEBUG][logstash.runner ] -------- Logstash Settings ( means modified) --------- [2018-08-06T11:25:00,444][DEBUG][logstash.runner ] node.name: "stelk02" [2018-08-06T11:25:00,444][DEBUG][logstash.runner ] path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data") [2018-08-06T11:25:00,445][DEBUG][logstash.runner ] modules.cli: [] [2018-08-06T11:25:00,445][DEBUG][logstash.runner ] modules: [] [2018-08-06T11:25:00,445][DEBUG][logstash.runner ] modules_setup: false [2018-08-06T11:25:00,445][DEBUG][logstash.runner ] config.test_and_exit: false [2018-08-06T11:25:00,445][DEBUG][logstash.runner ] config.reload.automatic: false [2018-08-06T11:25:00,446][DEBUG][logstash.runner ] config.reload.interval: 3000000000 [2018-08-06T11:25:00,446][DEBUG][logstash.runner ] config.support_escapes: false [2018-08-06T11:25:00,446][DEBUG][logstash.runner ] metric.collect: true [2018-08-06T11:25:00,446][DEBUG][logstash.runner ] pipeline.id: "main" [2018-08-06T11:25:00,446][DEBUG][logstash.runner ] pipeline.system: false [2018-08-06T11:25:00,446][DEBUG][logstash.runner ] pipeline.workers: 4 [2018-08-06T11:25:00,447][DEBUG][logstash.runner ] pipeline.output.workers: 1 [2018-08-06T11:25:00,447][DEBUG][logstash.runner ] pipeline.batch.size: 125 [2018-08-06T11:25:00,447][DEBUG][logstash.runner ] pipeline.batch.delay: 50 [2018-08-06T11:25:00,447][DEBUG][logstash.runner ] pipeline.unsafe_shutdown: false [2018-08-06T11:25:00,447][DEBUG][logstash.runner ] pipeline.java_execution: false [2018-08-06T11:25:00,448][DEBUG][logstash.runner ] pipeline.reloadable: true [2018-08-06T11:25:00,448][DEBUG][logstash.runner ] path.plugins: [] [2018-08-06T11:25:00,448][DEBUG][logstash.runner ] config.debug: true (default: false) [2018-08-06T11:25:00,448][DEBUG][logstash.runner ] log.level: "trace" (default: "info") [2018-08-06T11:25:00,448][DEBUG][logstash.runner ] version: false [2018-08-06T11:25:00,448][DEBUG][logstash.runner ] help: false [2018-08-06T11:25:00,449][DEBUG][logstash.runner ] log.format: "plain" [2018-08-06T11:25:00,449][DEBUG][logstash.runner ] http.host: "127.0.0.1" [2018-08-06T11:25:00,449][DEBUG][logstash.runner ] http.port: 9600..9700 [2018-08-06T11:25:00,449][DEBUG][logstash.runner ] http.environment: "production" [2018-08-06T11:25:00,449][DEBUG][logstash.runner ] queue.type: "memory" [2018-08-06T11:25:00,450][DEBUG][logstash.runner ] queue.drain: false [2018-08-06T11:25:00,450][DEBUG][logstash.runner ] queue.page_capacity: 67108864 [2018-08-06T11:25:00,450][DEBUG][logstash.runner ] queue.max_bytes: 1073741824 [2018-08-06T11:25:00,450][DEBUG][logstash.runner ] queue.max_events: 0 [2018-08-06T11:25:00,450][DEBUG][logstash.runner ] queue.checkpoint.acks: 1024 [2018-08-06T11:25:00,450][DEBUG][logstash.runner ] queue.checkpoint.writes: 1024 [2018-08-06T11:25:00,451][DEBUG][logstash.runner ] queue.checkpoint.interval: 1000 [2018-08-06T11:25:00,451][DEBUG][logstash.runner ] dead_letter_queue.enable: false [2018-08-06T11:25:00,451][DEBUG][logstash.runner ] dead_letter_queue.max_bytes: 1073741824 [2018-08-06T11:25:00,451][DEBUG][logstash.runner ] slowlog.threshold.warn: -1 [2018-08-06T11:25:00,451][DEBUG][logstash.runner ] slowlog.threshold.info: -1 [2018-08-06T11:25:00,452][DEBUG][logstash.runner ] slowlog.threshold.debug: -1 [2018-08-06T11:25:00,452][DEBUG][logstash.runner ] slowlog.threshold.trace: -1 [2018-08-06T11:25:00,452][DEBUG][logstash.runner ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore" [2018-08-06T11:25:00,452][DEBUG][logstash.runner ] keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore") [2018-08-06T11:25:00,452][DEBUG][logstash.runner ] path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue") [2018-08-06T11:25:00,452][DEBUG][logstash.runner ] path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue") [2018-08-06T11:25:00,453][DEBUG][logstash.runner ] path.settings: "/etc/logstash" (default: "/usr/share/logstash/config") [2018-08-06T11:25:00,453][DEBUG][logstash.runner ] path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs") [2018-08-06T11:25:00,453][DEBUG][logstash.runner ] xpack.management.enabled: false [2018-08-06T11:25:00,453][DEBUG][logstash.runner ] xpack.management.logstash.poll_interval: 5000000000 [2018-08-06T11:25:00,453][DEBUG][logstash.runner ] xpack.management.pipeline.id: ["main"] [2018-08-06T11:25:00,454][DEBUG][logstash.runner ] xpack.management.elasticsearch.username: "logstash_system" [2018-08-06T11:25:00,454][DEBUG][logstash.runner ] xpack.management.elasticsearch.url: ["https://localhost:9200"] [2018-08-06T11:25:00,454][DEBUG][logstash.runner ] xpack.management.elasticsearch.sniffing: false [2018-08-06T11:25:00,454][DEBUG][logstash.runner ] xpack.monitoring.enabled: false [2018-08-06T11:25:00,454][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.url: ["http://localhost:9200"] [2018-08-06T11:25:00,455][DEBUG][logstash.runner ] xpack.monitoring.collection.interval: 10000000000 [2018-08-06T11:25:00,455][DEBUG][logstash.runner ] xpack.monitoring.collection.timeout_interval: 600000000000 [2018-08-06T11:25:00,455][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.username: "logstash_system" [2018-08-06T11:25:00,455][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate" [2018-08-06T11:25:00,455][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.sniffing: false [2018-08-06T11:25:00,456][DEBUG][logstash.runner ] xpack.monitoring.collection.pipeline.details.enabled: true [2018-08-06T11:25:00,456][DEBUG][logstash.runner ] xpack.monitoring.collection.config.enabled: true [2018-08-06T11:25:00,456][DEBUG][logstash.runner ] node.uuid: "" [2018-08-06T11:25:00,456][DEBUG][logstash.runner ] --------------- Logstash Settings ------------------- [2018-08-06T11:25:00,572][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"} [2018-08-06T11:25:00,702][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2018-08-06T11:25:39,720][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [2018-08-06T11:25:39,731][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x20c3548c @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>} [2018-08-06T11:25:39,734][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [2018-08-06T11:25:39,735][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x7d83f099 @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>} [2018-08-06T11:25:40,691][DEBUG][logstash.runner ] -------- Logstash Settings ( means modified) --------- [2018-08-06T11:25:40,692][DEBUG][logstash.runner ] node.name: "stelk02" [2018-08-06T11:25:40,693][DEBUG][logstash.runner ] path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data") [2018-08-06T11:25:40,693][DEBUG][logstash.runner ] modules.cli: [] [2018-08-06T11:25:40,693][DEBUG][logstash.runner ] modules: [] [2018-08-06T11:25:40,693][DEBUG][logstash.runner ] modules_setup: false [2018-08-06T11:25:40,693][DEBUG][logstash.runner ] config.test_and_exit: false [2018-08-06T11:25:40,694][DEBUG][logstash.runner ] config.reload.automatic: false [2018-08-06T11:25:40,694][DEBUG][logstash.runner ] config.reload.interval: 3000000000 [2018-08-06T11:25:40,694][DEBUG][logstash.runner ] config.support_escapes: false [2018-08-06T11:25:40,694][DEBUG][logstash.runner ] metric.collect: true [2018-08-06T11:25:40,694][DEBUG][logstash.runner ] pipeline.id: "main" [2018-08-06T11:25:40,695][DEBUG][logstash.runner ] pipeline.system: false [2018-08-06T11:25:40,695][DEBUG][logstash.runner ] pipeline.workers: 4 [2018-08-06T11:25:40,695][DEBUG][logstash.runner ] pipeline.output.workers: 1 [2018-08-06T11:25:40,695][DEBUG][logstash.runner ] pipeline.batch.size: 125 [2018-08-06T11:25:40,695][DEBUG][logstash.runner ] pipeline.batch.delay: 50 [2018-08-06T11:25:40,695][DEBUG][logstash.runner ] pipeline.unsafe_shutdown: false [2018-08-06T11:25:40,696][DEBUG][logstash.runner ] pipeline.java_execution: false [2018-08-06T11:25:40,696][DEBUG][logstash.runner ] pipeline.reloadable: true [2018-08-06T11:25:40,696][DEBUG][logstash.runner ] path.plugins: [] [2018-08-06T11:25:40,696][DEBUG][logstash.runner ] config.debug: true (default: false) [2018-08-06T11:25:40,696][DEBUG][logstash.runner ] log.level: "trace" (default: "info") [2018-08-06T11:25:40,697][DEBUG][logstash.runner ] version: false [2018-08-06T11:25:40,697][DEBUG][logstash.runner ] help: false [2018-08-06T11:25:40,697][DEBUG][logstash.runner ] log.format: "plain" [2018-08-06T11:25:40,697][DEBUG][logstash.runner ] http.host: "127.0.0.1" [2018-08-06T11:25:40,697][DEBUG][logstash.runner ] http.port: 9600..9700 [2018-08-06T11:25:40,697][DEBUG][logstash.runner ] http.environment: "production" [2018-08-06T11:25:40,698][DEBUG][logstash.runner ] queue.type: "memory" [2018-08-06T11:25:40,698][DEBUG][logstash.runner ] queue.drain: false [2018-08-06T11:25:40,698][DEBUG][logstash.runner ] queue.page_capacity: 67108864 [2018-08-06T11:25:40,698][DEBUG][logstash.runner ] queue.max_bytes: 1073741824 [2018-08-06T11:25:40,698][DEBUG][logstash.runner ] queue.max_events: 0 [2018-08-06T11:25:40,698][DEBUG][logstash.runner ] queue.checkpoint.acks: 1024 [2018-08-06T11:25:40,699][DEBUG][logstash.runner ] queue.checkpoint.writes: 1024 [2018-08-06T11:25:40,699][DEBUG][logstash.runner ] queue.checkpoint.interval: 1000 [2018-08-06T11:25:40,699][DEBUG][logstash.runner ] dead_letter_queue.enable: false [2018-08-06T11:25:40,699][DEBUG][logstash.runner ] dead_letter_queue.max_bytes: 1073741824 [2018-08-06T11:25:40,699][DEBUG][logstash.runner ] slowlog.threshold.warn: -1 [2018-08-06T11:25:40,699][DEBUG][logstash.runner ] slowlog.threshold.info: -1 [2018-08-06T11:25:40,700][DEBUG][logstash.runner ] slowlog.threshold.debug: -1 [2018-08-06T11:25:40,700][DEBUG][logstash.runner ] slowlog.threshold.trace: -1 [2018-08-06T11:25:40,700][DEBUG][logstash.runner ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore" [2018-08-06T11:25:40,700][DEBUG][logstash.runner ] keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore") [2018-08-06T11:25:40,700][DEBUG][logstash.runner ] path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue") [2018-08-06T11:25:40,700][DEBUG][logstash.runner ] path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue") [2018-08-06T11:25:40,700][DEBUG][logstash.runner ] path.settings: "/etc/logstash" (default: "/usr/share/logstash/config") [2018-08-06T11:25:40,701][DEBUG][logstash.runner ] path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs") [2018-08-06T11:25:40,701][DEBUG][logstash.runner ] xpack.management.enabled: false [2018-08-06T11:25:40,701][DEBUG][logstash.runner ] xpack.management.logstash.poll_interval: 5000000000 [2018-08-06T11:25:40,701][DEBUG][logstash.runner ] xpack.management.pipeline.id: ["main"] [2018-08-06T11:25:40,701][DEBUG][logstash.runner ] xpack.management.elasticsearch.username: "logstash_system" [2018-08-06T11:25:40,701][DEBUG][logstash.runner ] xpack.management.elasticsearch.url: ["https://localhost:9200"] [2018-08-06T11:25:40,702][DEBUG][logstash.runner ] xpack.management.elasticsearch.sniffing: false [2018-08-06T11:25:40,702][DEBUG][logstash.runner ] xpack.monitoring.enabled: false [2018-08-06T11:25:40,702][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.url: ["http://localhost:9200"] [2018-08-06T11:25:40,702][DEBUG][logstash.runner ] xpack.monitoring.collection.interval: 10000000000 [2018-08-06T11:25:40,702][DEBUG][logstash.runner ] xpack.monitoring.collection.timeout_interval: 600000000000 [2018-08-06T11:25:40,702][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.username: "logstash_system" [2018-08-06T11:25:40,703][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate" [2018-08-06T11:25:40,703][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.sniffing: false [2018-08-06T11:25:40,703][DEBUG][logstash.runner ] xpack.monitoring.collection.pipeline.details.enabled: true [2018-08-06T11:25:40,703][DEBUG][logstash.runner ] xpack.monitoring.collection.config.enabled: true [2018-08-06T11:25:40,703][DEBUG][logstash.runner ] node.uuid: "" [2018-08-06T11:25:40,703][DEBUG][logstash.runner ] --------------- Logstash Settings ------------------- [2018-08-06T11:25:40,832][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"} [2018-08-06T11:25:40,933][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

robcowart commented 6 years ago

I just looked back through the whole thread and realized I missed that you were using OpenJDK instead of Oracle JDK. I double-checked the docs, and while they claim support for OpenJDK, I have only ever used Oracle JDK. Since I can't see anything wrong with your configuration, I would try using the Oracle JDK and see if that changes things.

andressalesa commented 6 years ago

I uninstalled openjdk and installed Oracle Java:

In Ubuntu:

root@stelk01:/etc/logstash/elastiflow# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

Same error...

[2018-08-06T12:38:23,711][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [2018-08-06T12:38:23,737][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0xd42f73 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>} [2018-08-06T12:38:23,747][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [2018-08-06T12:38:23,748][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x581b5524 @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>} [2018-08-06T12:38:25,219][DEBUG][logstash.runner ] -------- Logstash Settings ( means modified) --------- [2018-08-06T12:38:25,221][DEBUG][logstash.runner ] node.name: "stelk01" [2018-08-06T12:38:25,221][DEBUG][logstash.runner ] path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data") [2018-08-06T12:38:25,221][DEBUG][logstash.runner ] modules.cli: [] [2018-08-06T12:38:25,222][DEBUG][logstash.runner ] modules: [] [2018-08-06T12:38:25,222][DEBUG][logstash.runner ] modules_setup: false [2018-08-06T12:38:25,222][DEBUG][logstash.runner ] config.test_and_exit: false [2018-08-06T12:38:25,222][DEBUG][logstash.runner ] config.reload.automatic: false [2018-08-06T12:38:25,223][DEBUG][logstash.runner ] config.reload.interval: 3000000000 [2018-08-06T12:38:25,223][DEBUG][logstash.runner ] config.support_escapes: false [2018-08-06T12:38:25,223][DEBUG][logstash.runner ] metric.collect: true [2018-08-06T12:38:25,223][DEBUG][logstash.runner ] pipeline.id: "main" [2018-08-06T12:38:25,223][DEBUG][logstash.runner ] pipeline.system: false [2018-08-06T12:38:25,224][DEBUG][logstash.runner ] pipeline.workers: 4 [2018-08-06T12:38:25,224][DEBUG][logstash.runner ] pipeline.output.workers: 1 [2018-08-06T12:38:25,224][DEBUG][logstash.runner ] pipeline.batch.size: 125 [2018-08-06T12:38:25,224][DEBUG][logstash.runner ] pipeline.batch.delay: 50 [2018-08-06T12:38:25,224][DEBUG][logstash.runner ] pipeline.unsafe_shutdown: false [2018-08-06T12:38:25,224][DEBUG][logstash.runner ] pipeline.java_execution: false [2018-08-06T12:38:25,225][DEBUG][logstash.runner ] pipeline.reloadable: true [2018-08-06T12:38:25,225][DEBUG][logstash.runner ] path.plugins: [] [2018-08-06T12:38:25,225][DEBUG][logstash.runner ] config.debug: true (default: false) [2018-08-06T12:38:25,225][DEBUG][logstash.runner ] log.level: "trace" (default: "info") [2018-08-06T12:38:25,225][DEBUG][logstash.runner ] version: false [2018-08-06T12:38:25,226][DEBUG][logstash.runner ] help: false [2018-08-06T12:38:25,226][DEBUG][logstash.runner ] log.format: "plain" [2018-08-06T12:38:25,226][DEBUG][logstash.runner ] http.host: "127.0.0.1" [2018-08-06T12:38:25,226][DEBUG][logstash.runner ] http.port: 9600..9700 [2018-08-06T12:38:25,227][DEBUG][logstash.runner ] http.environment: "production" [2018-08-06T12:38:25,227][DEBUG][logstash.runner ] queue.type: "memory" [2018-08-06T12:38:25,227][DEBUG][logstash.runner ] queue.drain: false [2018-08-06T12:38:25,227][DEBUG][logstash.runner ] queue.page_capacity: 67108864 [2018-08-06T12:38:25,227][DEBUG][logstash.runner ] queue.max_bytes: 1073741824 [2018-08-06T12:38:25,227][DEBUG][logstash.runner ] queue.max_events: 0 [2018-08-06T12:38:25,228][DEBUG][logstash.runner ] queue.checkpoint.acks: 1024 [2018-08-06T12:38:25,228][DEBUG][logstash.runner ] queue.checkpoint.writes: 1024 [2018-08-06T12:38:25,228][DEBUG][logstash.runner ] queue.checkpoint.interval: 1000 [2018-08-06T12:38:25,228][DEBUG][logstash.runner ] dead_letter_queue.enable: false [2018-08-06T12:38:25,228][DEBUG][logstash.runner ] dead_letter_queue.max_bytes: 1073741824 [2018-08-06T12:38:25,229][DEBUG][logstash.runner ] slowlog.threshold.warn: -1 [2018-08-06T12:38:25,229][DEBUG][logstash.runner ] slowlog.threshold.info: -1 [2018-08-06T12:38:25,229][DEBUG][logstash.runner ] slowlog.threshold.debug: -1 [2018-08-06T12:38:25,229][DEBUG][logstash.runner ] slowlog.threshold.trace: -1 [2018-08-06T12:38:25,229][DEBUG][logstash.runner ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore" [2018-08-06T12:38:25,230][DEBUG][logstash.runner ] keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore") [2018-08-06T12:38:25,230][DEBUG][logstash.runner ] path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue") [2018-08-06T12:38:25,230][DEBUG][logstash.runner ] path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue") [2018-08-06T12:38:25,230][DEBUG][logstash.runner ] path.settings: "/etc/logstash" (default: "/usr/share/logstash/config") [2018-08-06T12:38:25,231][DEBUG][logstash.runner ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs") [2018-08-06T12:38:25,231][DEBUG][logstash.runner ] xpack.management.enabled: false [2018-08-06T12:38:25,231][DEBUG][logstash.runner ] xpack.management.logstash.poll_interval: 5000000000 [2018-08-06T12:38:25,231][DEBUG][logstash.runner ] xpack.management.pipeline.id: ["main"] [2018-08-06T12:38:25,231][DEBUG][logstash.runner ] xpack.management.elasticsearch.username: "logstash_system" [2018-08-06T12:38:25,231][DEBUG][logstash.runner ] xpack.management.elasticsearch.url: ["https://localhost:9200"] [2018-08-06T12:38:25,232][DEBUG][logstash.runner ] xpack.management.elasticsearch.sniffing: false [2018-08-06T12:38:25,232][DEBUG][logstash.runner ] xpack.monitoring.enabled: false [2018-08-06T12:38:25,232][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.url: ["http://localhost:9200"] [2018-08-06T12:38:25,232][DEBUG][logstash.runner ] xpack.monitoring.collection.interval: 10000000000 [2018-08-06T12:38:25,232][DEBUG][logstash.runner ] xpack.monitoring.collection.timeout_interval: 600000000000 [2018-08-06T12:38:25,233][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.username: "logstash_system" [2018-08-06T12:38:25,233][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate" [2018-08-06T12:38:25,233][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.sniffing: false [2018-08-06T12:38:25,233][DEBUG][logstash.runner ] xpack.monitoring.collection.pipeline.details.enabled: true [2018-08-06T12:38:25,234][DEBUG][logstash.runner ] xpack.monitoring.collection.config.enabled: true [2018-08-06T12:38:25,234][DEBUG][logstash.runner ] node.uuid: "" [2018-08-06T12:38:25,234][DEBUG][logstash.runner ] --------------- Logstash Settings ------------------- [2018-08-06T12:38:25,344][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"} [2018-08-06T12:38:25,455][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

In Centos:

Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

[2018-08-06T12:51:35,022][DEBUG][logstash.runner ] xpack.monitoring.collection.timeout_interval: 600000000000 [2018-08-06T12:51:35,023][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.username: "logstash_system" [2018-08-06T12:51:35,024][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate" [2018-08-06T12:51:35,025][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.sniffing: false [2018-08-06T12:51:35,026][DEBUG][logstash.runner ] xpack.monitoring.collection.pipeline.details.enabled: true [2018-08-06T12:51:35,026][DEBUG][logstash.runner ] xpack.monitoring.collection.config.enabled: true [2018-08-06T12:51:35,027][DEBUG][logstash.runner ] node.uuid: "" [2018-08-06T12:51:35,028][DEBUG][logstash.runner ] --------------- Logstash Settings ------------------- [2018-08-06T12:51:35,135][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"} [2018-08-06T12:51:35,277][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

andressalesa commented 6 years ago

Hi Rob,

I have installed all again in Ubuntu 16 and it works. I made all same way. I don't understand what could be the fault.

it's fantastic!!!

Thank you

VanDuy91 commented 6 years ago

I installed on Ubuntu 16 but the error have not resolved. p/s: java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

jvm: -Xms4g -Xmx4g image

sudo /usr/share/elasticsearch/bin/elasticsearch -V Version: 6.4.0, Build: default/deb/595516e/2018-08-17T23:18:47.308994Z, JVM: 1.8.0_181

The VM: RAM=12GB; storage=120GB

robcowart commented 6 years ago

You will need to set the Logstash logging level to debug to see what is happening. I think that you will find that this is a Logstash issue.

From my recent testing I believe the best performance and stability can be has from Logstash 6.1.3. You can then upgrade individual plugins where necessary.

VanDuy91 commented 6 years ago

You will need to set the Logstash logging level to debug to see what is happening. I think that you will find that this is a Logstash issue.

From my recent testing I believe the best performance and stability can be has from Logstash 6.1.3. You can then upgrade individual plugins where necessary.

I installed ELK version 6.1.3 but it has the error in timepicker: it just show data if I chose "today" and another one is not (ELK just pushed data some minutes before). Kibana and EL on the difference VPS. Time on:

image As you can see, the data show at about 10:00 am, but the current time is 9:00 am. How can I fix it? Thank you.

techarun-weinvest commented 6 years ago

when I run the below command to check the configuration of logstash is fine.

sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -t

I'm getting below error,

Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2018-09-19T11:36:21,312][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "pipppeline.batch.delay" hasn't been registered>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:36:in get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:69:inset_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:88:in block in merge'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:88:in merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:137:invalidate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:278:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `

'"]} [2018-09-19T11:36:21,327][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Help in fixing the error

VanDuy91 commented 6 years ago

You will need to set the Logstash logging level to debug to see what is happening. I think that you will find that this is a Logstash issue. From my recent testing I believe the best performance and stability can be has from Logstash 6.1.3. You can then upgrade individual plugins where necessary.

I installed ELK version 6.1.3 but it has the error in timepicker: it just show data if I chose "today" and another one is not (ELK just pushed data some minutes before). Kibana and EL on the difference VPS. Time on:

  • kibana: Local time: Wed 2018-09-19 08:59:31 +07 Universal time: Wed 2018-09-19 01:59:31 UTC RTC time: Wed 2018-09-19 01:59:31
  • logstash+elasticsearch: Local time: Wed 2018-09-19 08:59:39 +07 Universal time: Wed 2018-09-19 01:59:39 UTC RTC time: Wed 2018-09-19 01:59:39

image As you can see, the data show at about 10:00 am, but the current time is 9:00 am. How can I fix it? Thank you.

You will need to set the Logstash logging level to debug to see what is happening. I think that you will find that this is a Logstash issue. From my recent testing I believe the best performance and stability can be has from Logstash 6.1.3. You can then upgrade individual plugins where necessary.

I installed ELK version 6.1.3 but it has the error in timepicker: it just show data if I chose "today" and another one is not (ELK just pushed data some minutes before). Kibana and EL on the difference VPS. Time on:

  • kibana: Local time: Wed 2018-09-19 08:59:31 +07 Universal time: Wed 2018-09-19 01:59:31 UTC RTC time: Wed 2018-09-19 01:59:31
  • logstash+elasticsearch: Local time: Wed 2018-09-19 08:59:39 +07 Universal time: Wed 2018-09-19 01:59:39 UTC RTC time: Wed 2018-09-19 01:59:39

image As you can see, the data show at about 10:00 am, but the current time is 9:00 am. How can I fix it? Thank you.

I just wait about 60 minutes then the error is fixed. :))

robcowart commented 6 years ago

@techarun-weinvest please open your problem as a separate issue. Also you need to provide a lot more information. In particular it is important to understand how you setup your environment variables and the rest of the configuration to be able to run Logstash interactively.

kuriackovskij commented 5 years ago

Maybe someone who has the same issue find my comment useful... I had the same error as described in this topic, Ubuntu 18, VM on ESXi 6.7 host, 10GB/6GB heap...re-installed several times. I noticed that all 4 CPUs are 90-100% constantly since you power-on the VM. I then went to VM settings and set Memory Reservation to 5GB (it's disabled by default for VMs). After reboot no errors anymore, CPUs are relatively calm now.

wantes-nsd commented 5 years ago

Listing how I fixed this error: my pipelines.yml file didn't have an emply line in between each new entry. For example, as you see below, if you don't have an empty line between your entries, then you'll get the error discussed above.

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

VS

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"
stepw66 commented 5 years ago

Hi Rob,

I have installed all again in Ubuntu 16 and it works. I made all same way. I don't understand what could be the fault.

it's fantastic!!!

Thank you

now i have

Now I have the same problem as you, do not know how to solve the problem. I follow this example. https://pandaways.com/elastiflow-with-mikrotik-and-centos-7/?fbclid=IwAR1kERZF-3BSjl4Hy0TY-2ejTDcou37YCpA8Lv2f2ok-CZFL_bDG-lVxrNc

vistalba commented 5 years ago

Hi together

I ran into the same issue as well :( ElastiFlow was working fine a few days. After a reboot of the server I run into this issue. I reconfigured the vm on my ESX 6.7 server to get 16GB of RAM and all of them is reserved. Unfortunatly it stuck in this state :( Don't know what to do now.

Logstash is consuming all of the avaiable cpu power on all (4) cores permanently.

Logoutput:

[2019-02-10T16:37:44,017][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Logoutput with debug:

[2019-02-10T16:41:44,780][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x55d42164 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2019-02-10T16:41:44,781][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2019-02-10T16:41:44,782][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x12eb0558 @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2019-02-10T16:41:45,055][DEBUG][logstash.runner          ] -------- Logstash Settings (* means modified) ---------
[2019-02-10T16:41:45,055][DEBUG][logstash.runner          ] node.name: "srv-rachel"
[2019-02-10T16:41:45,055][DEBUG][logstash.runner          ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[2019-02-10T16:41:45,055][DEBUG][logstash.runner          ] modules.cli: []
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] modules: []
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] modules_list: []
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] modules_variable_list: []
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] modules_setup: false
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] config.test_and_exit: false
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] config.reload.automatic: false
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] config.reload.interval: 3000000000
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] config.support_escapes: false
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] config.field_reference.parser: "COMPAT"
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] metric.collect: true
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] pipeline.id: "main"
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] pipeline.system: false
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] pipeline.workers: 4
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] pipeline.output.workers: 1
[2019-02-10T16:41:45,056][DEBUG][logstash.runner          ] pipeline.batch.size: 125
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] pipeline.batch.delay: 50
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] pipeline.unsafe_shutdown: false
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] pipeline.java_execution: false
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] pipeline.reloadable: true
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] path.plugins: []
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] *config.debug: true (default: false)
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] *log.level: "debug" (default: "info")
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] version: false
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] help: false
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] log.format: "plain"
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] http.host: "127.0.0.1"
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] http.port: 9600..9700
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] http.environment: "production"
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] queue.type: "memory"
[2019-02-10T16:41:45,057][DEBUG][logstash.runner          ] queue.drain: false
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] queue.page_capacity: 67108864
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] queue.max_bytes: 1073741824
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] queue.max_events: 0
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] queue.checkpoint.acks: 1024
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] queue.checkpoint.writes: 1024
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] queue.checkpoint.interval: 1000
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] queue.checkpoint.retry: false
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] dead_letter_queue.enable: false
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] dead_letter_queue.max_bytes: 1073741824
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] slowlog.threshold.warn: -1
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] slowlog.threshold.info: -1
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] slowlog.threshold.debug: -1
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] slowlog.threshold.trace: -1
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[2019-02-10T16:41:45,058][DEBUG][logstash.runner          ] *keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore")
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.management.enabled: false
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.management.logstash.poll_interval: 5000000000
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.management.pipeline.id: ["main"]
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.management.elasticsearch.username: "logstash_system"
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.management.elasticsearch.url: ["https://localhost:9200"]
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.management.elasticsearch.ssl.verification_mode: "certificate"
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.management.elasticsearch.sniffing: false
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.url: ["http://localhost:9200"]
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: 10000000000
[2019-02-10T16:41:45,059][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: 600000000000
[2019-02-10T16:41:45,060][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"
[2019-02-10T16:41:45,060][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
[2019-02-10T16:41:45,060][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.sniffing: false
[2019-02-10T16:41:45,060][DEBUG][logstash.runner          ] xpack.monitoring.collection.pipeline.details.enabled: true
[2019-02-10T16:41:45,060][DEBUG][logstash.runner          ] xpack.monitoring.collection.config.enabled: true
[2019-02-10T16:41:45,060][DEBUG][logstash.runner          ] node.uuid: ""
[2019-02-10T16:41:45,060][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2019-02-10T16:41:45,101][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2019-02-10T16:41:45,143][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Any ideas how to get it work again?

OS: Ubuntu 18.04

nicolaipre commented 3 years ago

Listing how I fixed this error: my pipelines.yml file didn't have an emply line in between each new entry. For example, as you see below, if you don't have an empty line between your entries, then you'll get the error discussed above.

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

VS

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

Thank you! This solved the problem for me!