search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.27k stars 24.86k forks source link

ES doesn't start due to a problem with REGEX matching of entries in '/proc/self/cgroup' #23218

Closed mg-2014 closed 7 years ago

mg-2014 commented 7 years ago

Elasticsearch version: 5.2.1

Plugins installed: []

JVM version: 1.8.0_112

OS version: Linux (Debian 8.6)

Description of the problem including expected versus actual behavior:

ES doesn't start due to a problem with REGEX matching of entries in /proc/self/cgroup (see stacktrace below).

The problem seems to be this pattern in OsProbe:191:

private static final Pattern CONTROL_GROUP_PATTERN = Pattern.compile("\\d+:([^:,]+(?:,[^:,]+)?):(/.*)");

It works for one or two components only.

Unfortunately my Debian 8 installation has a line with 11 components. Thus I'm unable to start ES on any of our Debian 8 Linux-VMs. Is there a workaround, e.g. a configuration option that skips this test altogether?

Provide logs (if relevant):

[2017-02-16T18:43:12,635][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: No match found
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.2.1.jar:5.2.1]
Caused by: java.lang.IllegalStateException: No match found
    at java.util.regex.Matcher.group(Matcher.java:536) ~[?:1.8.0_112]
    at org.elasticsearch.monitor.os.OsProbe.getControlGroups(OsProbe.java:216) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:414) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:466) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:45) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:45) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.node.Node.<init>(Node.java:345) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.1.jar:5.2.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.1.jar:5.2.1]
    ... 6 more
jasontedor commented 7 years ago

Thanks for the report @mg-2014 and sorry for this.

jasontedor commented 7 years ago

I opened #23219.

mg-2014 commented 7 years ago

Thanks for fixing it @jasontedor.

phile314-fh commented 7 years ago

I am also running into this problem. Any ideas when this fix will be released?

jasontedor commented 7 years ago

I'm very sorry, but we do not provide release timelines.

ghost commented 7 years ago

how to resolve this?

dakrone commented 7 years ago

@mrunalpagnis-RS the fix is available in 5.2.2+, 5.3.0+, and 5.4.0+, if you upgrade it should be resolved.

D3epDiv3r commented 2 years ago

Can you post a working solution for all those who encounter this issue?! I just encountered this issue and can't seem to find a fix anywhere... Please is there a way to fix this!!!!!

jasontedor commented 2 years ago

@D3epDiv3r This should be fixed in all versions since 5.2.2. Are you on an older version?

D3epDiv3r commented 2 years ago

Hey @jasontedor, thanks for the quick response! I appreciate it! I made this post on stackoverflow and on the discussions page on Elastic here: https://discuss.elastic.co/t/elasticsearch-fails-to-start-java-lang-illegal-state-exception-no-match-found/319239

This is the issue I am having:

I have been trying to get Elastic Stack to run for at least 2 days now, I have been given a project in school that requires me to use Elasticsearch 5.2.0 specifically (Logstash 5.2.0, and Kibana 5.2.0) I have not been able to get past installing Elasticsearch yet because it always fails to run the service. I have Java8 and Java11 (using Java8 for ES 5.2.0). The service always fails to run and anytime I check the error logs I keep getting the below:

[2022-11-17T20:37:58,110][INFO ][o.e.p.PluginsService     ] [F-2-m97] loaded module [reindex]
[2022-11-17T20:37:58,110][INFO ][o.e.p.PluginsService     ] [F-2-m97] loaded module [transport-netty3]
[2022-11-17T20:37:58,110][INFO ][o.e.p.PluginsService     ] [F-2-m97] loaded module [transport-netty4]
[2022-11-17T20:37:58,110][INFO ][o.e.p.PluginsService     ] [F-2-m97] no plugins loaded
[2022-11-17T20:37:58,779][ERROR][o.e.b.Bootstrap          ] Exception
java.lang.IllegalStateException: No match found
        at java.util.regex.Matcher.group(Matcher.java:645) ~[?:?]
        at org.elasticsearch.monitor.os.OsProbe.getControlGroups(OsProbe.java:216) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:414) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:466) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:45) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:45) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:345) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.2.0.jar:5.2.0]
[2022-11-17T20:37:58,788][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: No match found
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.2.0.jar:5.2.0]
Caused by: java.lang.IllegalStateException: No match found
        at java.util.regex.Matcher.group(Matcher.java:645) ~[?:?]
        at org.elasticsearch.monitor.os.OsProbe.getControlGroups(OsProbe.java:216) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:414) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:466) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:45) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:45) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:345) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.0.jar:5.2.0]
        ... 6 more

I have checked almost all online resources there is, with no luck! No one has a solution to this issue, at least not one I have been able to figure out... Does anyone have a fix for this?

D3epDiv3r commented 2 years ago

Also, I would have used the latest version of the Elastic Stack, however since my school explicitly states to use version 5.2.0 and a lot of configurations have changed since the release of that version to 8.5, I am not sure what to do. I was told to use the dashboards provided by Microsoft here (https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-intrusion-detection-open-source-tools#create-a-kibana-dashboard) and non of those dashboards are able to be imported into the ELK stack 8.5... Also, it has become too challenging to add index patterns to that version. Do you think the dashboards will work on 5.2.2 and how similar is that version to 5.2.0 (in terms of layout, functionality, and configurations)? Can I also install Elasticsearch 5.2.2, Logstash 5.2.2, and Kibana 5.2.2? because I noticed that different versions just do not work together!

jasontedor commented 2 years ago

5.2.2 should be backwards compatible with 5.2.0. I encourage you to at least upgrade to that version so that you get a fix for the issue that you're encountering, and pick up any other bug fixes that were delivered in the 5.2 series.

D3epDiv3r commented 2 years ago

That makes perfect sense! Will do just that, thank you so much 👍

D3epDiv3r commented 2 years ago

This is odd... I still face the same issue with v5.2.2

        ... 6 more
[2022-11-18T14:57:54,900][INFO ][o.e.n.Node               ] [] initializing ...
[2022-11-18T14:57:54,996][INFO ][o.e.e.NodeEnvironment    ] [5Yuafc8] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [26.5gb], net total_space [28.8gb], spins? [possibly], types [ext4]
[2022-11-18T14:57:54,996][INFO ][o.e.e.NodeEnvironment    ] [5Yuafc8] heap size [1.9gb], compressed ordinary object pointers [true]
[2022-11-18T14:57:54,998][INFO ][o.e.n.Node               ] node name [5Yuafc8] derived from node ID [5Yuafc8cTMyJO3VwKScGpA]; set [node.name] to override
[2022-11-18T14:57:55,000][INFO ][o.e.n.Node               ] version[5.2.2], pid[9279], build[f9d9b74/2017-02-24T17:26:45.835Z], OS[Linux/5.15.0-1023-azure/amd64], JVM[Private Build/OpenJDK 64-Bit Server VM/1.8.0_352/25.352->
[2022-11-18T14:57:56,097][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [aggs-matrix-stats]
[2022-11-18T14:57:56,098][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [ingest-common]
[2022-11-18T14:57:56,098][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [lang-expression]
[2022-11-18T14:57:56,098][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [lang-groovy]
[2022-11-18T14:57:56,098][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [lang-mustache]
[2022-11-18T14:57:56,099][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [lang-painless]
[2022-11-18T14:57:56,099][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [percolator]
[2022-11-18T14:57:56,099][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [reindex]
[2022-11-18T14:57:56,099][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [transport-netty3]
[2022-11-18T14:57:56,099][INFO ][o.e.p.PluginsService     ] [5Yuafc8] loaded module [transport-netty4]
[2022-11-18T14:57:56,100][INFO ][o.e.p.PluginsService     ] [5Yuafc8] no plugins loaded
[2022-11-18T14:57:56,993][ERROR][o.e.b.Bootstrap          ] Exception
java.lang.IllegalStateException: No match found
        at java.util.regex.Matcher.group(Matcher.java:536) ~[?:1.8.0_352]
        at org.elasticsearch.monitor.os.OsProbe.getControlGroups(OsProbe.java:216) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:414) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:466) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:45) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:45) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.node.Node.<init>(Node.java:345) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.2.2.jar:5.2.2]
[2022-11-18T14:57:57,003][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: No match found
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.2.2.jar:5.2.2]
Caused by: java.lang.IllegalStateException: No match found
        at java.util.regex.Matcher.group(Matcher.java:536) ~[?:1.8.0_352]
        at org.elasticsearch.monitor.os.OsProbe.getControlGroups(OsProbe.java:216) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:414) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:466) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:45) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:45) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.node.Node.<init>(Node.java:345) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
        ... 6 more

Is there something I am doing wrong?

D3epDiv3r commented 2 years ago

Hello, apparently the problem was not with the ELK stack version, but with the OS version (see: Elastisc Support Matrix). I had to downgrade from Ubuntu 20.04 to Ubuntu 16.04 because ES 5.2.* works well on 16.04 but with some of its own bugs. Thanks for your help!