search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.43k stars 1.07k forks source link

Error when upgrading from ES6 to ES7 #9488

Closed H2Cyber closed 4 years ago

H2Cyber commented 4 years ago

Issue

I am running GL4-RC2 alongside ES6, both on ubuntu 20.04 and installed using packages. I am trying to upgrade ES from 6 to 7.

In elasticsearch.yml, the following line is uncommented as recommended in the GL documentation : #action.auto_create_index: false

So, to upgrade from ES6 to 7, I start by stopping ES : sudo systemctl stop elasticsearch.service

Then I install ES v7 by following the steps described here : wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list sudo apt-get update sudo apt install elasticsearch

As a result, the ES service won't start and would throw the following error :

[2020-11-15T14:00:02,845][ERROR][o.e.b.Bootstrap ] [graylog] Exception java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.watcher.Watcher] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:722) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:658) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:479) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:168) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.node.Node.<init>(Node.java:346) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.10.0.jar:7.10.0] at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.10.0.jar:7.10.0] Caused by: java.lang.reflect.InvocationTargetException at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:715) ~[elasticsearch-7.10.0.jar:7.10.0] ... 15 more Caused by: java.lang.IllegalArgumentException: the [action.auto_create_index] setting value [false] is too restrictive. disable [action.auto_create_index] or set it to [.watches,.triggered_watches,.watcher-history-*] at org.elasticsearch.xpack.watcher.Watcher.validAutoCreateIndex(Watcher.java:616) ~[?:?] at org.elasticsearch.xpack.watcher.Watcher.<init>(Watcher.java:248) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:715) ~[elasticsearch-7.10.0.jar:7.10.0] ... 15 more [2020-11-15T14:00:02,944][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [graylog] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.watcher.Watcher] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.10.0.jar:7.10.0] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.10.0.jar:7.10.0] Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.watcher.Watcher] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:722) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:658) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:479) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:168) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.node.Node.<init>(Node.java:346) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.0.jar:7.10.0] ... 6 more Caused by: java.lang.reflect.InvocationTargetException at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:715) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:658) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:479) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:168) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.node.Node.<init>(Node.java:346) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.0.jar:7.10.0] ... 6 more Caused by: java.lang.IllegalArgumentException: the [action.auto_create_index] setting value [false] is too restrictive. disable [action.auto_create_index] or set it to [.watches,.triggered_watches,.watcher-history-*] at org.elasticsearch.xpack.watcher.Watcher.validAutoCreateIndex(Watcher.java:616) ~[?:?] at org.elasticsearch.xpack.watcher.Watcher.<init>(Watcher.java:248) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:715) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:658) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:479) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:168) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.node.Node.<init>(Node.java:346) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.0.jar:7.10.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.0.jar:7.10.0] ... 6 more

Possible Solution

The problem was solved by making sur the following line is commented in elasticsearch.yml : #action.auto_create_index: false

Steps to Reproduce (for bugs)

  1. Make sure you are running GL4-RC2 along with ES6
  2. Stop ES : sudo systemctl stop elasticsearch.service
  3. Install ES v7 by following the steps described here
  4. ES7 won't start correctly until the action.auto_create_index: false is commented in its configuration

Your Environment

bernd commented 4 years ago

@aim4r The key error here is the following:

Caused by: java.lang.IllegalArgumentException: the [action.auto_create_index] setting value [false] is too restrictive. disable [action.auto_create_index] or set it to [.watches,.triggered_watches,.watcher-history-*]

You are using the regular Elasticsearch distribution which includes the x-pack features. With that you cannot use action.auto_create_index set to false as noted in the error message. Setting it to .watches,.triggered_watches,.watcher-history-* should make it work.

We are recommending the use of the oss packages in our documentation where a value of false works.

I am closing this issue because it's not a Graylog bug.

H2Cyber commented 4 years ago

@bernd thanks for the response, spot on.

The upgrade guidance in the Graylog documentation simply points to the ES official documentation, with no mention of the recommended use of ES oss packages there :

When upgrading Elasticsearch from one major version to another, please read the upgrade guides provided by elastic