Elasticsearch not booting when upgrading to 8.4 with plugin pre-installed

jguay commented 2 years ago

Elasticsearch Version

8.4.2

Installed Plugins

mapper-size

Java Version

bundled

OS Version

Any

Problem Description

When upgrading to 8.4.x (from 8.3.3 or lower) whereby elasticsearch on premise had a plugin installed, the node won't boot with this error in the logs fatal exception while booting Elasticsearch java.lang.IllegalArgumentException: Unknown properties for plugin [mapper-size] in plugin descriptor: [type]

Steps to Reproduce

Install elasticsearch 8.3.3 with plugin repository-gcs per discuss post (other plugins are affected)
Upgrade in place to 8.4.2

Furthermore command sudo bin/elasticsearch-plugin remove plugin-name in 8.4.x won't work and the plugin folder has to be removed manually from plugins folder of elasticsearch to workaround the issue (and reinstalled if plugin is not bundled)

This seems a breaking change was introduced from recent changes but documentation does not specify requirement related to removal old plugins

Logs (if relevant)

[2022-09-01T12:34:56,789][ERROR][o.e.b.Elasticsearch ] [node1] fatal exception while booting Elasticsearch
java.lang.IllegalArgumentException: Unknown properties for plugin [mapper-size] in plugin descriptor: [type]
at org.elasticsearch.plugins.PluginDescriptor.readFromProperties(PluginDescriptor.java:218) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.plugins.PluginsUtils.readPluginBundle(PluginsUtils.java:162) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.plugins.PluginsUtils.findBundles(PluginsUtils.java:143) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.plugins.PluginsUtils.getPluginBundles(PluginsUtils.java:125) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:141) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(PluginsService.java:579) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.node.Node.<init>(Node.java:399) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214) ~[elasticsearch-8.4.2.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.4.2.jar:?]

elasticsearchmachine commented 2 years ago

Pinging @elastic/es-core-infra (Team:Core/Infra)

grcevski commented 2 years ago

Hi @jguay, we will need to add a deprecation warning as well as documentation on how to remove the old plugin.

piyushkp commented 2 years ago

@grcevski getting similar error upgrading from 8.2.0 to 8.4.0

java.lang.IllegalArgumentException: Unknown properties for plugin [analysis-icu] in plugin descriptor: [modulename] at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:211) at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:413) at org.elasticsearch.plugins.PluginsService.findBundles(PluginsService.java:395) at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:388) at org.elasticsearch.plugins.PluginsService.(PluginsService.java:153) at org.elasticsearch.node.Node.(Node.java:393) at org.elasticsearch.node.Node.(Node.java:291) at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:234) at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) at org.elasticsearch.cli.Command.main(Command.java:77) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)

piyushkp commented 2 years ago

rjernst Any updates on this issue ? it's blocker for in-place upgrade

jguay commented 2 years ago

Just to add a bit of context :

Removing plugin documentation already mentions that plugin can be removed manually from folder if the CLI does not work
upgrade document already mentions to test for plugin compatibility
The main change in 8.4 the error is fatal level (so elasticsearch won't start)... In 8.3 we used to get a warning for example like java.lang.IllegalArgumentException: Plugin [mapper-size] was built for Elasticsearch version 8.2.3 but version 8.3.0 is running. In both cases; the plugin needed to be removed and reinstalled for the same version (there is no guaranty any plugin is compatible between different major/minor or even fix pack release)

ChrisHegarty commented 2 years ago

fatal exception while booting Elasticsearch java.lang.IllegalArgumentException: Unknown properties for plugin [mapper-size] in plugin descriptor: [type]

Specifically, this issue (the mapper-size plugin has a type=isolated property specified in its plugin descriptor) results from the change to Remove legacy bootstrap plugins #87775. The type property is no longer a valid property (since 8.4), therefore the parsing of the plugin descriptor treats it as an error.

ChrisHegarty commented 2 years ago

@jguay

The main change in 8.4 the error is fatal level (so elasticsearch won't start)... In 8.3 we used to get a warning for example like java.lang.IllegalArgumentException: Plugin [mapper-size] was built for Elasticsearch version 8.2.3 but version 8.3.0 is running.

Sorry, this is not what I see. In my local testing, both the above scenarios are fatal - prevent ES from starting/running. For example:

[2022-10-12T12:20:50,699][ERROR][o.e.b.Elasticsearch      ] [chegar-MBP.local] fatal exception while booting Elasticsearch
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [mapper-size] was built for Elasticsearch version 8.2.3 but version 8.3.0 is running
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:228) [elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) [elasticsearch-8.3.0.jar:?]
Caused by: java.lang.IllegalArgumentException: Plugin [mapper-size] was built for Elasticsearch version 8.2.3 but version 8.3.0 is running
    at org.elasticsearch.plugins.PluginsUtils.verifyCompatibility(PluginsUtils.java:85) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:379) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:274) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:149) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(PluginsService.java:552) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.node.Node.<init>(Node.java:388) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.node.Node.<init>(Node.java:300) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.0.jar:?]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.0.jar:?]
    ... 1 more
java.lang.IllegalArgumentException: Plugin [mapper-size] was built for Elasticsearch version 8.2.3 but version 8.3.0 is running
    at org.elasticsearch.server@8.3.0/org.elasticsearch.plugins.PluginsUtils.verifyCompatibility(PluginsUtils.java:85)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:379)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:274)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:149)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(PluginsService.java:552)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.node.Node.<init>(Node.java:388)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.node.Node.<init>(Node.java:300)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:230)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:230)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224)
    at org.elasticsearch.server@8.3.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
For complete error details, refer to the log at /Users/chegar/binaries/elasticsearch-8.3.0/logs/elasticsearch.log
ERROR: Elasticsearch did not exit normally - check the logs at /Users/chegar/binaries/elasticsearch-8.3.0/logs/elasticsearch.log

ERROR: Elasticsearch exited unexpectedly

rjernst commented 2 months ago

Although the error message is a little more confusing, I do not think it is worth special casing this situation. It only affects users who fail to upgrade their plugins, and presumably almost 2 years later they will have gotten past this issue. I'm going to close this, but in the future we will take more care with changes to plugin descriptors in minor versions.

elastic / elasticsearch