ElasticSearch 5.4.0 fails to start - GLIBC_2.7 not found

[dmarkhas]

Elasticsearch version: 5.4.0

Plugins installed: [x-pack 5.4.0]

JVM version (java -version): 1.8.0.92

OS version (uname -a if on a Unix-like system): SLES10SP3

Description of the problem including expected versus actual behavior:

ElasticSearch fails to start after upgrading from 5.3.0 to 5.4.0. I assume this is related to the change from JNA 4.2.2 to JNA 4.4.0, which is mentioned at the very bottom of the release notes for 5.4.0. The version of glibc on my system (running SLES10SP3) is 2.4, which works perfectly fine with JNA 4.2.2 and ElasticSearch 5.3.0, so unless you guys can rebuild JNA with an older glibc version, this is a breaking change that effectively drops support for some operating systems that were supported until now (even if unofficially).

Provide logs (if relevant): [2017-05-05T07:30:57,376][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled. java.lang.UnsatisfiedLinkError: /tmp/jna-1604831803/jna1365978125964325355.tmp: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/jna-1604831803/jna1365978125964325355.tmp) at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_92] at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_92] at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_92] at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_92] at java.lang.System.load(System.java:1086) ~[?:1.8.0_92] at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0.jar:4.4.0 (b0)] at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0.jar:4.4.0 (b0)] at com.sun.jna.Native.(Native.java:190) ~[jna-4.4.0.jar:4.4.0 (b0)] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_92] at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_92] at org.elasticsearch.bootstrap.Natives.(Natives.java:45) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0]

[nik9000]

I agree this is caused by the upgrade of JNA.

SLES10 isn't in our compatibility matrix which means we don't test against it and we don't support it. In #23640 we reworked the change so that we could support JNA on all of our supported OSes but we didn't go back to SLES10 because we don't support it.

Sorry we broke you. We aren't going to rebuild JNA against SLES10 though.

[jasontedor]

We do not support SLES 10, and this should not be causing Elasticsearch to fail to start. While this message will appear in the logs, Elasticsearch should march on. Would you please double check this?

However, I am going to close this, we are not going to rebuild for OS that we do not support, sorry.

[dmarkhas]

Thanks for the replies guys. I understand that you can't support an infinite range of legacy operating systems, my intention was to bring attention to the fact that the JNA change could potentially be a breaking change and (IMO) should be documented as such.

@jasontedor you're right that ES attempts to proceed without JNA, but then runs into another exception when trying to load x-pack 5.4.0 (which I assume is related), and finally crashes:

[2017-05-05T07:31:15,608][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [arm-or-011] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to create native process factories for Machine Learning]; nested: FileNotFoundException[/tmp/controller_log_3984
 (No such file or directory)];
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.4.0.jar:5.4.0]
Caused by: org.elasticsearch.ElasticsearchException: Failed to create native process factories for Machine Learning
        at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:297) ~[?:?]
        at org.elasticsearch.xpack.XPackPlugin.createComponents(XPackPlugin.java:288) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$7(Node.java:410) ~[elasticsearch-5.4.0.jar:5.4.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_92]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_92]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_92]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_92]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_92]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_92]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_92]
        at org.elasticsearch.node.Node.<init>(Node.java:412) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.0.jar:5.4.0]
        ... 6 more

[nik9000]

@jasontedor you're right that ES attempts to proceed without JNA, but then runs into another exception when trying to load x-pack 5.4.0 (which I assume is related), and finally crashes:

It is only kind of related. I expect the failure is also caused by glibc. You can disable machine learning with xpack.ml.enabled.

[jasontedor]

@dmarkhas You can try xpack.ml.enabled=false to get past that one.

I understand that you can't support an infinite range of legacy operating systems, my intention was to bring attention to the fact that the JNA change could potentially be a breaking change and (IMO) should be documented as such.

Sorry, we document breaking changes for our supported OS, we can not envision every scenario that every user is running in and outline breaking changes for them.

xkcd 1172 is relevant here:

[dmarkhas]

Got it :) For the record, setting xpack.ml.enabled:false didn't help - I had to remove x-pack entirely.

[droberts195]

For reference Machine Learning currently requires glibc 2.9, although that's not guaranteed forever - we only guarantee that it will work with the version of glibc used by the oldest version of Linux supported by Elastic. I believe this is currently SLES 11 SP4, which uses glibc 2.11.

If you wanted to use the X-Pack features other than Machine Learning the solution is probably to rm -rf $ES_HOME/plugins/x-pack/platform/linux-x86_64 as well as setting xpack.ml.enabled: false in elasticsearch.yml after installing X-Pack but before starting Elasticsearch. Obviously this is still unsupported but it might get you back to where you were with 5.3.