elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.59k stars 24.63k forks source link

Centos 7 and - java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native #57018

Closed lejeczek closed 4 years ago

lejeczek commented 4 years ago

Elasticsearch version (bin/elasticsearch --version):

$ /usr/share/elasticsearch/bin/elasticsearch --version
Version: 7.7.0, Build: default/rpm/81a1e9eda8e6183f5237786246f6dced26a10eaf/2020-05-12T02:01:37.602180Z, JVM: 14

Plugins installed: []

JVM version (java -version): openjdk version "1.8.0_252" OpenJDK Runtime Environment (build 1.8.0_252-b09) OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

OS version (uname -a if on a Unix-like system):

3.10.0-1127.8.2.el7.x86_64

Description of the problem including expected versus actual behavior:

$ sudo journalctl -lf -o cat -u elasticsearch.service
Starting Elasticsearch...
fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33)
at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:116)
at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:126)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1510)
at org.elasticsearch.node.Node.start(Node.java:810)
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:317)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:402)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Failed to start Elasticsearch.
Unit elasticsearch.service entered failed state.
elasticsearch.service failed.

Steps to reproduce: A fresh installation and first systemd start.

Provide logs (if relevant):

[2020-05-20T22:25:41,410][INFO ][o.e.p.PluginsService     ] [whale.priv.xx.xx.priv.xx.xx.x] no plugins loaded
[2020-05-20T22:25:46,363][INFO ][o.e.x.s.a.s.FileRolesStore] [whale.priv.xx.xx.priv.xx.xx.x] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-05-20T22:25:47,054][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [whale.priv.xx.xx.priv.xx.xx.x] [controller/635112] [Main.cc@110] controller (64 bit): Version 7.7.0 (Build a8939d3da43f33) Copyright (c) 2020 Elasticsearch BV
[2020-05-20T22:25:47,892][INFO ][o.e.d.DiscoveryModule    ] [whale.priv.xx.xx.priv.xx.xx.x] using discovery type [zen] and seed hosts providers [settings]
[2020-05-20T22:25:48,913][INFO ][o.e.n.Node               ] [whale.priv.xx.xx.priv.xx.xx.x] initialized
[2020-05-20T22:25:48,914][INFO ][o.e.n.Node               ] [whale.priv.xx.xx.priv.xx.xx.x] starting ...
[2020-05-20T22:25:49,111][INFO ][o.e.t.TransportService   ] [whale.priv.xx.xx.priv.xx.xx.x] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-05-20T22:25:49,342][WARN ][o.e.b.BootstrapChecks    ] [whale.priv.xx.xx.priv.xx.xx.x] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2020-05-20T22:25:49,343][WARN ][o.e.b.BootstrapChecks    ] [whale.priv.xx.xx.priv.xx.xx.x] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-05-20T22:25:49,346][INFO ][o.e.c.c.Coordinator      ] [whale.priv.xx.xx.priv.xx.xx.x] cluster UUID [Sf_kyq4ATb2Doh7uy0ZyUA]
[2020-05-20T22:25:49,367][INFO ][o.e.c.c.ClusterBootstrapService] [whale.priv.xx.xx.priv.xx.xx.x] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-05-20T22:25:49,559][INFO ][o.e.c.s.MasterService    ] [whale.priv.xx.xx.priv.xx.xx.x] elected-as-master ([1] nodes joined)[{whale.priv.xx.xx.priv.xx.xx.x}{QpMP0s3jStqxSElpJUtxPQ}{YNrWU8aoRcaK0gWlol4voA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134992162816, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 4, version: 7, delta: master node changed {previous [], current [{whale.priv.xx.xx.priv.xx.xx.x}{QpMP0s3jStqxSElpJUtxPQ}{YNrWU8aoRcaK0gWlol4voA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134992162816, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2020-05-20T22:25:49,671][INFO ][o.e.c.s.ClusterApplierService] [whale.priv.xx.xx.priv.xx.xx.x] master node changed {previous [], current [{whale.priv.xx.xx.priv.xx.xx.x}{QpMP0s3jStqxSElpJUtxPQ}{YNrWU8aoRcaK0gWlol4voA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134992162816, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 4, version: 7, reason: Publication{term=4, version=7}
[2020-05-20T22:25:49,773][INFO ][o.e.h.AbstractHttpServerTransport] [whale.priv.xx.xx.priv.xx.xx.x] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-05-20T22:25:49,776][INFO ][o.e.n.Node               ] [whale.priv.xx.xx.priv.xx.xx.x] started
[2020-05-20T22:25:49,779][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [whale.priv.xx.xx.priv.xx.xx.x] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.xxcessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:116) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:126) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1510) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:810) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:317) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:402) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.7.0.jar:7.7.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.7.0.jar:7.7.0]
nik9000 commented 4 years ago

It'd probably be useful to see your distro version like from /etc/issue.

elasticmachine commented 4 years ago

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

rjernst commented 4 years ago

The log file provided appears to only be the tail. Can you please provide the rest of the file? JNA failed to load, and it should have an error above the log file snippet here.

Separately, we currently lack graceful degredation for the systemd notifier hook when jna is not available. Need to think about what to do in this case, since we can't dynamically change systemd's reliance on this configuration option we have within our service file.

tuckson commented 4 years ago

Check your tmp directory. Can be set in jvm.options file -Djava.io.tmpdir=<path> needs to be set to a writable directory. In default ES 7 this is set to a variable. Then of course this variable should be filled.

lejeczek commented 4 years ago

This is a head (it does not seem fixing ES_TMPDIR helps)

[2020-06-09T17:18:05,435][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] unable to load JNA native support library, native methods will be disabled. java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna8471419410690735218.tmp: /tmp/jna--1985354563/jna8471419410690735218.tmp: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?] at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2452) ~[?:?] at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2508) ~[?:?] at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2704) ~[?:?] at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2637) ~[?:?] at java.lang.Runtime.load0(Runtime.java:745) ~[?:?] at java.lang.System.load(System.java:1871) ~[?:?] at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)] at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)] at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)] at java.lang.Class.forName0(Native Method) ~[?:?] at java.lang.Class.forName(Class.java:340) ~[?:?] at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:110) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.7.0.jar:7.7.0] at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.7.0.jar:7.7.0] [2020-06-09T17:18:05,451][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot check if running as root because JNA is not available [2020-06-09T17:18:05,452][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot install system call filter because JNA is not available [2020-06-09T17:18:05,453][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot register console handler because JNA is not available [2020-06-09T17:18:05,455][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot getrlimit RLIMIT_NPROC because JNA is not available [2020-06-09T17:18:05,455][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot getrlimit RLIMIT_AS because JNA is not available [2020-06-09T17:18:05,455][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot getrlimit RLIMIT_FSIZE because JNA is not available [2020-06-09T17:18:05,732][INFO ][o.e.e.NodeEnvironment ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [129.8gb], net total_space [249.9gb], types [rootfs] [2020-06-09T17:18:05,735][INFO ][o.e.e.NodeEnvironment ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] heap size [1gb], compressed ordinary object pointers [true] [2020-06-09T17:18:05,893][INFO ][o.e.n.Node ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] node name [whale.priv.aaaa.ccc.priv.ddd.ee.ff], node ID [QpMP0s3jStqxSElpJUtxPQ], cluster name [elasticsearch] [2020-06-09T17:18:05,898][INFO ][o.e.n.Node ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] version[7.7.0], pid[2051470], build[default/rpm/81a1e9eda8e6183f5237786246f6dced26a10eaf/2020-05-12T02:01:37.602180Z], OS[Linux/3.10.0-1127.8.2.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14/14+36] [2020-06-09T17:18:05,898][INFO ][o.e.n.Node ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] JVM home [/usr/share/elasticsearch/jdk] [2020-06-09T17:18:05,899][INFO ][o.e.n.Node ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rp ...

rjernst commented 4 years ago

We have not experienced this in our own testing. These are kinds of failures to initialize JNA are usually environmental issues, for example having /tmp mounted noexec. See the documentation for what seems like this exact issue: https://www.elastic.co/guide/en/elasticsearch/reference/master/executable-jna-tmpdir.html. As this is not a bug, I hope you don't mind I close this issue.

sktrinh12 commented 4 years ago

Hi Sorry to bring this issue up again; I got the same error on a centos machine and then installed openJDK which then gave me the error below. Can anyone please give me some clues on how to resolve the error? I followed that link about noexec for tmpdir but it kinda is over my head. Please be gentle, I am a newbie.

[2020-07-30T13:10:36,809][WARN ][o.e.b.Natives            ]  unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-17213976625080293983/jna18232491376079922055.tmp: /tmp/elasticsearch-17213976625080293983/jna18232491376079922055.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2452) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2508) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2704) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2637) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:745) ~[?:?]
        at java.lang.System.load(System.java:1871) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018) ~[jna-5.5.0.jar:5.5.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) ~[jna-5.5.0.jar:5.5.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:195) ~[jna-5.5.0.jar:5.5.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:340) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:110) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.8.1.jar:7.8.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.8.1.jar:7.8.1]
vipin12117 commented 4 years ago

change in /etc/elasticsearch/jvm.options

-Djava.io.tmpdir=${ES_TMPDIR}

-Djava.io.tmpdir=/var/log/elasticsearch

issue will fix.

elbembo commented 3 years ago

change in /etc/elasticsearch/jvm.options

-Djava.io.tmpdir=${ES_TMPDIR}

-Djava.io.tmpdir=/var/log/elasticsearch

issue will fix.

after try every thing from every where you are right

jaredbrogan commented 3 years ago

change in /etc/elasticsearch/jvm.options

-Djava.io.tmpdir=${ES_TMPDIR}

-Djava.io.tmpdir=/var/log/elasticsearch

issue will fix.

Yep, that seemed to fix our problem as well. The only thing I did differently was update /etc/sysconfig/elasticsearch instead of /etc/elasticsearch/jvm.options and added the following:

# Elasticsearch temp directory
ES_TMPDIR=/var/log/elasticsearch

Thanks for the help!

BBvT commented 3 years ago

change in /etc/elasticsearch/jvm.options

-Djava.io.tmpdir=${ES_TMPDIR}

-Djava.io.tmpdir=/var/log/elasticsearch

issue will fix.

Fixed my issue too! Thanks.

lukeshep commented 3 years ago

Cpanel / CENTOS 7.9 kvm [ip-160-153-250-187] v90.0.16 elasticsearch --version Version: 7.9.2, Build: default/rpm/d34da0ea4a966c4e49417f2da2f244e3e97b4e6e/2020-09-23T00:45:33.626720Z, JVM: 15

Worked for me too thank you so much!

ddrager commented 3 years ago

Ran into this issue, more hosts are providing servers with /tmp as noexec. Updating to exec or changing tmpdir fixed it.

aeu commented 2 years ago

I would like to add that @ddrager's comment is the correct one, it is caused by /tmp being mounted noexec.

alainstout commented 1 year ago

This seem to solve it for me too thanks so much after tons of hours seeking why I don't get elasticsearch work for magento centos. :-)

suraydan commented 1 year ago

mude em /etc/elasticsearch/jvm.options

-Djava.io.tmpdir=${ES_TMPDIR} -Djava.io.tmpdir=/var/log/elasticsearch

problema será corrigido.

Solved my problems..

Tks.