search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.71k stars 24.67k forks source link

ES 8.0 RPM Install - libjli.so cannot be located launching scripts/binaries from the bin directory - RHEL 8 #84380

Open qadv opened 2 years ago

qadv commented 2 years ago

Elasticsearch Version

8.0

Installed Plugins

No response

Java Version

bundled

OS Version

Linux 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

I am deploying a new ES cluster running 8.0 using RHEL 8.5. Fairly vanilla deployment and Elasticsearch is running on the first node with auto-security used, no JVM installed (using built-in). SELINUX is permissive.

Elastic is launching using systemd and I am able to query the cluster information:

{
  "name" : "es-mstr-01.corp.nklab.com.au",
  "cluster_name" : "nkl-prd-es-clr-01",
  "cluster_uuid" : "zpb0mWSGSVSVvByeZ-fl4w",
  "version" : {
    "number" : "8.0.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "1b6a7ece17463df5ff54a3e1302d825889aa1161",
    "build_date" : "2022-02-03T16:47:57.507843096Z",
    "build_snapshot" : false,
    "lucene_version" : "9.0.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

When I attempt to run any of the scripts in the bin folder to generate tokens for adding the additional nodes I receive the following:

[template@es-mstr-01 ~]$ cd /usr/share/elasticsearch/bin
[template@es-mstr-01 bin]$ sudo ./elasticsearch-create-enrollment-token -s node
/usr/share/elasticsearch/jdk/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
[template@es-mstr-01 bin]$ ./elasticsearch-create-enrollment-token -s node
./elasticsearch-env: line 83: /usr/share/elasticsearch/jdk/bin/java: Operation not permitted

Additionally, the following output can be seen when attempting to locate libjli.so:

[template@es-mstr-01 bin]$ locate libjli.so
/usr/share/elasticsearch/jdk/lib/libjli.so
[template@es-mstr-01 bin]$ sudo ldd /usr/share/elasticsearch/jdk/bin/java
    linux-vdso.so.1 (0x00007ffeb3bf4000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f1f7e6bf000)
    libjli.so => /usr/share/elasticsearch/jdk/bin/../lib/libjli.so (0x00007f1f7e4ae000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1f7e28e000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f1f7e08a000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f1f7dcc5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1f7ead9000)
[template@es-mstr-01 bin]$

Any ideas would be greatly appreciated. I have also attempted to add the path using ldconfig but this made no difference.

First lodged in community forums here

Steps to Reproduce

[template@es-mstr-01 ~]$ cd /usr/share/elasticsearch/bin
[template@es-mstr-01 bin]$ sudo ./elasticsearch-create-enrollment-token -s node
/usr/share/elasticsearch/jdk/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
[template@es-mstr-01 bin]$ ./elasticsearch-create-enrollment-token -s node
./elasticsearch-env: line 83: /usr/share/elasticsearch/jdk/bin/java: Operation not permitted

Logs (if relevant)

No response

elasticmachine commented 2 years ago

Pinging @elastic/es-delivery (Team:Delivery)

elasticmachine commented 2 years ago

Pinging @elastic/es-security (Team:Security)

pugnascotia commented 2 years ago

Is it possible to run Java on its own? (I'm expecting no, but it's worth checking):

./jdk/bin/java -version

What are the permissions on the directories and files?

ls -l /usr/share/elasticsearch
ls -l /usr/share/elasticsearch/lib

Have you tried disabling SELinux entirely? That would tell us whether the issue lies with SELinux, even if it is in permissive mode.

qadv commented 2 years ago

Is it possible to run Java on its own? (I'm expecting no, but it's worth checking):

No Java was installed, I have installed the latest RHEL 8.5 repo JDK and this made no difference.

[nkelly@es-mstr-01 ~]$ java --version
openjdk 17.0.2 2022-01-18 LTS
OpenJDK Runtime Environment 21.9 (build 17.0.2+8-LTS)
OpenJDK 64-Bit Server VM 21.9 (build 17.0.2+8-LTS, mixed mode, sharing)

What are the permissions on the directories and files?

[nkelly@es-mstr-01 ~]$ ls -l /usr/share/elasticsearch
total 860
drwxr-xr-x.  2 root root   4096 Feb 24 11:53 bin
drwxr-xr-x.  9 root root    121 Feb 23 13:50 jdk
drwxr-xr-x.  3 root root   4096 Feb 23 13:50 lib
-rw-r--r--.  1 root root   3860 Feb  4 02:47 LICENSE.txt
drwxr-xr-x. 66 root root   4096 Feb 23 13:50 modules
-rw-rw-r--.  1 root root 858789 Feb  4 02:52 NOTICE.txt
drwxr-xr-x.  2 root root      6 Feb  4 02:55 plugins
-rw-r--r--.  1 root root   2710 Feb  4 02:47 README.asciidoc
[nkelly@es-mstr-01 ~]$ ls -l /usr/share/elasticsearch/lib
total 28488
-rw-r--r--. 1 root root    18427 Jan  6 03:51 ecs-logging-core-1.2.0.jar
-rw-r--r--. 1 root root 13739943 Feb  4 02:49 elasticsearch-8.0.0.jar
-rw-r--r--. 1 root root    27243 Feb  4 02:48 elasticsearch-cli-8.0.0.jar
-rw-r--r--. 1 root root    61254 Feb  4 02:48 elasticsearch-core-8.0.0.jar
-rw-r--r--. 1 root root    53650 Feb  4 02:48 elasticsearch-geo-8.0.0.jar
-rw-r--r--. 1 root root    45227 Feb  4 02:52 elasticsearch-launchers-8.0.0.jar
-rw-r--r--. 1 root root  1770639 Feb  4 02:50 elasticsearch-log4j-8.0.0.jar
-rw-r--r--. 1 root root    26653 Feb  4 02:48 elasticsearch-lz4-8.0.0.jar
-rw-r--r--. 1 root root    14322 Feb  4 02:48 elasticsearch-plugin-classloader-8.0.0.jar
-rw-r--r--. 1 root root    19016 Feb  4 02:48 elasticsearch-secure-sm-8.0.0.jar
-rw-r--r--. 1 root root   174548 Feb  4 02:48 elasticsearch-x-content-8.0.0.jar
-rw-r--r--. 1 root root   114165 May 11  2020 HdrHistogram-2.1.9.jar
-rw-r--r--. 1 root root  1159086 May 11  2020 hppc-0.8.1.jar
-rw-r--r--. 1 root root   349273 May 11  2020 jackson-core-2.10.4.jar
-rw-r--r--. 1 root root    58567 May 11  2020 jackson-dataformat-cbor-2.10.4.jar
-rw-r--r--. 1 root root    90817 May 11  2020 jackson-dataformat-smile-2.10.4.jar
-rw-r--r--. 1 root root    46788 May 11  2020 jackson-dataformat-yaml-2.10.4.jar
-rw-r--r--. 1 root root    16319 Feb  4 02:48 java-version-checker-8.0.0.jar
-rw-r--r--. 1 root root  1756400 Nov 24 19:22 jna-5.10.0.jar
-rw-r--r--. 1 root root    78074 May 11  2020 jopt-simple-5.0.2.jar
-rw-r--r--. 1 root root    26669 Jan  6 03:51 log4j2-ecs-layout-1.2.0.jar
-rw-r--r--. 1 root root   301872 Jan  5 19:38 log4j-api-2.17.1.jar
-rw-r--r--. 1 root root  1847266 Jan  6 03:51 lucene-analysis-common-9.0.0.jar
-rw-r--r--. 1 root root   491440 Jan  6 03:51 lucene-backward-codecs-9.0.0.jar
-rw-r--r--. 1 root root  3452680 Jan  6 03:51 lucene-core-9.0.0.jar
-rw-r--r--. 1 root root    96925 Jan  6 03:51 lucene-grouping-9.0.0.jar
-rw-r--r--. 1 root root   265536 Jan  6 03:51 lucene-highlighter-9.0.0.jar
-rw-r--r--. 1 root root   154128 Jan  6 03:51 lucene-join-9.0.0.jar
-rw-r--r--. 1 root root    47747 Jan  6 03:51 lucene-memory-9.0.0.jar
-rw-r--r--. 1 root root    95553 Jan  6 03:51 lucene-misc-9.0.0.jar
-rw-r--r--. 1 root root   493728 Jan  6 03:51 lucene-queries-9.0.0.jar
-rw-r--r--. 1 root root   374128 Jan  6 03:51 lucene-queryparser-9.0.0.jar
-rw-r--r--. 1 root root   242158 Jan  6 03:51 lucene-sandbox-9.0.0.jar
-rw-r--r--. 1 root root   312790 Jan  6 03:51 lucene-spatial3d-9.0.0.jar
-rw-r--r--. 1 root root   235638 Jan  6 03:51 lucene-suggest-9.0.0.jar
-rw-r--r--. 1 root root   682804 Jul  2  2021 lz4-java-1.8.0.jar
-rw-r--r--. 1 root root   309001 May 11  2020 snakeyaml-1.26.jar
-rw-r--r--. 1 root root    51208 May 11  2020 t-digest-3.2.jar
drwxr-xr-x. 7 root root      101 Feb 23 13:50 tools

Have you tried disabling SELinux entirely? That would tell us whether the issue lies with SELinux, even if it is in permissive mode.

Yes, this made no difference.

mark-vieira commented 2 years ago

No Java was installed, I have installed the latest RHEL 8.5 repo JDK and this made no difference.

Yes, but Elasticsearch is not configured to use that JDK. By default it uses the JDK bundled with Elasticsearch itself. So what we actually want to test is /usr/share/elasticsearch/jdk/bin/java -version. Alternatively you can set ES_JAVA_HOME to the location of the JDK you installed form the RHEL repo and see if that works.

qadv commented 2 years ago

Yes, but Elasticsearch is not configured to use that JDK. By default it uses the JDK bundled with Elasticsearch itself. So what we actually want to test is /usr/share/elasticsearch/jdk/bin/java -version.

[nkelly@es-mstr-01 ~]$ /usr/share/elasticsearch/jdk/bin/java -version
-bash: /usr/share/elasticsearch/jdk/bin/java: Operation not permitted
[nkelly@es-mstr-01 ~]$ sudo /usr/share/elasticsearch/jdk/bin/java -version
/usr/share/elasticsearch/jdk/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

Alternatively you can set ES_JAVA_HOME to the location of the JDK you installed from the RHEL repo and see if that works.

[template@es-mstr-01 bin]$ export ES_JAVA_HOME=/usr/bin/java
[template@es-mstr-01 bin]$ sh ./elasticsearch-create-enrollment-token -s node
could not find java in ES_JAVA_HOME at /usr/bin/java/bin/java
[template@es-mstr-01 bin]$ export ES_JAVA_HOME=/usr/bin
[template@es-mstr-01 bin]$ sh ./elasticsearch-create-enrollment-token -s node
could not find java in ES_JAVA_HOME at /usr/bin/bin/java
[template@es-mstr-01 bin]$ export ES_JAVA_HOME=/usr
[template@es-mstr-01 bin]$ sh ./elasticsearch-create-enrollment-token -s node

ERROR: could not write file [/etc/elasticsearch/users_roles]

[template@es-mstr-01 bin]$ sudo sh ./elasticsearch-create-enrollment-token -s node
/usr/share/elasticsearch/jdk/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

I also updated the systemd ES_JAVA_HOME path and confirmed that elasticsearch still starts correctly.

mark-vieira commented 2 years ago

I also updated the systemd ES_JAVA_HOME path and confirmed that elasticsearch still starts correctly.

I'm a bit confused. It doesn't seem as though that worked. ES_JAVA_HOME should point to the location of the JDK, not just the java binary. Also, since it seems you need elevated permissions for some of these actions you'll need to pass -E to sudo for the ES_JAVA_HOME environment variable to persist.

qadv commented 2 years ago

Apologies for delay.

The following has enabled me to use the binaries:

ES_JAVA_HOME set to /usr after installing java-17-openjdk and executing the binaries with sudo -E.

It would be nice to be able to just use the bundled JRE for both running elastic search and using the binaries.

mark-vieira commented 2 years ago

So as I understand it, Elasticsearch itself managed by systemd starts up and runs fine using the bundled JDK but when running any of the CLI tools is when you find the issue? Sounds like just a file permissions issue. @pugnascotia maybe we need to tweak something here so that folks other than the elasticsearch user can utilize the bundled JDK?

pugnascotia commented 2 years ago

I'll see if I can replicate this.

bjmaynard01 commented 2 years ago

I am running into the exact same issue on Rocky Linux 8.6 while trying to setup ES 8.3 from the yum repo. It installs and I can start the service with no issue without installing java at all, but as soon as I try to execute a script from the /usr/share/elasticsearch/bin directory it says the shared libraries are not there.

Update: Mine is, for the moment at least, temporarily fixed. I installed ES without applying the RHEL STIG first and it works as expected. Something in the STIG is causing the issue for ES, have not identified it yet.

qadv commented 2 years ago

So after some digging, I found that my issue was related to the File Access Policy Daemon blocking access to the files.

https://www.redhat.com/en/blog/stop-unauthorized-applications-rhel-8s-file-access-policy-daemon

bjmaynard01 commented 2 years ago

So after some digging, I found that my issue was related to the File Access Policy Daemon blocking access to the files.

https://www.redhat.com/en/blog/stop-unauthorized-applications-rhel-8s-file-access-policy-daemon

Thank you, it was the same issue for me. Once I reverted/tweaked those STIG settings, this problem went away.