search

wazuh / wazuh-indexer

Wazuh indexer, the Wazuh search engine
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
11 stars 19 forks source link

[BUG] Installation of indexer fails #269

Closed HachimanSec closed 4 months ago

HachimanSec commented 5 months ago

Describe the bug Installation via sudo bash wazuh-install.sh -a fails at installation of the Indexer.

The error in journalctl is: journalctl.log wazuh-install.log

Jun 14 10:10:48 wazuh systemd[1]: Starting Wazuh-indexer...
░░ Subject: A start job for unit wazuh-indexer.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit wazuh-indexer.service has begun execution.
░░ 
░░ The job identifier is 4723.
Jun 14 10:10:50 wazuh systemd-entrypoint[135304]: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 14 10:10:50 wazuh systemd-entrypoint[135304]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.10.0.jar)
Jun 14 10:10:50 wazuh systemd-entrypoint[135304]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jun 14 10:10:50 wazuh systemd-entrypoint[135304]: WARNING: System::setSecurityManager will be removed in a future release
Jun 14 10:10:50 wazuh systemd-entrypoint[135304]: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 14 10:10:50 wazuh systemd-entrypoint[135304]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.10.0.jar)

It seems to be somewhat similiar to https://github.com/wazuh/wazuh/issues/22122 - but I haven't found a solution there.

To Reproduce Steps to reproduce the behavior: Call sudo bash wazuh-install.sh -a on Ubuntu 22.04 Latest apt update and upgrade has been done.

Expected behavior An all in one installation as described here: https://documentation.wazuh.com/current/quickstart.html

Plugins

Screenshots

Host/Environment (please complete the following information):

cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Additional context I also posted on Slack to see if anyone experienced this https://wazuh.slack.com/archives/C0A933R8E/p1718361281257639

EDIT 2❗Executing CIS Build Script L1 Server kills the wazuh-indexer with the described error.

EDIT 1 ❗It appears that the problem occurs once the CIS Build script has been executed. I have setup a new Ubuntu 20.04 server and installed the all-in-one package. It worked. After I executed the build script from CIS for server L2 the indexer fails to start.

HachimanSec commented 5 months ago

The output after applying CIS build script Server L1: 


14/06/2024 12:11:08 INFO: --- Wazuh indexer ---
14/06/2024 12:11:08 INFO: Starting Wazuh indexer installation.
Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: wazuh-indexer 0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded. Need to get 759 MB of archives. After this operation, 1,050 MB of additional disk space will be used. Get:1 https://packages.wazuh.com/4.x/apt stable/main amd64
wazuh-indexer amd64 4.8.0-1 [759 MB] Fetched 759 MB in 5s (168 MB/s) Selecting previously unselected package wazuh-indexer.^M (Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 41311 files and directories currently installed.)^M Preparing to unpack .../wazuh-indexer_4.8.0-1_amd64.deb ...^M Creating wazuh-indexer group... OK^M Creating wazuh-indexer user... OK^M Unpacking wazuh-indexer (4.8.0-1) ...^M Setting up wazuh-indexer (4.8.0-1) ...^M Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore^M Processing triggers for libc-bin (2.35-0ubuntu3.8) ...^M NEEDRESTART-VER: 3.5 NEEDRESTART-KCUR: 5.15.0-105-generic NEEDRESTART-KEXP: 5.15.0-112-generic NEEDRESTART-KSTA: 3 NEEDRESTART-SVC: dbus.service NEEDRESTART-SVC: getty@tty1.service NEEDRESTART-SVC: networkd-dispatcher.service NEEDRESTART-SVC: systemd-logind.service NEEDRESTART-SVC: unattended-upgrades.service
14/06/2024 12:11:53 INFO: Wazuh indexer installation finished.
14/06/2024 12:11:53 INFO: Wazuh indexer post-install configuration finished.
14/06/2024 12:11:53 INFO: Starting service wazuh-indexer.
Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service → /lib/systemd/system/wazuh-indexer.service.
Job for wazuh-indexer.service failed because the control process exited with error code.
See "systemctl status wazuh-indexer.service" and "journalctl -xeu wazuh-indexer.service" for details.
14/06/2024 12:12:03 ERROR: wazuh-indexer could not be started.
Jun 14 12:11:54 wazuh-main systemd[1]: Starting Wazuh-indexer...
Jun 14 12:11:56 wazuh-main systemd-entrypoint[37534]: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 14 12:11:56 wazuh-main systemd-entrypoint[37534]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.10.0.jar)
Jun 14 12:11:56 wazuh-main systemd-entrypoint[37534]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jun 14 12:11:56 wazuh-main systemd-entrypoint[37534]: WARNING: System::setSecurityManager will be removed in a future release
Jun 14 12:11:56 wazuh-main systemd-entrypoint[37534]: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 14 12:11:56 wazuh-main systemd-entrypoint[37534]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.10.0.jar)
Jun 14 12:11:56 wazuh-main systemd-entrypoint[37534]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jun 14 12:11:56 wazuh-main systemd-entrypoint[37534]: WARNING: System::setSecurityManager will be removed in a future release
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]: fatal error in thread [main], exiting
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:47)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.systemd.Libsystemd.<clinit>(Libsystemd.java:46)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:126)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:137)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.plugins.ClusterPlugin.onNodeStarted(ClusterPlugin.java:102)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.node.Node.lambda$start$28(Node.java:1439)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.node.Node.start(Node.java:1439)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.cli.Command.main(Command.java:101)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /tmp/opensearch-12518441532525806421/jna14463842273615614935.tmp: /tmp/opensearch-12518441532525806421/jna14463842273615614935.tmp: failed to map segment from shared object [in thread "main"]
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2394)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.lang.Runtime.load0(Runtime.java:755)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.lang.System.load(System.java:1953)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at com.sun.jna.Native.<clinit>(Native.java:195)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.lang.Class.forName0(Native Method)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.lang.Class.forName(Class.java:375)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.Natives.<clinit>(Natives.java:60)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:123)

Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.lang.Runtime.load0(Runtime.java:755)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.lang.System.load(System.java:1953)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at com.sun.jna.Native.<clinit>(Native.java:195)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.lang.Class.forName0(Native Method)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at java.base/java.lang.Class.forName(Class.java:375)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.Natives.<clinit>(Natives.java:60)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:123)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]:         ... 7 more
Jun 14 12:12:03 wazuh-main systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Jun 14 12:12:03 wazuh-main systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Jun 14 12:12:03 wazuh-main systemd[1]: Failed to start Wazuh-indexer.
Jun 14 12:12:03 wazuh-main systemd[1]: wazuh-indexer.service: Consumed 32.598s CPU time.
14/06/2024 12:12:04 INFO: --- Removing existing Wazuh installation ---
14/06/2024 12:12:04 INFO: Removing Wazuh indexer.
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
  wazuh-indexer*
0 upgraded, 0 newly installed, 1 to remove and 1 not upgraded.
After this operation, 1,050 MB disk space will be freed.
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database
... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 42484 files and directories currently installed.)
Removing wazuh-indexer (4.8.0-1) ...
Stopping wazuh-indexer service... OK
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database
... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 41350 files and directories currently installed.)
Purging configuration files for wazuh-indexer (4.8.0-1) ...
Deleting configuration directory... OK
dpkg: warning: while removing wazuh-indexer, directory '/var/log/wazuh-indexer' not empty so not removed
dpkg: warning: while removing wazuh-indexer, directory '/var/lib/wazuh-indexer' not empty so not removed
dpkg: warning: while removing wazuh-indexer, directory '/usr/lib/systemd/system' not empty so not removed
14/06/2024 12:12:07 INFO: Wazuh indexer removed.
14/06/2024 12:12:07 INFO: Installation cleaned. Check the /var/log/wazuh-install.log file to learn more about the issue.
AlexRuiz7 commented 5 months ago

Hi @HachimanSec

The error cause is:

Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]: fatal error in thread [main], exiting
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /tmp/opensearch-12518441532525806421/jna14463842273615614935.tmp: /tmp/opensearch-12518441532525806421/jna14463842273615614935.tmp: failed to map segment from shared object [in thread "main"]

We have experienced problems during the installation of Wazuh Indexer if the Operative System has the noexec flag enabled on the /tmp folder (see https://github.com/wazuh/wazuh-packages/issues/1539). I think this could be the cause of your problem. Another user posted a workaround. It's worth trying. We have already fixed the issue for 4.9.0.

The warnings messages are known and expected, there is nothing wrong there.

AlexRuiz7 commented 4 months ago

Closed due to inactivity.

aLuViAn87 commented 3 days ago

Hi @HachimanSec

The error cause is:

Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]: fatal error in thread [main], exiting
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
Jun 14 12:12:03 wazuh-main systemd-entrypoint[37534]: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /tmp/opensearch-12518441532525806421/jna14463842273615614935.tmp: /tmp/opensearch-12518441532525806421/jna14463842273615614935.tmp: failed to map segment from shared object [in thread "main"]

We have experienced problems during the installation of Wazuh Indexer if the Operative System has the noexec flag enabled on the /tmp folder (see wazuh/wazuh-packages#1539). I think this could be the cause of your problem. Another user posted a workaround. It's worth trying. We have already fixed the issue for 4.9.0.

The warnings messages are known and expected, there is nothing wrong there.

This was my exact problem, as the server was pre-hardened, all of the tmp filesystems have nodev,noexec flags on mount.

AlexRuiz7 commented 3 days ago

We are working on that on #501