search

wazuh / wazuh-indexer

Wazuh indexer, the Wazuh search engine
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
6 stars 16 forks source link

[BUG] `wazuh-cluster.log` rotation access denied #205

Closed AlexRuiz7 closed 2 months ago

AlexRuiz7 commented 2 months ago

Description

The daily log file rotation fails due to missing runtime permissions.

ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied
Full log

```log Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation") Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/java.security.AccessController.checkPermission(AccessController.java:1068) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.checkWriteExtended(UnixFileAttributeViews.java:195) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:264) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:299) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.util.FileUtils.defineFilePosixAttributeView(FileUtils.java:181) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.FileManager.defineAttributeView(FileManager.java:216) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.FileManager.createOutputStream(FileManager.java:203) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(RollingFileManager.java:398) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(RollingFileManager.java:308) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:300) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:683) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:641) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:624) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:560) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.Logger.log(Logger.java:163) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2168) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2122) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2105) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1991) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1854) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1288) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.node.Node.(Node.java:428) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.node.Node.(Node.java:401) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.cli.Command.main(Command.java:101) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ```

This problem is also reproducible in OpenSearch (see https://github.com/opensearch-project/OpenSearch/issues/9609). We've put into practice the solution proposed in OpenSearch's forums about this exact error. The results turned positive (see https://github.com/wazuh/wazuh-packages/issues/2139#issuecomment-2049442145).

https://github.com/wazuh/wazuh-packages/issues/2139#issuecomment-2049442145

AlexRuiz7 commented 2 months ago

The solution consists of adding the code below to /etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy and restart the wazuh-indexer.

grant {
  permission java.lang.RuntimePermission "accessUserInformation";
};
[root@rhel7 vagrant]# journalctl --no-pager  -xeu wazuh-indexer
-- Logs begin at Thu 2024-04-11 11:00:16 UTC, end at Thu 2024-04-11 11:01:01 UTC. --
Apr 11 11:00:25 rhel7.localdomain systemd[1]: Starting Wazuh-indexer...
-- Subject: Unit wazuh-indexer.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit wazuh-indexer.service has begun starting up.
Apr 11 11:00:27 rhel7.localdomain systemd-entrypoint[1015]: WARNING: A terminally deprecated method in java.lang.System has been called
Apr 11 11:00:27 rhel7.localdomain systemd-entrypoint[1015]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.10.0.jar)
Apr 11 11:00:27 rhel7.localdomain systemd-entrypoint[1015]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Apr 11 11:00:27 rhel7.localdomain systemd-entrypoint[1015]: WARNING: System::setSecurityManager will be removed in a future release
Apr 11 11:00:28 rhel7.localdomain systemd-entrypoint[1015]: WARNING: A terminally deprecated method in java.lang.System has been called
Apr 11 11:00:28 rhel7.localdomain systemd-entrypoint[1015]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.10.0.jar)
Apr 11 11:00:28 rhel7.localdomain systemd-entrypoint[1015]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Apr 11 11:00:28 rhel7.localdomain systemd-entrypoint[1015]: WARNING: System::setSecurityManager will be removed in a future release
Apr 11 11:00:36 rhel7.localdomain systemd[1]: Started Wazuh-indexer.
-- Subject: Unit wazuh-indexer.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit wazuh-indexer.service has finished starting up.
-- 
-- The start-up result is done.
AlexRuiz7 commented 2 months ago

Update 2024.04.16

We tried this on @Tostti's environment, which runs on Ubuntu 22.04.3 LTS, and the error still persists.

The error didn't reproduce in a RHEL7 environment running on Vagrant.

AlexRuiz7 commented 2 months ago

Update 2024.04.24

We reviewed @Tostti's environment on April 17th, and edited the jvm.options file, removing a reference to an outdated security policy file.

The environment has been working since without errors. Evidences below.

root@tostti:/home/tostti# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-04-18 09:27:18 -03; 5 days ago
       Docs: https://documentation.wazuh.com/
   Main PID: 1018 (java)
      Tasks: 117 (limit: 18885)
     Memory: 2.1G
        CPU: 59min 39.846s
     CGroup: /system.slice/wazuh-indexer.service
             └─1018 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=tru>

abr 18 09:26:46 tostti.com systemd[1]: Starting Wazuh-indexer...
abr 18 09:26:53 tostti.com systemd-entrypoint[1018]: WARNING: A terminally deprecated method in java.lang.System has been called
abr 18 09:26:53 tostti.com systemd-entrypoint[1018]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.8.0.jar)
abr 18 09:26:53 tostti.com systemd-entrypoint[1018]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
abr 18 09:26:53 tostti.com systemd-entrypoint[1018]: WARNING: System::setSecurityManager will be removed in a future release
abr 18 09:26:56 tostti.com systemd-entrypoint[1018]: WARNING: A terminally deprecated method in java.lang.System has been called
abr 18 09:26:56 tostti.com systemd-entrypoint[1018]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.8.0.jar)
abr 18 09:26:56 tostti.com systemd-entrypoint[1018]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
abr 18 09:26:56 tostti.com systemd-entrypoint[1018]: WARNING: System::setSecurityManager will be removed in a future release
abr 18 09:27:18 tostti.com systemd[1]: Started Wazuh-indexer.
root@tostti:/home/tostti# ls -l /var/log/wazuh-indexer/
total 72308
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 49308634 abr 24 08:16 gc.log
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:11 gc.log.00
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    41234 abr 10 10:13 gc.log.01
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:13 gc.log.02
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    38455 abr 10 10:16 gc.log.03
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:16 gc.log.04
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    38058 abr 10 10:20 gc.log.05
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:20 gc.log.06
-rw-r--r-- 1 wazuh-indexer wazuh-indexer   103871 abr 10 10:36 gc.log.07
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:36 gc.log.08
-rw-r--r-- 1 wazuh-indexer wazuh-indexer   922465 abr 10 13:26 gc.log.09
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 10 15:14 gc.log.10
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  1556178 abr 11 09:20 gc.log.11
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 11 09:24 gc.log.12
-rw-r--r-- 1 wazuh-indexer wazuh-indexer   190560 abr 11 10:27 gc.log.13
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 11 10:31 gc.log.14
-rw-r--r-- 1 wazuh-indexer wazuh-indexer   160360 abr 11 11:15 gc.log.15
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 11 11:43 gc.log.16
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  8284785 abr 15 10:47 gc.log.17
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 15 10:47 gc.log.18
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  5423342 abr 17 12:59 gc.log.19
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 17 12:59 gc.log.20
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  7266304 abr 18 09:22 gc.log.21
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 18 09:26 gc.log.22
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    75921 abr 11 00:00 wazuh-cluster-2024-04-10-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    64329 abr 11 00:00 wazuh-cluster-2024-04-10-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    36802 abr 12 00:00 wazuh-cluster-2024-04-11-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    33445 abr 12 00:00 wazuh-cluster-2024-04-11-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     4423 abr 13 00:00 wazuh-cluster-2024-04-12-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     3510 abr 13 00:00 wazuh-cluster-2024-04-12-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     4467 abr 14 00:00 wazuh-cluster-2024-04-13-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     3393 abr 14 00:00 wazuh-cluster-2024-04-13-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     4680 abr 15 00:00 wazuh-cluster-2024-04-14-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     3473 abr 15 00:00 wazuh-cluster-2024-04-14-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    14933 abr 16 00:00 wazuh-cluster-2024-04-15-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    13410 abr 16 00:00 wazuh-cluster-2024-04-15-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     6572 abr 17 00:00 wazuh-cluster-2024-04-16-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     4804 abr 17 00:00 wazuh-cluster-2024-04-16-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    14971 abr 18 00:00 wazuh-cluster-2024-04-17-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    13508 abr 18 00:00 wazuh-cluster-2024-04-17-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    16035 abr 19 00:00 wazuh-cluster-2024-04-18-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    14315 abr 19 00:00 wazuh-cluster-2024-04-18-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4366 abr 20 00:00 wazuh-cluster-2024-04-19-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     3448 abr 20 00:00 wazuh-cluster-2024-04-19-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4385 abr 21 00:00 wazuh-cluster-2024-04-20-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     3477 abr 21 00:00 wazuh-cluster-2024-04-20-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4196 abr 22 00:00 wazuh-cluster-2024-04-21-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     3299 abr 22 00:00 wazuh-cluster-2024-04-21-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4777 abr 23 00:00 wazuh-cluster-2024-04-22-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     3753 abr 23 00:00 wazuh-cluster-2024-04-22-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     5610 abr 24 00:00 wazuh-cluster-2024-04-23-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4278 abr 24 00:00 wazuh-cluster-2024-04-23-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    34058 abr 24 06:42 wazuh-cluster_deprecation.json
-rw-r----- 1 wazuh-indexer wazuh-indexer    17303 abr 24 06:42 wazuh-cluster_deprecation.log
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_index_indexing_slowlog.json
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_index_indexing_slowlog.log
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_index_search_slowlog.json
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_index_search_slowlog.log
-rw-r----- 1 wazuh-indexer wazuh-indexer    20525 abr 24 08:15 wazuh-cluster.log
-rw-r----- 1 wazuh-indexer wazuh-indexer    58392 abr 24 08:15 wazuh-cluster_server.json
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_task_detailslog.json
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_task_detailslog.log