AlexRuiz7
commented
7 months ago Can we check the permissions for the folder /var/log/wazuh-indexer?
For example, this is the folder of my AIO.
[root@rhel7 vagrant]# ls -la /var/log/wazuh-indexer
total 872
drwxr-x---. 2 wazuh-indexer wazuh-indexer 4096 Apr 10 09:34 .
drwxr-xr-x. 11 root root 4096 Apr 10 09:34 ..
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 44618 Apr 10 09:37 gc.log
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 2015 Apr 8 09:24 gc.log.00
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 91443 Apr 8 09:40 gc.log.01
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 2015 Apr 8 12:23 gc.log.02
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 117841 Apr 8 13:05 gc.log.03
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 2015 Apr 8 14:23 gc.log.04
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 123727 Apr 8 15:14 gc.log.05
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 2015 Apr 9 11:28 gc.log.06
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 113211 Apr 9 12:05 gc.log.07
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 2015 Apr 9 12:06 gc.log.08
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 153049 Apr 9 14:30 gc.log.09
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 2015 Apr 10 09:34 gc.log.10
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 23396 Apr 9 11:28 wazuh-cluster-2024-04-08-1.json.gz
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 16753 Apr 9 11:28 wazuh-cluster-2024-04-08-1.log.gz
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 14628 Apr 10 09:34 wazuh-cluster-2024-04-09-1.json.gz
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 12954 Apr 10 09:34 wazuh-cluster-2024-04-09-1.log.gz
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 30651 Apr 10 09:37 wazuh-cluster.log
-rw-r-----. 1 wazuh-indexer wazuh-indexer 14085 Apr 10 09:35 wazuh-cluster_deprecation.json
-rw-r-----. 1 wazuh-indexer wazuh-indexer 8496 Apr 10 09:35 wazuh-cluster_deprecation.log
-rw-r-----. 1 wazuh-indexer wazuh-indexer 0 Apr 8 09:24 wazuh-cluster_index_indexing_slowlog.json
-rw-r-----. 1 wazuh-indexer wazuh-indexer 0 Apr 8 09:24 wazuh-cluster_index_indexing_slowlog.log
-rw-r-----. 1 wazuh-indexer wazuh-indexer 0 Apr 8 09:24 wazuh-cluster_index_search_slowlog.json
-rw-r-----. 1 wazuh-indexer wazuh-indexer 0 Apr 8 09:24 wazuh-cluster_index_search_slowlog.log
-rw-r--r--. 1 wazuh-indexer wazuh-indexer 64715 Apr 10 09:37 wazuh-cluster_server.json
-rw-r-----. 1 wazuh-indexer wazuh-indexer 0 Apr 8 09:24 wazuh-cluster_task_detailslog.json
-rw-r-----. 1 wazuh-indexer wazuh-indexer 0 Apr 8 09:24 wazuh-cluster_task_detailslog.logAlso, can we have the content of /etc/wazuh-indexer/opensearch.yml ?
Rebits
Mario156090
gand0rf
byalexandrepedrosa
Describe the bug
It has been detected in E2E UX tests - Vulnerability Detection that wazuh-indexer fails to start in case of rebooting the system
After shutting down an AIO instance with wazuh-indexer running, and starting it the next day, wazuh-indexer failed to start. Reviewing the logs we can see that this is produced due to
/var/log/wazuh-indexerdirectory no longer existing, failing the wazuh-indexer process due to it can not access the/var/log/wazuh-indexer/gc.logfileThe issue was solved by creating the
/var/log/wazuh-indexer/gc.logfile and the/var/log/wazuh-indexerdirectory manuallySteps to reproduce
It was not possible to reproduce the deletion of the /var/log/wazuh-indexer directory through system rebooting, although the same error can be replicated deleting manually the complete directory in a working environment
To Reproduce
/var/log/wazuh-indexerdirectoryEvidences
Indexer Logs
``` Apr 8 15:28:02 ip-172-31-87-231 opensearch-dashboards[60492]: {"type":"response","@timestamp":"2024-04-08T15:28:02Z","tags":[],"pid":60492,"method":"post","statusCode":200,"req":{"url":"/api/opensearch-dashboards/suggestions/values/wazuh-states-vulnerabilities","method":"post","headers":{"host":"172.31.87.231:4443","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://172.31.87.231:4443/app/vulnerability-detection","content-type":"application/json","osd-version":"2.10.0","osd-xsrf":"osd-fetch","content-length":"49","origin":"https://172.31.87.231:4443","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.10.0.12","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","referer":"https://172.31.87.231:4443/app/vulnerability-detection"},"res":{"statusCode":200,"responseTime":14,"contentLength":9},"message":"POST /api/opensearch-dashboards/suggestions/values/wazuh-states-vulnerabilities 200 14ms - 9.0B"} Apr 9 09:44:54 ip-172-31-87-231 systemd[1]: Started wazuh-dashboard. Apr 9 09:44:54 ip-172-31-87-231 kernel: [ 1.553469] systemd[1]: Configuration file /lib/systemd/system/wazuh-indexer.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:44:54 ip-172-31-87-231 kernel: [ 1.556862] systemd[1]: Configuration file /etc/systemd/system/wazuh-dashboard.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:44:57 ip-172-31-87-231 systemd-entrypoint[890]: [0.000s][error][logging] Error opening log file '/var/log/wazuh-indexer/gc.log': No such file or directory Apr 9 09:44:57 ip-172-31-87-231 systemd-entrypoint[890]: [0.000s][error][logging] Initialization of output 'file=/var/log/ Apr 8 15:28:02 ip-172-31-87-231 opensearch-dashboards[60492]: {"type":"response","@timestamp":"2024-04-08T15:28:02Z","tags":[],"pid":60492,"method":"post","statusCode":200,"req":{"url":"/api/opensearch-dashboards/suggestions/values/wazuh-states-vulnerabilities","method":"post","headers":{"host":"172.31.87.231:4443","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://172.31.87.231:4443/app/vulnerability-detection","content-type":"application/json","osd-version":"2.10.0","osd-xsrf":"osd-fetch","content-length":"49","origin":"https://172.31.87.231:4443","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.10.0.12","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","referer":"https://172.31.87.231:4443/app/vulnerability-detection"},"res":{"statusCode":200,"responseTime":14,"contentLength":9},"message":"POST /api/opensearch-dashboards/suggestions/values/wazuh-states-vulnerabilities 200 14ms - 9.0B"} Apr 9 09:44:54 ip-172-31-87-231 systemd[1]: Started wazuh-dashboard. Apr 9 09:44:54 ip-172-31-87-231 kernel: [ 1.553469] systemd[1]: Configuration file /lib/systemd/system/wazuh-indexer.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:44:54 ip-172-31-87-231 kernel: [ 1.556862] systemd[1]: Configuration file /etc/systemd/system/wazuh-dashboard.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:44:57 ip-172-31-87-231 systemd-entrypoint[890]: [0.000s][error][logging] Error opening log file '/var/log/wazuh-indexer/gc.log': No such file or directory Apr 9 09:44:57 ip-172-31-87-231 systemd-entrypoint[890]: [0.000s][error][logging] Initialization of output 'file=/var/log/ Apr 8 15:28:02 ip-172-31-87-231 opensearch-dashboards[60492]: {"type":"response","@timestamp":"2024-04-08T15:28:02Z","tags":[],"pid":60492,"method":"post","statusCode":200,"req":{"url":"/api/opensearch-dashboards/suggestions/values/wazuh-states-vulnerabilities","method":"post","headers":{"host":"172.31.87.231:4443","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://172.31.87.231:4443/app/vulnerability-detection","content-type":"application/json","osd-version":"2.10.0","osd-xsrf":"osd-fetch","content-length":"49","origin":"https://172.31.87.231:4443","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.10.0.12","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","referer":"https://172.31.87.231:4443/app/vulnerability-detection"},"res":{"statusCode":200,"responseTime":14,"contentLength":9},"message":"POST /api/opensearch-dashboards/suggestions/values/wazuh-states-vulnerabilities 200 14ms - 9.0B"} Apr 9 09:44:54 ip-172-31-87-231 systemd[1]: Started wazuh-dashboard. Apr 9 09:44:54 ip-172-31-87-231 kernel: [ 1.553469] systemd[1]: Configuration file /lib/systemd/system/wazuh-indexer.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:44:54 ip-172-31-87-231 kernel: [ 1.556862] systemd[1]: Configuration file /etc/systemd/system/wazuh-dashboard.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:44:57 ip-172-31-87-231 systemd-entrypoint[890]: [0.000s][error][logging] Error opening log file '/var/log/wazuh-indexer/gc.log': No such file or directory Apr 9 09:44:57 ip-172-31-87-231 systemd-entrypoint[890]: [0.000s][error][logging] Initialization of output 'file=/var/log/wazuh-indexer/gc.log' using options 'filecount=32,filesize=64m' failed. Apr 9 09:44:57 ip-172-31-87-231 systemd-entrypoint[890]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details. Apr 9 09:44:57 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE Apr 9 09:44:57 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'. Apr 9 09:44:57 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Consumed 2.291s CPU time. Apr 9 09:45:02 ip-172-31-87-231 opensearch-dashboards[412]: {"type":"log","@timestamp":"2024-04-09T09:45:02Z","tags":["info","plugins-system"],"pid":412,"message":"Setting up [48] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,expressions,data,home,apmOss,savedObjects,notificationsDashboards,management,indexPatternManagement,advancedSettings,console,opensearchUiShared,indexManagementDashboards,reportsDashboards,embeddable,dashboard,visualizations,visTypeVega,visTypeTimeline,visTypeTable,visBuilder,visTypeMarkdown,visAugmenter,alertingDashboards,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,dataExplorer,legacyExport,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,securityDashboards,wazuhCore,wazuhCheckUpdates,wazuh,bfetch]"} Apr 9 09:49:59 ip-172-31-87-231 systemd[1]: Configuration file /lib/systemd/system/wazuh-indexer.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:49:59 ip-172-31-87-231 systemd[1]: Configuration file /etc/systemd/system/wazuh-dashboard.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:50:10 ip-172-31-87-231 systemd[1]: Configuration file /lib/systemd/system/wazuh-indexer.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:50:10 ip-172-31-87-231 systemd[1]: Configuration file /etc/systemd/system/wazuh-dashboard.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:50:10 ip-172-31-87-231 systemd[1]: Configuration file /lib/systemd/system/wazuh-indexer.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:50:10 ip-172-31-87-231 systemd[1]: Configuration file /etc/systemd/system/wazuh-dashboard.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:50:17 ip-172-31-87-231 systemd[1]: Configuration file /etc/systemd/system/wazuh-dashboard.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:50:22 ip-172-31-87-231 systemd[1]: Configuration file /lib/systemd/system/wazuh-indexer.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:51:14 ip-172-31-87-231 systemd-entrypoint[9363]: [0.000s][error][logging] Error opening log file '/var/log/wazuh-indexer/gc.log': No such file or directory Apr 9 09:51:14 ip-172-31-87-231 systemd-entrypoint[9363]: [0.000s][error][logging] Initialization of output 'file=/var/log/wazuh-indexer/gc.log' using options 'filecount=32,filesize=64m' failed. Apr 9 09:51:14 ip-172-31-87-231 systemd-entrypoint[9363]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details. Apr 9 09:51:14 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE Apr 9 09:51:14 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'. Apr 9 09:51:14 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Consumed 2.111s CPU time. Apr 9 09:51:17 ip-172-31-87-231 systemd[1]: Configuration file /lib/systemd/system/wazuh-indexer.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:51:17 ip-172-31-87-231 systemd[1]: Configuration file /etc/systemd/system/wazuh-dashboard.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:51:18 ip-172-31-87-231 systemd[1]: Configuration file /lib/systemd/system/wazuh-indexer.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:51:18 ip-172-31-87-231 systemd[1]: Configuration file /etc/systemd/system/wazuh-dashboard.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Apr 9 09:53:44 ip-172-31-87-231 systemd-entrypoint[10119]: [0.000s][error][logging] Error opening log file '/var/log/wazuh-indexer/gc.log': No such file or directory Apr 9 09:53:44 ip-172-31-87-231 systemd-entrypoint[10119]: [0.000s][error][logging] Initialization of output 'file=/var/log/wazuh-indexer/gc.log' using options 'filecount=32,filesize=64m' failed. Apr 9 09:53:44 ip-172-31-87-231 systemd-entrypoint[10119]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details. Apr 9 09:53:44 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE Apr 9 09:53:44 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'. Apr 9 09:53:44 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Consumed 1.902s CPU time. Apr 9 09:54:40 ip-172-31-87-231 systemd-entrypoint[10277]: [0.000s][error][logging] Error opening log file '/var/log/wazuh-indexer/gc.log': Permission denied Apr 9 09:54:40 ip-172-31-87-231 systemd-entrypoint[10277]: [0.000s][error][logging] Initialization of output 'file=/var/log/wazuh-indexer/gc.log' using options 'filecount=32,filesize=64m' failed. Apr 9 09:54:40 ip-172-31-87-231 systemd-entrypoint[10277]: Could not rename log file '/var/log/wazuh-indexer/gc.log' to '/var/log/wazuh-indexer/gc.log.00' (Permission denied). Apr 9 09:54:40 ip-172-31-87-231 systemd-entrypoint[10277]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details. Apr 9 09:54:40 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE Apr 9 09:54:40 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'. Apr 9 09:54:40 ip-172-31-87-231 systemd[1]: wazuh-indexer.service: Consumed 1.980s CPU time. Apr 9 09:55:14 ip-172-31-87-231 systemd-entrypoint[10308]: Could not rename log file '/var/log/wazuh-indexer/gc.log' to '/var/log/wazuh-indexer/gc.log.00' (Permission denied). Apr 9 09:55:15 ip-172-31-87-231 systemd-entrypoint[10308]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.10.0.jar) Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Unable to create file /var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation.log Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@1fc32e4f[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=rw-r-----, fileOwner=null]] Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@1fc32e4f[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=rw-r-----, fileOwner=null]] Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Unable to create file /var/log/wazuh-indexer/wazuh-indexer-cluster_task_detailslog.json Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_task_detailslog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@66c92293[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_task_detailslog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={%notEmpty{"type": "%OpenSearchMessageField{type}"}, "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, "taskId": "%OpenSearchMessageField{taskId}"}%notEmpty{, "action": "%OpenSearchMessageField{action}"}%notEmpty{, "description": "%OpenSearchMessageField{description}"}%notEmpty{, "start_time_millis": "%OpenSearchMessageField{start_time_millis}"}%notEmpty{, "resource_stats": "%OpenSearchMessageField{resource_stats}"}%notEmpty{, "metadata": "%OpenSearchMessageField{metadata}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_task_detailslog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@66c92293[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_task_detailslog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={%notEmpty{"type": "%OpenSearchMessageField{type}"}, "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, "taskId": "%OpenSearchMessageField{taskId}"}%notEmpty{, "action": "%OpenSearchMessageField{action}"}%notEmpty{, "description": "%OpenSearchMessageField{description}"}%notEmpty{, "start_time_millis": "%OpenSearchMessageField{start_time_millis}"}%notEmpty{, "resource_stats": "%OpenSearchMessageField{resource_stats}"}%notEmpty{, "metadata": "%OpenSearchMessageField{metadata}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Unable to create file /var/log/wazuh-indexer/wazuh-indexer-cluster.log Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@22f31dec[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=rw-r-----, fileOwner=null]] Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@22f31dec[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=rw-r-----, fileOwner=null]] Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Unable to create file /var/log/wazuh-indexer/wazuh-indexer-cluster_index_search_slowlog.json Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_index_search_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@76a2ddf3[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_index_search_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={"type": "index_search_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%OpenSearchMessageField{message}"}%notEmpty{, "took": "%OpenSearchMessageField{took}"}%notEmpty{, "took_millis": "%OpenSearchMessageField{took_millis}"}%notEmpty{, "total_hits": "%OpenSearchMessageField{total_hits}"}%notEmpty{, "types": "%OpenSearchMessageField{types}"}%notEmpty{, "stats": "%OpenSearchMessageField{stats}"}%notEmpty{, "search_type": "%OpenSearchMessageField{search_type}"}%notEmpty{, "total_shards": "%OpenSearchMessageField{total_shards}"}%notEmpty{, "source": "%OpenSearchMessageField{source}"}%notEmpty{, "id": "%OpenSearchMessageField{id}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_index_search_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@76a2ddf3[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_index_search_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={"type": "index_search_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%OpenSearchMessageField{message}"}%notEmpty{, "took": "%OpenSearchMessageField{took}"}%notEmpty{, "took_millis": "%OpenSearchMessageField{took_millis}"}%notEmpty{, "total_hits": "%OpenSearchMessageField{total_hits}"}%notEmpty{, "types": "%OpenSearchMessageField{types}"}%notEmpty{, "stats": "%OpenSearchMessageField{stats}"}%notEmpty{, "search_type": "%OpenSearchMessageField{search_type}"}%notEmpty{, "total_shards": "%OpenSearchMessageField{total_shards}"}%notEmpty{, "source": "%OpenSearchMessageField{source}"}%notEmpty{, "id": "%OpenSearchMessageField{id}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Unable to create file /var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog.json Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@29526c05[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={"type": "index_indexing_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%OpenSearchMessageField{message}"}%notEmpty{, "took": "%OpenSearchMessageField{took}"}%notEmpty{, "took_millis": "%OpenSearchMessageField{took_millis}"}%notEmpty{, "doc_type": "%OpenSearchMessageField{doc_type}"}%notEmpty{, "id": "%OpenSearchMessageField{id}"}%notEmpty{, "routing": "%OpenSearchMessageField{routing}"}%notEmpty{, "source": "%OpenSearchMessageField{source}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@29526c05[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={"type": "index_indexing_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%OpenSearchMessageField{message}"}%notEmpty{, "took": "%OpenSearchMessageField{took}"}%notEmpty{, "took_millis": "%OpenSearchMessageField{took_millis}"}%notEmpty{, "doc_type": "%OpenSearchMessageField{doc_type}"}%notEmpty{, "id": "%OpenSearchMessageField{id}"}%notEmpty{, "routing": "%OpenSearchMessageField{routing}"}%notEmpty{, "source": "%OpenSearchMessageField{source}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Unable to create file /var/log/wazuh-indexer/wazuh-indexer-cluster_server.json Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_server.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@7fb95505[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster-%d{yyyy-MM-dd}-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_server.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@7fb95505[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster-%d{yyyy-MM-dd}-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Unable to create file /var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation.json Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@5ce8d869[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={"type": "deprecation", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, "x-opaque-id": "%OpenSearchMessageField{x-opaque-id}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@5ce8d869[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_deprecation-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=OpenSearchJsonLayout{patternLayout={"type": "deprecation", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:opensearch.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, "x-opaque-id": "%OpenSearchMessageField{x-opaque-id}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson } Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Unable to create file /var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog.log Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: ERROR StatusConsoleListener Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@2b76ff4e[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=rw-r-----, fileOwner=null]] Apr 9 09:55:16 ip-172-31-87-231 systemd-entrypoint[10308]: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@4716be8b] unable to create manager for [/var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@2b76ff4e[pattern=/var/log/wazuh-indexer/wazuh-indexer-cluster_index_indexing_slowlog-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=rw-r-----, fileOwner=null]] ```