search

wazuh / wazuh-indexer

Wazuh indexer, the Wazuh search engine
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
11 stars 21 forks source link

Wazuh-indexer error: "index template ss4o_metrics_template has index patterns ss4o_metrics " #171

Closed dupuju closed 9 months ago

dupuju commented 9 months ago

Deployment requirements

Component Installation Type OS
Indexer Quickstart - Amazon Linux 2023 x86_64
Server Same as indexer, all-in-one - -
Dashboard Same as indexer, all-in-one - -
Agent Installing Wazuh agents - Windows 11 x86_64, Debian 11 x86_64

Description

upgrading Wazuh indexer to 4.8 🔴

Encountered an issue, when upgrading wazuh-indexer for release test Release 4.8.0 - Beta 2 - E2E UX tests - Central components upgrade #22110 https://github.com/wazuh/wazuh/issues/22110

systemctl start wazuh-indexer[root@ip-172-31-42-25 ec2-user]# systemctl enable wazuh-indexer
[root@ip-172-31-42-25 ec2-user]# systemctl start wazuh-indexer
[root@ip-172-31-42-25 ec2-user]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; preset: disabled)
     Active: active (running) since Tue 2024-02-27 12:49:17 UTC; 18s ago
       Docs: https://documentation.wazuh.com
   Main PID: 66189 (java)
      Tasks: 69 (limit: 9375)
     Memory: 4.2G
        CPU: 56.006s
     CGroup: /system.slice/wazuh-indexer.service
             └─66189 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=>

Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(Priori>
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpe>
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.lang.Thread.run(Thread.java:833)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
lines 1-21/21 (END)

Further logs on this error:

[root@ip-172-31-42-25 wazuh-indexer]# ls
gc.log     gc.log.03                           wazuh-cluster-2024-02-27-1.log.gz   wazuh-cluster_deprecation.json             wazuh-cluster_index_search_slowlog.json  wazuh-cluster_task_detailslog.log
gc.log.00  gc.log.04                           wazuh-cluster-2024-02-28-1.json.gz  wazuh-cluster_deprecation.log              wazuh-cluster_index_search_slowlog.log
gc.log.01  wazuh-cluster-2024-02-27-1.json.gz  wazuh-cluster-2024-02-28-1.log.gz   wazuh-cluster_index_indexing_slowlog.json  wazuh-cluster_server.json
gc.log.02  wazuh-cluster-2024-02-27-1.log      wazuh-cluster.log                   wazuh-cluster_index_indexing_slowlog.log   wazuh-cluster_task_detailslog.json
[root@ip-172-31-42-25 wazuh-indexer]# cd wazuh-cluster-2024-02-27-1.log
bash: cd: wazuh-cluster-2024-02-27-1.log: Not a directory
[root@ip-172-31-42-25 wazuh-indexer]# cat wazuh-cluster-2024-02-27-1.log
.....
[2024-02-27T12:49:18,102][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:18,110][ERROR][o.o.b.Bootstrap          ] [node-1] Exception
java.lang.IllegalArgumentException: index template [ss4o_metrics_template] has index patterns [ss4o_metrics-*-*] matching patterns from existing templates [ss4o_metric_template] with patterns (ss4o_metric_template => [ss4o_metrics-*-*]) that have the same priority [1], multiple index templates may not match during index creation, please use a different priority
        at org.opensearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:560) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:493) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2024-02-27T12:49:18,151][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: index template [ss4o_metrics_template] has index patterns [ss4o_metrics-*-*] matching patterns from existing templates [ss4o_metric_template] with patterns (ss4o_metric_template => [ss4o_metrics-*-*]) that have the same priority [1], multiple index templates may not match during index creation, please use a different priority
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.10.0.jar:2.10.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.10.0.jar:2.10.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.10.0.jar:2.10.0]
Caused by: java.lang.IllegalArgumentException: index template [ss4o_metrics_template] has index patterns [ss4o_metrics-*-*] matching patterns from existing templates [ss4o_metric_template] with patterns (ss4o_metric_template => [ss4o_metrics-*-*]) that have the same priority [1], multiple index templates may not match during index creation, please use a different priority
        at org.opensearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:560) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:493) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-02-27T12:49:18,178][INFO ][o.o.s.l.LogTypeService   ] [node-1] Loading builtin types!
[2024-02-27T12:49:18,185][INFO ][o.o.s.l.LogTypeService   ] [node-1] Indexing [418] fieldMappingDocs from logTypes: 23
[2024-02-27T12:49:18,319][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,331][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,331][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,337][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,337][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,337][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,337][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,338][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,338][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,338][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-27T12:49:18,384][INFO ][o.o.s.l.LogTypeService   ] [node-1] Loading builtin types!
[2024-02-27T12:49:18,386][INFO ][o.o.s.l.LogTypeService   ] [node-1] Indexing [418] fieldMappingDocs from logTypes: 23
[2024-02-27T12:49:18,583][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:18,676][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:18,748][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[wazuh-alerts-4.x-2024.02.27/Rh2y3g0lTmKYfKOeGdYkqw]
[2024-02-27T12:49:18,848][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.opensearch-observability/AJFKUYZLQtedoinTtOFgqQ]
[2024-02-27T12:49:18,939][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:19,087][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:19,135][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:19,139][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[wazuh-alerts-4.x-2024.02.27][1], [wazuh-alerts-4.x-2024.02.27][2]]]).
[2024-02-27T12:49:19,170][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:19,205][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/kFSgs0D5Q2qm_ujMlvLZag]
[2024-02-27T12:49:19,247][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:19,284][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opensearch-sap-log-types-config][0]]]).
[2024-02-27T12:49:19,325][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:19,328][INFO ][o.o.s.l.LogTypeService   ] [node-1] Loading builtin types!
[2024-02-27T12:49:19,330][INFO ][o.o.s.l.LogTypeService   ] [node-1] Indexing [418] fieldMappingDocs from logTypes: 23
[2024-02-27T12:49:19,416][INFO ][o.o.s.l.LogTypeService   ] [node-1] Indexing [418] fieldMappingDocs
[2024-02-27T12:49:19,493][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/kFSgs0D5Q2qm_ujMlvLZag]
[2024-02-27T12:49:19,511][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opensearch-sap-log-types-config/kFSgs0D5Q2qm_ujMlvLZag] update_mapping [_doc]
[2024-02-27T12:49:19,555][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:19,601][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/kFSgs0D5Q2qm_ujMlvLZag]
[2024-02-27T12:49:19,621][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opensearch-sap-log-types-config/kFSgs0D5Q2qm_ujMlvLZag] update_mapping [_doc]
[2024-02-27T12:49:19,666][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:20,288][INFO ][o.o.s.l.LogTypeService   ] [node-1] Loaded [418] field mapping docs successfully!
[2024-02-27T12:49:20,318][INFO ][o.o.s.l.LogTypeService   ] [node-1] Indexing [22] customLogTypes
[2024-02-27T12:49:20,385][INFO ][o.o.s.l.LogTypeService   ] [node-1] Loaded [22] customLogType docs successfully!
[2024-02-27T12:49:20,386][INFO ][o.o.s.SecurityAnalyticsPlugin] [node-1] LogType config index successfully created and builtin log types loaded
[2024-02-27T12:49:27,166][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.plugins-ml-config/1PVBgUKbTZq1fGzuXvC1Sw]
[2024-02-27T12:49:27,173][INFO ][o.o.c.m.MetadataCreateIndexService] [node-1] [.plugins-ml-config] creating index, cause [api], templates [], shards [1]/[1]
[2024-02-27T12:49:27,174][INFO ][o.o.c.r.a.AllocationService] [node-1] updating number_of_replicas to [0] for indices [.plugins-ml-config]
[2024-02-27T12:49:27,209][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.plugins-ml-config/1PVBgUKbTZq1fGzuXvC1Sw]
[2024-02-27T12:49:27,233][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:27,272][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-config][0]]]).
[2024-02-27T12:49:27,301][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:49:27,305][INFO ][o.o.m.i.MLIndicesHandler ] [node-1] create index:.plugins-ml-config
[2024-02-27T12:49:27,349][INFO ][o.o.m.c.MLSyncUpCron     ] [node-1] ML configuration initialized successfully
[2024-02-27T12:49:31,713][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing on REST API is enabled.
[2024-02-27T12:49:31,715][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2024-02-27T12:49:31,716][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing on Transport API is enabled.
[2024-02-27T12:49:31,716][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2024-02-27T12:49:31,716][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing of request body is enabled.
[2024-02-27T12:49:31,717][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Bulk requests resolution is disabled during request auditing.
[2024-02-27T12:49:31,718][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Index resolution is enabled during request auditing.
[2024-02-27T12:49:31,724][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Sensitive headers auditing is enabled.
[2024-02-27T12:49:31,725][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing requests from kibanaserver users is disabled.
[2024-02-27T12:49:31,725][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing of external configuration is disabled.
[2024-02-27T12:49:31,725][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing of internal configuration is enabled.
[2024-02-27T12:49:31,725][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing only metadata information for read request is enabled.
[2024-02-27T12:49:31,725][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing will watch {} for read requests.
[2024-02-27T12:49:31,726][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing read operation requests from kibanaserver users is disabled.
[2024-02-27T12:49:31,726][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing only metadata information for write request is enabled.
[2024-02-27T12:49:31,726][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing diffs for write requests is disabled.
[2024-02-27T12:49:31,726][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing write operation requests from kibanaserver users is disabled.
[2024-02-27T12:49:31,726][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing will watch <NONE> for write requests.
[2024-02-27T12:49:31,727][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] .opendistro_security is used as internal security index.
[2024-02-27T12:49:31,727][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Internal index used for posting audit logs is null
[2024-02-27T12:49:31,728][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Hot-reloading of audit configuration is enabled
[2024-02-27T12:49:31,728][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node 'node-1' initialized
[2024-02-27T12:50:17,145][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Performing move cluster state metadata.
[2024-02-27T12:50:17,146][INFO ][o.o.i.i.MetadataService  ] [node-1] ISM config index not exist, so we cancel the metadata migration job.
[2024-02-27T12:51:17,145][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Cancel background move metadata process.
[2024-02-27T12:51:17,146][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Performing move cluster state metadata.
[2024-02-27T12:51:17,146][INFO ][o.o.i.i.MetadataService  ] [node-1] Move metadata has finished.
[2024-02-27T12:54:16,699][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T12:54:17,155][INFO ][o.o.i.i.PluginVersionSweepCoordinator] [node-1] Canceling sweep ism plugin version job
[2024-02-27T12:55:24,737][INFO ][o.o.c.s.ClusterSettings  ] [node-1] updating [cluster.routing.allocation.enable] from [primaries] to [all]
[2024-02-27T12:55:24,739][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T12:59:16,701][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T13:04:16,701][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T13:09:16,702][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T13:14:16,703][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T13:19:16,703][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T13:24:16,704][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T13:27:49,392][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T13:28:02,870][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[IQqRJDZVTRmr5vPZgR7HeA/e38j0ttzSVKqQi-Kg_CWJw]
[2024-02-27T13:28:02,966][INFO ][o.o.c.m.MetadataIndexTemplateService] [node-1] adding template [wazuh] for index patterns [wazuh-alerts-4.x-*, wazuh-archives-4.x-*]
[2024-02-27T13:28:03,000][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T13:29:16,705][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T13:32:35,828][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.kibana_2/zGM36GLOTy2AbksMwBWLIA]
[2024-02-27T13:32:35,842][INFO ][o.o.c.m.MetadataCreateIndexService] [node-1] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1]
[2024-02-27T13:32:35,843][INFO ][o.o.c.r.a.AllocationService] [node-1] updating number_of_replicas to [0] for indices [.kibana_2]
[2024-02-27T13:32:35,867][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.kibana_2/zGM36GLOTy2AbksMwBWLIA]
[2024-02-27T13:32:35,890][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T13:32:35,918][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_2][0]]]).
[2024-02-27T13:32:35,942][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T13:32:36,061][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.kibana_2/zGM36GLOTy2AbksMwBWLIA]
[2024-02-27T13:32:36,106][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.kibana_2/zGM36GLOTy2AbksMwBWLIA] update_mapping [_doc]
[2024-02-27T13:32:36,162][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T13:32:36,190][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.kibana_2/zGM36GLOTy2AbksMwBWLIA]
[2024-02-27T13:32:36,227][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.kibana_2/zGM36GLOTy2AbksMwBWLIA] update_mapping [_doc]
[2024-02-27T13:32:36,278][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T13:32:36,389][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T13:32:36,895][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[OIo8azRMREag-g5P4eq-Rg/S8POnx5USG-Jl12HubRxiw]
[2024-02-27T13:32:37,091][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[Aa6DYu-0QzGWdftdNYbZIg/Qur5UvYhTwGSV1jWdntJDQ]
[2024-02-27T13:32:38,358][INFO ][o.o.c.m.MetadataUpdateSettingsService] [node-1] updating number_of_replicas to [0] for indices [wazuh-monitoring-2024.9w]
[2024-02-27T13:32:38,380][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[wazuh-monitoring-2024.9w/is3YCVTxSO6OFwESnoVK7A]
[2024-02-27T13:32:38,391][INFO ][o.o.c.m.MetadataMappingService] [node-1] [wazuh-monitoring-2024.9w/is3YCVTxSO6OFwESnoVK7A] update_mapping [_doc]
[2024-02-27T13:32:38,446][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-02-27T13:34:16,706][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T13:39:16,706][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2024-02-27T13:44:16,707][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep

journalctl -u wazuh-indexer
..........
Feb 27 10:12:46 ip-172-31-42-25.ec2.internal systemd[1]: Starting wazuh-indexer.service - Wazuh-indexer...
Feb 27 10:12:49 ip-172-31-42-25.ec2.internal systemd-entrypoint[35597]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 27 10:12:49 ip-172-31-42-25.ec2.internal systemd-entrypoint[35597]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2>
Feb 27 10:12:49 ip-172-31-42-25.ec2.internal systemd-entrypoint[35597]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 27 10:12:49 ip-172-31-42-25.ec2.internal systemd-entrypoint[35597]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 10:12:51 ip-172-31-42-25.ec2.internal systemd-entrypoint[35597]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 27 10:12:51 ip-172-31-42-25.ec2.internal systemd-entrypoint[35597]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.8>
Feb 27 10:12:51 ip-172-31-42-25.ec2.internal systemd-entrypoint[35597]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 27 10:12:51 ip-172-31-42-25.ec2.internal systemd-entrypoint[35597]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 10:13:09 ip-172-31-42-25.ec2.internal systemd[1]: Started wazuh-indexer.service - Wazuh-indexer.
Feb 27 12:44:59 ip-172-31-42-25.ec2.internal systemd[1]: Stopping wazuh-indexer.service - Wazuh-indexer...
Feb 27 12:44:59 ip-172-31-42-25.ec2.internal systemd[1]: wazuh-indexer.service: Deactivated successfully.
Feb 27 12:44:59 ip-172-31-42-25.ec2.internal systemd[1]: Stopped wazuh-indexer.service - Wazuh-indexer.
Feb 27 12:44:59 ip-172-31-42-25.ec2.internal systemd[1]: wazuh-indexer.service: Consumed 2min 49.247s CPU time.
Feb 27 12:48:52 ip-172-31-42-25.ec2.internal systemd[1]: Starting wazuh-indexer.service - Wazuh-indexer...
Feb 27 12:48:55 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 27 12:48:55 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2>
Feb 27 12:48:55 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 27 12:48:55 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 12:48:57 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 27 12:48:57 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.1>
Feb 27 12:48:57 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 27 12:48:57 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 12:49:17 ip-172-31-42-25.ec2.internal systemd[1]: Started wazuh-indexer.service - Wazuh-indexer.
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: uncaught exception in thread [main]
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: java.lang.IllegalArgumentException: index template [ss4o_metrics_template] has index patterns [ss4o_metrics-*-*] matching patterns from ex>
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:560)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:493)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(Priori>
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpe>
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.lang.Thread.run(Thread.java:833)
Feb 27 12:49:18 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster_server.json" got access denied (>
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:  java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.security.AccessController.checkPermission(AccessController.java:1068)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.checkWriteExtended(UnixFileAttributeViews.java:195)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:264)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:299)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.util.FileUtils.defineFilePosixAttributeView(FileUtils.java:181)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.appender.FileManager.defineAttributeView(FileManager.java:216)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.appender.FileManager.createOutputStream(FileManager.java:203)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(RollingFileManager.java:398)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(RollingFileManager.java:308)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:300)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:683)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:641)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:624)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:560)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.core.Logger.log(Logger.java:163)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2168)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2122)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2105)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2003)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1870)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1412)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.metadata.MetadataUpdateSettingsService$1.execute(MetadataUpdateSettingsService.java:256)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(Priori>
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpe>
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]:         at java.base/java.lang.Thread.run(Thread.java:833)
Feb 28 00:00:02 ip-172-31-42-25.ec2.internal systemd-entrypoint[66189]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied ("java.la>
lines 41-88
[7]+  Stopped                 journalctl -u wazuh-indexer
.....

[root@ip-172-31-42-25 wazuh-indexer]# grep -i -E -R "error|critical|fatal|warning" /var/log/wazuh-indexer/
....
h-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:09,694][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,433][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,433][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,433][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,434][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,434][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,434][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,434][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,434][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,435][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T10:13:10,435][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:48:57,757][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms3916m, -Xmx3916m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-2475322892435352911, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=2053111808, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:10,463][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,110][ERROR][o.o.b.Bootstrap          ] [node-1] Exception
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,151][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:  at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.10.0.jar:2.10.0]
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,319][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,331][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,331][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,337][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,337][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,337][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,337][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,338][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,338][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@47533818] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
/var/log/wazuh-indexer/wazuh-cluster-2024-02-27-1.log:[2024-02-27T12:49:18,338][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opense
f-galland commented 9 months ago

I just replicated this in my own environment.

Installed 4.7.2 on top of al2023 and then updated to 4.8.0 following the upgrade guide.

Feb 29 18:59:36 localhost systemd-entrypoint[6134]: java.lang.IllegalArgumentException: index template [ss4o_metrics_template] has index patterns [ss4o_metrics-*-*] matching patterns from existing templates [ss4o_metric_template] with patterns (ss4o_metric_template => [ss4o_metrics-*-*]) that have the same priority [1], multiple index templates may not match during index creation, please use a different priority
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:560)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:493)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]:         at java.base/java.lang.Thread.run(Thread.java:833)
Feb 29 18:59:36 localhost systemd-entrypoint[6134]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
f-galland commented 9 months ago

We were able to replicate this by installing opensearch 2.8, and then upgrading to 2.10.

This non-fatal error is a known opensearch issue, so we are moving it to blocked.

AlexRuiz7 commented 9 months ago

We'll add a note to the upgrade guide to manually remove the old templates before the upgrade.

curl -XDELETE "https://opensearch-node1:9200/_index_template/ss4o_*_template"

More info: https://github.com/opensearch-project/observability/issues/1771