Multiple messages in Wazuh indexer `wazuh-cluster.log` file asking to run `securityadmin` after reboot

rauldpm commented 2 years ago

Hi team

Restarting the Wazuh AMI I have seen a strange behavior that I had not seen before in the Wazuh indexer wazuh-cluster.log file, this log show multiple messages indicating that the cluster has not been initialized, found mostly after a reboot.

The WUI is accessible and everything seems to work correctly
This has been found on CentOS 7, Ubuntu Focal, and Amazon Linux 2 for now.

This is the log obtained in CentOS 7: wazuh-cluster.log This is the log obtained in Ubuntu Focal: wazuh-cluster-ubuntu.log

Regards. Raúl.

rauldpm commented 2 years ago

Update report - wazuh-cluster.log analysis

Regarding the error messages about the .opendistro_security index after installing the components, It can see an informative message indicating that the index will not be created and that securityadmin should be used

Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster

[2022-05-27T17:41:58,084][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Node started
[2022-05-27T17:41:58,085][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2022-05-27T17:41:58,086][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] 0 OpenSearch Security modules loaded so far: []
[2022-05-27T17:41:58,089][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Background init thread started. Install default config?: false
[2022-05-27T17:41:58,094][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-05-27T17:41:58,094][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-05-27T17:41:58,094][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-05-27T17:41:58,095][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-05-27T17:41:58,095][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-05-27T17:41:58,095][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-05-27T17:41:58,095][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-05-27T17:41:58,095][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-05-27T17:41:58,095][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)

rauldpm commented 2 years ago

Update report - wazuh-cluster.log analysis

After the reboot, when starting the node, the same INFO message can be found indicating to use securityadmin, followed this time by an error indicating SERVICE_UNAVAILABLE/1/state not recovered / initialized

[2022-05-27T17:46:03,361][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Node started
[2022-05-27T17:46:03,361][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2022-05-27T17:46:03,366][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] 0 OpenSearch Security modules loaded so far: []
[2022-05-27T17:46:03,367][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Background init thread started. Install default config?: false
[2022-05-27T17:46:03,373][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:202) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:188) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:76) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:53) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:194) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:234) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:554) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:211) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:102) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-05-27T17:46:03,448][INFO ][o.o.a.c.HashRing         ] [node-1] Node added: [B-6cGpMuSXG9gLMKrorwzw]

It seem that because of that, the cluster was in RED (only appears once), then changes to GREEN and show multiples messages about Not yet initialized

[2022-05-27T17:46:04,053][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[wazuh-alerts-4.x-2022.05.27][2]]]).
[2022-05-27T17:46:05,263][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:05,287][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:05,290][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:05,291][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:07,569][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:07,571][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:07,574][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:07,577][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:10,086][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:10,088][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:10,090][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:10,091][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:10,980][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-27T17:46:11,523][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing on REST API is enabled.

miguelfdez99 commented 2 years ago

After investigating I realized that this error could be more related to OpenSearch than our script.

I found the following option in the OpenSearch documentation: https://opensearch.org/docs/latest/security-plugin/configuration/yaml/ The opensearch.yml file also contains the plugins.security.allow_default_init_securityindex property. When set to true, the security plugin uses default security settings if an attempt to create the security index fails when OpenSearch launches. Default security settings are stored in YAML files contained in the opensearch-project/security/config directory. By default, this setting is false.

plugins.security.allow_default_init_securityindex: true

I added this line and try the AIO installation: After rebooting the machine only appears the message below the red line.

But this behavior is not always like this, I have tested it a couple of times and I came to realize that sometimes it needs two reboots to make the message Not yet initialized (you may need to run securityadmin) disappear.

Ubuntu 20:

root@ubuntu2004:/home/vagrant# cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -e "Error"
[2022-09-06T08:43:23,356][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms991m, -Xmx991m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-10811804082937580646, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=520093696, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2022-09-06T08:43:27,265][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2022-09-06T08:43:29,208][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T08:43:29,522][ERROR][o.o.p.o.OSGlobals        ] [node-1] Error in static initialization of OSGlobals with exception: java.security.AccessControlException: access denied ("java.io.FilePermission" "/proc/self/task" "read")
[2022-09-06T08:44:05,560][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
[2022-09-06T08:44:07,366][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
[2022-09-06T08:44:10,476][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
[2022-09-06T08:44:17,264][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
[2022-09-06T08:44:25,715][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
[2022-09-06T08:44:48,100][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

After reboot:

[2022-09-06T08:48:30,343][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms991m, -Xmx991m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-3952549123134384867, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=520093696, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]

This is normal behavior but sometimes it might needs a second reboot.

Installation without plugins.security.allow_default_init_securityindex: true

root@ubuntu1804:/home/vagrant# cat  /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -e "Error"
[2022-09-06T07:20:30,225][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms996m, -Xmx996m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-3640983844229863171, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=522190848, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2022-09-06T07:20:34,292][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2022-09-06T07:20:36,048][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,049][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,049][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,049][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,049][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,049][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,049][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,049][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,049][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,050][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:20:36,168][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:21:29,475][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
[2022-09-06T07:21:30,889][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
[2022-09-06T07:21:33,705][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
[2022-09-06T07:21:38,166][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
[2022-09-06T07:21:47,850][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

First reboot

[2022-09-06T07:25:03,059][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms996m, -Xmx996m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-14128101609961101462, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=522190848, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2022-09-06T07:25:13,865][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2022-09-06T07:25:16,591][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:25:18,958][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:18,973][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:18,980][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:18,982][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:21,342][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:21,345][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:21,347][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:21,349][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:21,908][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:23,842][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:23,846][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:23,848][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:25:23,850][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)

Second reboot

[2022-09-06T07:53:20,961][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms996m, -Xmx996m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-8839322984367738948, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=522190848, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2022-09-06T07:53:30,203][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2022-09-06T07:53:32,641][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-09-06T07:53:33,474][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:33,511][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:33,513][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:33,518][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:33,621][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:35,847][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:35,849][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:35,851][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:35,855][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:38,350][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:38,354][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:38,357][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-09-06T07:53:38,359][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)

miguelfdez99 commented 2 years ago

Comparing /var/log/wazuh-indexer/wazuh-cluster.log after rebooting

Current configuration

```` [2022-09-06T10:53:34,088][INFO ][o.o.s.a.r.AuditMessageRouter] [node-1] Closing AuditMessageRouter [2022-09-06T10:53:34,089][INFO ][o.o.s.a.s.SinkProvider ] [node-1] Closing DebugSink [2022-09-06T10:53:34,092][INFO ][o.o.n.Node ] [node-1] stopping ... [2022-09-06T10:53:34,155][INFO ][o.o.n.Node ] [node-1] stopped [2022-09-06T10:53:34,156][INFO ][o.o.n.Node ] [node-1] closing ... [2022-09-06T10:53:34,162][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Closing AuditLogImpl [2022-09-06T10:53:34,165][INFO ][o.o.n.Node ] [node-1] closed [2022-09-06T10:53:47,685][INFO ][o.o.n.Node ] [node-1] version[2.1.0], pid[773], build[rpm/388c80ad94529b1d9aad0a735c4740dce2932a32/2022-06-30T21:31:04.823801692Z], OS[Linux/4.15.0-189-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.3/17.0.3+7] [2022-09-06T10:53:47,692][INFO ][o.o.n.Node ] [node-1] JVM home [/usr/share/wazuh-indexer/jdk], using bundled JDK [true] [2022-09-06T10:53:47,692][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms996m, -Xmx996m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-14538925851172019784, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=522190848, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-06T10:53:48,857][WARN ][stderr ] [node-1] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". [2022-09-06T10:53:48,857][WARN ][stderr ] [node-1] SLF4J: Defaulting to no-operation (NOP) logger implementation [2022-09-06T10:53:48,857][WARN ][stderr ] [node-1] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [2022-09-06T10:53:48,879][INFO ][o.o.s.s.t.SSLConfig ] [node-1] SSL dual mode is disabled [2022-09-06T10:53:48,880][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] OpenSearch Config path is /etc/wazuh-indexer [2022-09-06T10:53:49,276][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] JVM supports TLSv1.3 [2022-09-06T10:53:49,277][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Config directory is /etc/wazuh-indexer/, from there the key- and truststore files are resolved relatively [2022-09-06T10:53:49,692][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Client Provider : JDK [2022-09-06T10:53:49,692][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Server Provider : JDK [2022-09-06T10:53:49,692][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS HTTP Provider : JDK [2022-09-06T10:53:49,693][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2] [2022-09-06T10:53:49,693][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for HTTP layer : [TLSv1.2] [2022-09-06T10:53:50,021][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Clustername: wazuh-cluster [2022-09-06T10:53:50,768][INFO ][o.o.p.c.PluginSettings ] [node-1] Trying to create directory /dev/shm/performanceanalyzer/. [2022-09-06T10:53:50,769][INFO ][o.o.p.c.PluginSettings ] [node-1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600 [2022-09-06T10:53:51,163][INFO ][o.o.i.r.ReindexPlugin ] [node-1] ReindexPlugin reloadSPI called [2022-09-06T10:53:51,171][INFO ][o.o.i.r.ReindexPlugin ] [node-1] Unable to find any implementation for RemoteReindexExtension [2022-09-06T10:53:51,260][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs [2022-09-06T10:53:51,311][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions [2022-09-06T10:53:51,316][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config [2022-09-06T10:53:51,327][INFO ][o.o.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats] [2022-09-06T10:53:51,328][INFO ][o.o.p.PluginsService ] [node-1] loaded module [analysis-common] [2022-09-06T10:53:51,329][INFO ][o.o.p.PluginsService ] [node-1] loaded module [geo] [2022-09-06T10:53:51,329][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-common] [2022-09-06T10:53:51,329][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-geoip] [2022-09-06T10:53:51,330][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-user-agent] [2022-09-06T10:53:51,330][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-expression] [2022-09-06T10:53:51,331][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-mustache] [2022-09-06T10:53:51,332][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-painless] [2022-09-06T10:53:51,333][INFO ][o.o.p.PluginsService ] [node-1] loaded module [mapper-extras] [2022-09-06T10:53:51,334][INFO ][o.o.p.PluginsService ] [node-1] loaded module [opensearch-dashboards] [2022-09-06T10:53:51,335][INFO ][o.o.p.PluginsService ] [node-1] loaded module [parent-join] [2022-09-06T10:53:51,335][INFO ][o.o.p.PluginsService ] [node-1] loaded module [percolator] [2022-09-06T10:53:51,336][INFO ][o.o.p.PluginsService ] [node-1] loaded module [rank-eval] [2022-09-06T10:53:51,336][INFO ][o.o.p.PluginsService ] [node-1] loaded module [reindex] [2022-09-06T10:53:51,336][INFO ][o.o.p.PluginsService ] [node-1] loaded module [repository-url] [2022-09-06T10:53:51,337][INFO ][o.o.p.PluginsService ] [node-1] loaded module [systemd] [2022-09-06T10:53:51,339][INFO ][o.o.p.PluginsService ] [node-1] loaded module [transport-netty4] [2022-09-06T10:53:51,339][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-alerting] [2022-09-06T10:53:51,340][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-anomaly-detection] [2022-09-06T10:53:51,340][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-asynchronous-search] [2022-09-06T10:53:51,341][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-cross-cluster-replication] [2022-09-06T10:53:51,342][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-index-management] [2022-09-06T10:53:51,343][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-job-scheduler] [2022-09-06T10:53:51,343][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-knn] [2022-09-06T10:53:51,344][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-ml] [2022-09-06T10:53:51,344][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-notifications] [2022-09-06T10:53:51,351][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-notifications-core] [2022-09-06T10:53:51,353][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-observability] [2022-09-06T10:53:51,353][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-performance-analyzer] [2022-09-06T10:53:51,355][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-reports-scheduler] [2022-09-06T10:53:51,355][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-security] [2022-09-06T10:53:51,355][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-sql] [2022-09-06T10:53:51,393][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml [2022-09-06T10:53:51,446][INFO ][o.o.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/ (/dev/vda3)]], net usable_space [110.9gb], net total_space [123gb], types [ext4] [2022-09-06T10:53:51,446][INFO ][o.o.e.NodeEnvironment ] [node-1] heap size [996mb], compressed ordinary object pointers [true] [2022-09-06T10:53:51,644][INFO ][o.o.n.Node ] [node-1] node name [node-1], node ID [RDGrznkQRQyzMJp5wORdzA], cluster name [wazuh-cluster], roles [ingest, remote_cluster_client, data, cluster_manager] [2022-09-06T10:53:54,414][WARN ][o.o.s.c.Salt ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes [2022-09-06T10:53:54,463][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. [2022-09-06T10:53:54,465][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration. [2022-09-06T10:53:54,465][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Message routing enabled: false [2022-09-06T10:53:54,529][INFO ][o.o.s.f.SecurityFilter ] [node-1] indices are made immutable. [2022-09-06T10:53:55,100][INFO ][o.o.a.b.ADCircuitBreakerService] [node-1] Registered memory breaker. [2022-09-06T10:53:55,622][INFO ][o.o.m.c.b.MLCircuitBreakerService] [node-1] Registered ML memory breaker. [2022-09-06T10:53:55,941][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,985][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,985][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,985][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,986][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,986][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,986][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,986][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,986][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,986][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,986][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,987][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,987][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,987][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,987][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,996][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,997][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,997][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,998][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,998][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,998][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,998][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,998][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,998][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,998][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:55,999][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection [2022-09-06T10:53:56,313][INFO ][o.o.t.NettyAllocator ] [node-1] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=996mb}] [2022-09-06T10:53:56,448][INFO ][o.o.d.DiscoveryModule ] [node-1] using discovery type [zen] and seed hosts providers [settings] [2022-09-06T10:53:57,121][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2022-09-06T10:53:57,573][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [node-1] PerformanceAnalyzer Enabled: true [2022-09-06T10:53:57,604][INFO ][o.o.n.Node ] [node-1] initialized [2022-09-06T10:53:57,605][INFO ][o.o.n.Node ] [node-1] starting ... [2022-09-06T10:53:57,738][INFO ][o.o.t.TransportService ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300} [2022-09-06T10:53:57,944][INFO ][o.o.c.c.Coordinator ] [node-1] cluster UUID [j3Xgz78SSvOwRtB2s0BAvA] [2022-09-06T10:53:58,092][INFO ][o.o.c.s.MasterService ] [node-1] elected-as-cluster-manager ([1] nodes joined)[{node-1}{RDGrznkQRQyzMJp5wORdzA}{OmtUt_WQSui_QIa8KqzzWg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 32, delta: cluster-manager node changed {previous [], current [{node-1}{RDGrznkQRQyzMJp5wORdzA}{OmtUt_WQSui_QIa8KqzzWg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]} [2022-09-06T10:53:58,145][INFO ][o.o.c.s.ClusterApplierService] [node-1] cluster-manager node changed {previous [], current [{node-1}{RDGrznkQRQyzMJp5wORdzA}{OmtUt_WQSui_QIa8KqzzWg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 2, version: 32, reason: Publication{term=2, version=32} [2022-09-06T10:53:58,150][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Cluster is not recovered yet. [2022-09-06T10:53:58,154][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T10:53:58,176][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Cache cluster manager node onClusterManager time: 1662461638176 [2022-09-06T10:53:58,178][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring. [2022-09-06T10:53:58,195][INFO ][o.o.h.AbstractHttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200} [2022-09-06T10:53:58,196][INFO ][o.o.n.Node ] [node-1] started [2022-09-06T10:53:58,198][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Node started [2022-09-06T10:53:58,199][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster [2022-09-06T10:53:58,214][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Background init thread started. Install default config?: false [2022-09-06T10:53:58,215][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] 0 OpenSearch Security modules loaded so far: [] [2022-09-06T10:53:58,217][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security) org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.1.0.jar:2.1.0] at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.1.0.jar:2.1.0] at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.1.0.jar:2.1.0] at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.1.0.jar:2.1.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:204) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.1.0.0.jar:2.1.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:202) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.1.0.0.jar:2.1.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:202) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.1.0.0.jar:2.1.0.0] at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.1.0.0.jar:2.1.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:202) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:174) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:102) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.1.0.jar:2.1.0] at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.1.0.0.jar:2.1.0.0] at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.1.0.0.jar:2.1.0.0] at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.1.0.0.jar:2.1.0.0] at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.1.0.0.jar:2.1.0.0] at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.1.0.0.jar:2.1.0.0] at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.1.0.0.jar:2.1.0.0] at java.lang.Thread.run(Thread.java:833) [?:?] [2022-09-06T10:53:58,251][INFO ][o.o.c.s.ClusterSettings ] [node-1] updating [plugins.index_state_management.template_migration.control] from [0] to [-1] [2022-09-06T10:53:58,268][INFO ][o.o.a.c.HashRing ] [node-1] Node added: [RDGrznkQRQyzMJp5wORdzA] [2022-09-06T10:53:58,271][INFO ][o.o.a.c.HashRing ] [node-1] Add data node to AD version hash ring: RDGrznkQRQyzMJp5wORdzA [2022-09-06T10:53:58,272][INFO ][o.o.a.c.HashRing ] [node-1] All nodes with known AD version: {RDGrznkQRQyzMJp5wORdzA=ADNodeInfo{version=2.1.0, isEligibleDataNode=true}} [2022-09-06T10:53:58,272][INFO ][o.o.a.c.HashRing ] [node-1] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0 [2022-09-06T10:53:58,272][INFO ][o.o.a.c.HashRing ] [node-1] Build AD version hash ring successfully [2022-09-06T10:53:58,273][INFO ][o.o.a.c.ADDataMigrator ] [node-1] Start migrating AD data [2022-09-06T10:53:58,273][INFO ][o.o.a.c.ADDataMigrator ] [node-1] AD job index doesn't exist, no need to migrate [2022-09-06T10:53:58,273][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Init AD version hash ring successfully [2022-09-06T10:53:58,273][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T10:53:58,293][INFO ][o.o.g.GatewayService ] [node-1] recovered [4] indices into cluster_state [2022-09-06T10:53:58,566][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T10:53:58,757][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T10:53:58,778][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T10:53:58,827][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T10:53:58,855][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T10:53:58,856][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_1][0]]]). [2022-09-06T10:53:58,870][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T10:54:00,193][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:00,208][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:00,211][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:00,213][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:02,577][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:02,580][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:02,582][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:02,585][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:05,077][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:05,081][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:05,083][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:05,085][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-06T10:54:06,361][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing on REST API is enabled. [2022-09-06T10:54:06,362][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. [2022-09-06T10:54:06,362][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing on Transport API is enabled. [2022-09-06T10:54:06,362][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. [2022-09-06T10:54:06,362][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of request body is enabled. [2022-09-06T10:54:06,362][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Bulk requests resolution is disabled during request auditing. [2022-09-06T10:54:06,362][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Index resolution is enabled during request auditing. [2022-09-06T10:54:06,362][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Sensitive headers auditing is enabled. [2022-09-06T10:54:06,362][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing requests from kibanaserver users is disabled. [2022-09-06T10:54:06,362][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of external configuration is disabled. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of internal configuration is enabled. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing only metadata information for read request is enabled. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing will watch {} for read requests. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing read operation requests from kibanaserver users is disabled. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing only metadata information for write request is enabled. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing diffs for write requests is disabled. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing write operation requests from kibanaserver users is disabled. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing will watch for write requests. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] .opendistro_security is used as internal security index. [2022-09-06T10:54:06,363][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Internal index used for posting audit logs is null [2022-09-06T10:54:06,364][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Hot-reloading of audit configuration is enabled [2022-09-06T10:54:06,364][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node 'node-1' initialized [2022-09-06T10:54:08,245][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[], allIndices=[.wazuh], types=[*], originalRequested=[.wazuh], remoteIndices=[]] [Action [indices:admin/get]] [RolesChecked [manage_wazuh_index, own_index, kibana_server]] [2022-09-06T10:54:08,245][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No permissions for [indices:admin/get] [2022-09-06T10:54:08,457][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[], allIndices=[.wazuh-version], types=[*], originalRequested=[.wazuh-version], remoteIndices=[]] [Action [indices:admin/get]] [RolesChecked [manage_wazuh_index, own_index, kibana_server]] [2022-09-06T10:54:08,457][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No permissions for [indices:admin/get] [2022-09-06T10:54:09,024][INFO ][o.o.c.m.MetadataUpdateSettingsService] [node-1] updating number_of_replicas to [0] for indices [wazuh-monitoring-2022.36w] [2022-09-06T10:54:59,031][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Performing move cluster state metadata. [2022-09-06T10:54:59,032][INFO ][o.o.i.i.MetadataService ] [node-1] ISM config index not exist, so we cancel the metadata migration job. [2022-09-06T10:55:59,031][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Cancel background move metadata process. [2022-09-06T10:55:59,032][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Performing move cluster state metadata. [2022-09-06T10:55:59,032][INFO ][o.o.i.i.MetadataService ] [node-1] Move metadata has finished. [2022-09-06T10:56:01,400][INFO ][o.o.c.m.MetadataMappingService] [node-1] [wazuh-alerts-4.x-2022.09.06/Ky7WeGobSViO9Qb_QMcgMQ] update_mapping [_doc] [2022-09-06T10:56:01,451][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration ````

With plugins.security.allow_default_init_securityindex: true

```` [2022-09-06T11:43:13,554][INFO ][o.o.s.a.r.AuditMessageRouter] [node-1] Closing AuditMessageRouter [2022-09-06T11:43:13,558][INFO ][o.o.s.a.s.SinkProvider ] [node-1] Closing DebugSink [2022-09-06T11:43:13,578][INFO ][o.o.n.Node ] [node-1] stopping ... [2022-09-06T11:43:13,775][INFO ][o.o.n.Node ] [node-1] stopped [2022-09-06T11:43:13,775][INFO ][o.o.n.Node ] [node-1] closing ... [2022-09-06T11:43:13,792][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Closing AuditLogImpl [2022-09-06T11:43:13,808][INFO ][o.o.n.Node ] [node-1] closed [2022-09-06T11:43:30,487][INFO ][o.o.n.Node ] [node-1] version[2.1.0], pid[594], build[rpm/388c80ad94529b1d9aad0a735c4740dce2932a32/2022-06-30T21:31:04.823801692Z], OS[Linux/5.4.0-122-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.3/17.0.3+7] [2022-09-06T11:43:30,501][INFO ][o.o.n.Node ] [node-1] JVM home [/usr/share/wazuh-indexer/jdk], using bundled JDK [true] [2022-09-06T11:43:30,502][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms991m, -Xmx991m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5618440898363663856, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=520093696, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-06T11:43:32,910][WARN ][stderr ] [node-1] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". [2022-09-06T11:43:32,913][WARN ][stderr ] [node-1] SLF4J: Defaulting to no-operation (NOP) logger implementation [2022-09-06T11:43:32,913][WARN ][stderr ] [node-1] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [2022-09-06T11:43:32,959][INFO ][o.o.s.s.t.SSLConfig ] [node-1] SSL dual mode is disabled [2022-09-06T11:43:32,961][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] OpenSearch Config path is /etc/wazuh-indexer [2022-09-06T11:43:33,661][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] JVM supports TLSv1.3 [2022-09-06T11:43:33,667][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Config directory is /etc/wazuh-indexer/, from there the key- and truststore files are resolved relatively [2022-09-06T11:43:34,359][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Client Provider : JDK [2022-09-06T11:43:34,359][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Server Provider : JDK [2022-09-06T11:43:34,360][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS HTTP Provider : JDK [2022-09-06T11:43:34,360][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2] [2022-09-06T11:43:34,360][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for HTTP layer : [TLSv1.2] [2022-09-06T11:43:34,860][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Clustername: wazuh-cluster [2022-09-06T11:43:35,675][INFO ][o.o.p.c.PluginSettings ] [node-1] Trying to create directory /dev/shm/performanceanalyzer/. [2022-09-06T11:43:35,679][INFO ][o.o.p.c.PluginSettings ] [node-1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600 [2022-09-06T11:43:36,168][INFO ][o.o.i.r.ReindexPlugin ] [node-1] ReindexPlugin reloadSPI called [2022-09-06T11:43:36,173][INFO ][o.o.i.r.ReindexPlugin ] [node-1] Unable to find any implementation for RemoteReindexExtension [2022-09-06T11:43:36,238][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions [2022-09-06T11:43:36,241][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs [2022-09-06T11:43:36,243][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config [2022-09-06T11:43:36,248][INFO ][o.o.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats] [2022-09-06T11:43:36,248][INFO ][o.o.p.PluginsService ] [node-1] loaded module [analysis-common] [2022-09-06T11:43:36,248][INFO ][o.o.p.PluginsService ] [node-1] loaded module [geo] [2022-09-06T11:43:36,249][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-common] [2022-09-06T11:43:36,249][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-geoip] [2022-09-06T11:43:36,249][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-user-agent] [2022-09-06T11:43:36,249][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-expression] [2022-09-06T11:43:36,249][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-mustache] [2022-09-06T11:43:36,253][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-painless] [2022-09-06T11:43:36,253][INFO ][o.o.p.PluginsService ] [node-1] loaded module [mapper-extras] [2022-09-06T11:43:36,254][INFO ][o.o.p.PluginsService ] [node-1] loaded module [opensearch-dashboards] [2022-09-06T11:43:36,254][INFO ][o.o.p.PluginsService ] [node-1] loaded module [parent-join] [2022-09-06T11:43:36,254][INFO ][o.o.p.PluginsService ] [node-1] loaded module [percolator] [2022-09-06T11:43:36,254][INFO ][o.o.p.PluginsService ] [node-1] loaded module [rank-eval] [2022-09-06T11:43:36,254][INFO ][o.o.p.PluginsService ] [node-1] loaded module [reindex] [2022-09-06T11:43:36,255][INFO ][o.o.p.PluginsService ] [node-1] loaded module [repository-url] [2022-09-06T11:43:36,255][INFO ][o.o.p.PluginsService ] [node-1] loaded module [systemd] [2022-09-06T11:43:36,255][INFO ][o.o.p.PluginsService ] [node-1] loaded module [transport-netty4] [2022-09-06T11:43:36,255][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-alerting] [2022-09-06T11:43:36,255][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-anomaly-detection] [2022-09-06T11:43:36,256][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-asynchronous-search] [2022-09-06T11:43:36,256][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-cross-cluster-replication] [2022-09-06T11:43:36,257][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-index-management] [2022-09-06T11:43:36,257][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-job-scheduler] [2022-09-06T11:43:36,257][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-knn] [2022-09-06T11:43:36,258][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-ml] [2022-09-06T11:43:36,258][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-notifications] [2022-09-06T11:43:36,259][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-notifications-core] [2022-09-06T11:43:36,259][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-observability] [2022-09-06T11:43:36,259][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-performance-analyzer] [2022-09-06T11:43:36,259][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-reports-scheduler] [2022-09-06T11:43:36,259][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-security] [2022-09-06T11:43:36,260][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-sql] [2022-09-06T11:43:36,275][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml [2022-09-06T11:43:36,335][INFO ][o.o.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/ (/dev/vda3)]], net usable_space [110.5gb], net total_space [123gb], types [ext4] [2022-09-06T11:43:36,335][INFO ][o.o.e.NodeEnvironment ] [node-1] heap size [992mb], compressed ordinary object pointers [true] [2022-09-06T11:43:36,424][INFO ][o.o.n.Node ] [node-1] node name [node-1], node ID [xKPC3cHZRRWmd8kY2DUjww], cluster name [wazuh-cluster], roles [ingest, remote_cluster_client, data, cluster_manager] [2022-09-06T11:43:38,749][WARN ][o.o.s.c.Salt ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes [2022-09-06T11:43:38,770][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. [2022-09-06T11:43:38,772][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration. [2022-09-06T11:43:38,772][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Message routing enabled: false [2022-09-06T11:43:38,800][INFO ][o.o.s.f.SecurityFilter ] [node-1] indices are made immutable. [2022-09-06T11:43:39,047][INFO ][o.o.a.b.ADCircuitBreakerService] [node-1] Registered memory breaker. [2022-09-06T11:43:39,271][INFO ][o.o.m.c.b.MLCircuitBreakerService] [node-1] Registered ML memory breaker. [2022-09-06T11:43:39,767][INFO ][o.o.t.NettyAllocator ] [node-1] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=992mb}] [2022-09-06T11:43:39,831][INFO ][o.o.d.DiscoveryModule ] [node-1] using discovery type [zen] and seed hosts providers [settings] [2022-09-06T11:43:40,165][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2022-09-06T11:43:40,474][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [node-1] PerformanceAnalyzer Enabled: true [2022-09-06T11:43:40,494][INFO ][o.o.n.Node ] [node-1] initialized [2022-09-06T11:43:40,498][INFO ][o.o.n.Node ] [node-1] starting ... [2022-09-06T11:43:40,577][INFO ][o.o.t.TransportService ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300} [2022-09-06T11:43:40,782][INFO ][o.o.c.c.Coordinator ] [node-1] cluster UUID [JeW1EaQZQFeHNt28Apr3QQ] [2022-09-06T11:43:40,880][INFO ][o.o.c.s.MasterService ] [node-1] elected-as-cluster-manager ([1] nodes joined)[{node-1}{xKPC3cHZRRWmd8kY2DUjww}{Vi3hjUyfSUeUlz5uuHsPaQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 43, delta: cluster-manager node changed {previous [], current [{node-1}{xKPC3cHZRRWmd8kY2DUjww}{Vi3hjUyfSUeUlz5uuHsPaQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]} [2022-09-06T11:43:40,957][INFO ][o.o.c.s.ClusterApplierService] [node-1] cluster-manager node changed {previous [], current [{node-1}{xKPC3cHZRRWmd8kY2DUjww}{Vi3hjUyfSUeUlz5uuHsPaQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 3, version: 43, reason: Publication{term=3, version=43} [2022-09-06T11:43:40,963][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Cluster is not recovered yet. [2022-09-06T11:43:40,966][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T11:43:40,990][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Cache cluster manager node onClusterManager time: 1662464620990 [2022-09-06T11:43:40,993][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring. [2022-09-06T11:43:41,010][INFO ][o.o.h.AbstractHttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200} [2022-09-06T11:43:41,010][INFO ][o.o.n.Node ] [node-1] started [2022-09-06T11:43:41,012][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Node started [2022-09-06T11:43:41,012][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Will attempt to create index .opendistro_security and default configs if they are absent [2022-09-06T11:43:41,022][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Background init thread started. Install default config?: true [2022-09-06T11:43:41,023][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] 0 OpenSearch Security modules loaded so far: [] [2022-09-06T11:43:41,067][INFO ][o.o.c.s.ClusterSettings ] [node-1] updating [plugins.index_state_management.template_migration.control] from [0] to [-1] [2022-09-06T11:43:41,097][INFO ][o.o.a.c.HashRing ] [node-1] Node added: [xKPC3cHZRRWmd8kY2DUjww] [2022-09-06T11:43:41,100][INFO ][o.o.a.c.HashRing ] [node-1] Add data node to AD version hash ring: xKPC3cHZRRWmd8kY2DUjww [2022-09-06T11:43:41,101][INFO ][o.o.a.c.HashRing ] [node-1] All nodes with known AD version: {xKPC3cHZRRWmd8kY2DUjww=ADNodeInfo{version=2.1.0, isEligibleDataNode=true}} [2022-09-06T11:43:41,101][INFO ][o.o.a.c.HashRing ] [node-1] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0 [2022-09-06T11:43:41,101][INFO ][o.o.a.c.HashRing ] [node-1] Build AD version hash ring successfully [2022-09-06T11:43:41,103][INFO ][o.o.a.c.ADDataMigrator ] [node-1] Start migrating AD data [2022-09-06T11:43:41,103][INFO ][o.o.a.c.ADDataMigrator ] [node-1] AD job index doesn't exist, no need to migrate [2022-09-06T11:43:41,103][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Init AD version hash ring successfully [2022-09-06T11:43:41,103][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T11:43:41,128][INFO ][o.o.g.GatewayService ] [node-1] recovered [4] indices into cluster_state [2022-09-06T11:43:41,130][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Index .opendistro_security already exists [2022-09-06T11:43:41,130][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node started, try to initialize it. Wait for at least yellow cluster state.... [2022-09-06T11:43:41,421][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T11:43:41,585][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T11:43:41,605][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T11:43:41,609][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'config' with /etc/wazuh-indexer/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2022-09-06T11:43:41,664][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id config, skipping update. [2022-09-06T11:43:41,664][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'roles' with /etc/wazuh-indexer/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2022-09-06T11:43:41,669][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id roles, skipping update. [2022-09-06T11:43:41,670][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2022-09-06T11:43:41,675][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id rolesmapping, skipping update. [2022-09-06T11:43:41,676][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2022-09-06T11:43:41,681][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id internalusers, skipping update. [2022-09-06T11:43:41,682][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2022-09-06T11:43:41,688][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id actiongroups, skipping update. [2022-09-06T11:43:41,688][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2022-09-06T11:43:41,691][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id tenants, skipping update. [2022-09-06T11:43:41,691][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true [2022-09-06T11:43:41,704][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T11:43:41,708][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id nodesdn, skipping update. [2022-09-06T11:43:41,709][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true [2022-09-06T11:43:41,713][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id whitelist, skipping update. [2022-09-06T11:43:41,714][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true [2022-09-06T11:43:41,723][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id allowlist, skipping update. [2022-09-06T11:43:41,723][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'audit' with /etc/wazuh-indexer/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2022-09-06T11:43:41,735][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T11:43:41,736][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_1][0]]]). [2022-09-06T11:43:41,750][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration [2022-09-06T11:43:41,760][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id audit, skipping update. [2022-09-06T11:43:41,905][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing on REST API is enabled. [2022-09-06T11:43:41,907][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. [2022-09-06T11:43:41,907][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing on Transport API is enabled. [2022-09-06T11:43:41,907][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. [2022-09-06T11:43:41,907][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of request body is enabled. [2022-09-06T11:43:41,907][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Bulk requests resolution is disabled during request auditing. [2022-09-06T11:43:41,908][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Index resolution is enabled during request auditing. [2022-09-06T11:43:41,908][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Sensitive headers auditing is enabled. [2022-09-06T11:43:41,909][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing requests from kibanaserver users is disabled. [2022-09-06T11:43:41,909][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of external configuration is disabled. [2022-09-06T11:43:41,909][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of internal configuration is enabled. [2022-09-06T11:43:41,909][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing only metadata information for read request is enabled. [2022-09-06T11:43:41,909][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing will watch {} for read requests. [2022-09-06T11:43:41,909][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing read operation requests from kibanaserver users is disabled. [2022-09-06T11:43:41,910][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing only metadata information for write request is enabled. [2022-09-06T11:43:41,910][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing diffs for write requests is disabled. [2022-09-06T11:43:41,910][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing write operation requests from kibanaserver users is disabled. [2022-09-06T11:43:41,910][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing will watch for write requests. [2022-09-06T11:43:41,910][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] .opendistro_security is used as internal security index. [2022-09-06T11:43:41,911][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Internal index used for posting audit logs is null [2022-09-06T11:43:41,912][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Hot-reloading of audit configuration is enabled [2022-09-06T11:43:41,912][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node 'node-1' initialized [2022-09-06T11:43:44,105][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[], allIndices=[.wazuh], types=[*], originalRequested=[.wazuh], remoteIndices=[]] [Action [indices:admin/get]] [RolesChecked [manage_wazuh_index, own_index, kibana_server]] [2022-09-06T11:43:44,105][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No permissions for [indices:admin/get] [2022-09-06T11:43:44,288][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[], allIndices=[.wazuh-version], types=[*], originalRequested=[.wazuh-version], remoteIndices=[]] [Action [indices:admin/get]] [RolesChecked [manage_wazuh_index, own_index, kibana_server]] [2022-09-06T11:43:44,288][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No permissions for [indices:admin/get] [2022-09-06T11:43:44,848][INFO ][o.o.c.m.MetadataUpdateSettingsService] [node-1] updating number_of_replicas to [0] for indices [wazuh-monitoring-2022.36w] ````

This is the part that is different from the current configuration:

[2022-09-06T11:43:41,128][INFO ][o.o.g.GatewayService     ] [node-1] recovered [4] indices into cluster_state
[2022-09-06T11:43:41,130][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Index .opendistro_security already exists
[2022-09-06T11:43:41,130][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node started, try to initialize it. Wait for at least yellow cluster state....
[2022-09-06T11:43:41,421][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2022-09-06T11:43:41,585][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2022-09-06T11:43:41,605][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2022-09-06T11:43:41,609][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'config' with /etc/wazuh-indexer/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-09-06T11:43:41,664][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id config, skipping update.
[2022-09-06T11:43:41,664][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'roles' with /etc/wazuh-indexer/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-09-06T11:43:41,669][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id roles, skipping update.
[2022-09-06T11:43:41,670][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-09-06T11:43:41,675][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id rolesmapping, skipping update.
[2022-09-06T11:43:41,676][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-09-06T11:43:41,681][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id internalusers, skipping update.
[2022-09-06T11:43:41,682][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-09-06T11:43:41,688][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id actiongroups, skipping update.
[2022-09-06T11:43:41,688][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-09-06T11:43:41,691][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id tenants, skipping update.
[2022-09-06T11:43:41,691][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2022-09-06T11:43:41,704][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2022-09-06T11:43:41,708][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id nodesdn, skipping update.
[2022-09-06T11:43:41,709][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2022-09-06T11:43:41,713][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id whitelist, skipping update.
[2022-09-06T11:43:41,714][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2022-09-06T11:43:41,723][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id allowlist, skipping update.
[2022-09-06T11:43:41,723][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'audit' with /etc/wazuh-indexer/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-09-06T11:43:41,735][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2022-09-06T11:43:41,736][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_1][0]]]).
[2022-09-06T11:43:41,750][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2022-09-06T11:43:41,760][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Index .opendistro_security already contains doc with id audit, skipping update.

miguelfdez99 commented 2 years ago

Update

It seems that adding plugins.security.allow_default_init_securityindex: true to the configuration may be a substitute for calling the securityadmin.sh.

Adding plugins.security.allow_default_init_securityindex: true after starting wazuh-indexer.

root@ubuntu2004:/home/vagrant# curl -k -u admin:admin https://localhost:9200
OpenSearch Security not initialized.root@ubuntu2004:/home/vagrant# nano /etc/wazuh-indexer/opensearch.yml 
root@ubuntu2004:/home/vagrant# systemctl restart wazuh-indexer
root@ubuntu2004:/home/vagrant# curl -k -u admin:admin https://localhost:9200
{
  "name" : "node-1",
  "cluster_name" : "wazuh-cluster",
  "cluster_uuid" : "WG7Nwp7LSzu6VutcX9AVFw",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32",
    "build_date" : "2022-06-30T21:31:04.823801692Z",
    "build_snapshot" : false,
    "lucene_version" : "9.2.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

If plugins.security.allow_default_init_securityindex: true is added after wazuh-indexer is initialize we have to restart it and then the cluster will be up and running.

Adding plugins.security.allow_default_init_securityindex: true before starting wazuh-indexer.

root@ubuntu2004:/home/vagrant# systemctl daemon-reload
root@ubuntu2004:/home/vagrant# systemctl enable wazuh-indexer
Synchronizing state of wazuh-indexer.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable wazuh-indexer
Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service → /lib/systemd/system/wazuh-indexer.service.
root@ubuntu2004:/home/vagrant# systemctl start wazuh-indexer
root@ubuntu2004:/home/vagrant# curl -k -u admin:admin https://localhost:9200
{
  "name" : "node-1",
  "cluster_name" : "wazuh-cluster",
  "cluster_uuid" : "nNbkca_aRvie36hgafGVKA",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32",
    "build_date" : "2022-06-30T21:31:04.823801692Z",
    "build_snapshot" : false,
    "lucene_version" : "9.2.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

There's no need to restart the wazuh-indexer if it is added before starting the wazuh-indexer.

Tests

[ ] AIO
[ ] Distributed
[ ] Distributed with several indexers

miguelfdez99 commented 2 years ago

TESTS

AIO

CentOS7

wazuh-indexer

```` [root@centos7 vagrant]# curl -sO https://packages.wazuh.com/4.3/wazuh-certs-tool.sh [root@centos7 vagrant]# nano config.yml [root@centos7 vagrant]# bash ./wazuh-certs-tool.sh -A 08/09/2022 06:35:55 INFO: Admin certificates created. 08/09/2022 06:35:55 INFO: Wazuh indexer certificates created. 08/09/2022 06:35:55 INFO: Wazuh server certificates created. 08/09/2022 06:35:55 INFO: Wazuh dashboard certificates created. [root@centos7 vagrant]# tar -cvf ./wazuh-certificates.tar -C ./wazuh-certificates/ . ./ ./root-ca.key ./root-ca.pem ./admin-key.pem ./admin.pem ./node-1-key.pem ./node-1.pem ./wazuh-1-key.pem ./wazuh-1.pem ./dashboard-key.pem ./dashboard.pem [root@centos7 vagrant]# rm -rf ./wazuh-certificates [root@centos7 vagrant]# curl -LO https://packages-dev.wazuh.com/staging/yum/wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 395M 100 395M 0 0 3707k 0 0:01:49 0:01:49 --:--:-- 8195k [root@centos7 vagrant]# yum install -y wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm Loaded plugins: fastestmirror Examining wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm: wazuh-indexer-4.4.0-0.40400.20220808.x86_64 Marking wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================== Package Arch Version Repository Size ========================================================================================== Installing: wazuh-indexer x86_64 4.4.0-0.40400.20220808 /wazuh-indexer-4.4.0-0.40400.20220808.x86_64 642 M Transaction Summary ========================================================================================== Install 1 Package Total size: 642 M Installed size: 642 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore Verifying : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Installed: wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 Complete! [root@centos7 vagrant]# nano /etc/wazuh-indexer/opensearch.yml [root@centos7 vagrant]# NODE_NAME=node-1 [root@centos7 vagrant]# mkdir /etc/wazuh-indexer/certs [root@centos7 vagrant]# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem [root@centos7 vagrant]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem [root@centos7 vagrant]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem [root@centos7 vagrant]# chmod 500 /etc/wazuh-indexer/certs [root@centos7 vagrant]# chmod 400 /etc/wazuh-indexer/certs/* [root@centos7 vagrant]# chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs [root@centos7 vagrant]# systemctl daemon-reload [root@centos7 vagrant]# systemctl enable wazuh-indexer Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service to /usr/lib/systemd/system/wazuh-indexer.service. [root@centos7 vagrant]# systemctl start wazuh-indexer [root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200 { "name" : "node-1", "cluster_name" : "wazuh-cluster", "cluster_uuid" : "eIvmGpuKTRixmMS9Sbf-Iw", "version" : { "number" : "7.10.2", "build_type" : "rpm", "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32", "build_date" : "2022-06-30T21:31:04.823801692Z", "build_snapshot" : false, "lucene_version" : "9.2.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } [root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cat/nodes?v ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles cluster_manager name 192.168.121.250 28 96 4 0.07 0.10 0.05 dimr cluster_manager,data,ingest,remote_cluster_client * node-1 ````

wazuh-server

```` [root@centos7 vagrant]# yum install wazuh-manager-4.4.0-0.40400.20220816.x86_64.rpm Loaded plugins: fastestmirror Examining wazuh-manager-4.4.0-0.40400.20220816.x86_64.rpm: wazuh-manager-4.4.0-0.40400.20220816.x86_64 Marking wazuh-manager-4.4.0-0.40400.20220816.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package wazuh-manager.x86_64 0:4.4.0-0.40400.20220816 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================== Package Arch Version Repository Size ========================================================================================== Installing: wazuh-manager x86_64 4.4.0-0.40400.20220816 /wazuh-manager-4.4.0-0.40400.20220816.x86_64 440 M Transaction Summary ========================================================================================== Install 1 Package Total size: 440 M Installed size: 440 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-manager-4.4.0-0.40400.20220816.x86_64 1/1 Verifying : wazuh-manager-4.4.0-0.40400.20220816.x86_64 1/1 Installed: wazuh-manager.x86_64 0:4.4.0-0.40400.20220816 Complete! [root@centos7 vagrant]# systemctl daemon-reload [root@centos7 vagrant]# systemctl enable wazuh-manager Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-manager.service to /usr/lib/systemd/system/wazuh-manager.service. [root@centos7 vagrant]# systemctl start wazuh-manager [root@centos7 vagrant]# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH [root@centos7 vagrant]# echo -e '[wazuh]\ngpgcheck=1\ngpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH\nenabled=1\nname=EL-$releasever - Wazuh\nbaseurl=https://packages.wazuh.com/4.x/yum/\nprotect=1' | tee /etc/yum.repos.d/wazuh.repo [wazuh] gpgcheck=1 gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH enabled=1 name=EL-$releasever - Wazuh baseurl=https://packages.wazuh.com/4.x/yum/ protect=1 [root@centos7 vagrant]# systemctl status wazuh-manager ● wazuh-manager.service - Wazuh manager Loaded: loaded (/usr/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2022-09-08 06:54:47 UTC; 51s ago Process: 3403 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/SUCCESS) CGroup: /system.slice/wazuh-manager.service ├─3462 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-... ├─3503 /var/ossec/bin/wazuh-authd ├─3521 /var/ossec/bin/wazuh-db ├─3534 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-... ├─3537 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-... ├─3551 /var/ossec/bin/wazuh-execd ├─3567 /var/ossec/bin/wazuh-analysisd ├─3610 /var/ossec/bin/wazuh-syscheckd ├─3629 /var/ossec/bin/wazuh-remoted ├─3659 /var/ossec/bin/wazuh-logcollector ├─3690 /var/ossec/bin/wazuh-monitord └─3700 /var/ossec/bin/wazuh-modulesd Sep 08 06:54:40 centos7.localdomain env[3403]: Started wazuh-db... Sep 08 06:54:41 centos7.localdomain env[3403]: Started wazuh-execd... Sep 08 06:54:42 centos7.localdomain env[3403]: Started wazuh-analysisd... Sep 08 06:54:43 centos7.localdomain env[3403]: Started wazuh-syscheckd... Sep 08 06:54:43 centos7.localdomain env[3403]: Started wazuh-remoted... Sep 08 06:54:44 centos7.localdomain env[3403]: Started wazuh-logcollector... Sep 08 06:54:44 centos7.localdomain env[3403]: Started wazuh-monitord... Sep 08 06:54:45 centos7.localdomain env[3403]: Started wazuh-modulesd... Sep 08 06:54:47 centos7.localdomain env[3403]: Completed. Sep 08 06:54:47 centos7.localdomain systemd[1]: Started Wazuh manager. [root@centos7 vagrant]# yum -y install filebeat Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile epel/x86_64/metalink | 28 kB 00:00:00 * base: mirrors.pt * epel: fedora.cu.be * extras: centos.uvigo.es * updates: centos.uvigo.es base | 3.6 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 wazuh | 3.4 kB 00:00:00 Not using downloaded wazuh/repomd.xml because it is older than what we have: Current : Wed Aug 31 23:26:35 2022 Downloaded: Wed Aug 31 13:56:41 2022 Resolving Dependencies --> Running transaction check ---> Package filebeat.x86_64 0:7.10.2-1 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================== Package Arch Version Repository Size ========================================================================================== Installing: filebeat x86_64 7.10.2-1 wazuh 21 M Transaction Summary ========================================================================================== Install 1 Package Total download size: 21 M Installed size: 70 M Downloading packages: filebeat-oss-7.10.2-x86_64.rpm | 21 MB 00:00:02 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : filebeat-7.10.2-1.x86_64 1/1 Verifying : filebeat-7.10.2-1.x86_64 1/1 Installed: filebeat.x86_64 0:7.10.2-1 Complete! [root@centos7 vagrant]# curl -so /etc/filebeat/filebeat.yml https://packages.wazuh.com/4.3/tpl/wazuh/filebeat/filebeat.yml [root@centos7 vagrant]# filebeat keystore create Created filebeat keystore [root@centos7 vagrant]# [root@centos7 vagrant]# echo admin | filebeat keystore add username --stdin --force Successfully updated the keystore [root@centos7 vagrant]# echo admin | filebeat keystore add password --stdin --force Successfully updated the keystore [root@centos7 vagrant]# curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/4.3/extensions/elasticsearch/7.x/wazuh-template.json [root@centos7 vagrant]# chmod go+r /etc/filebeat/wazuh-template.json [root@centos7 vagrant]# curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.2.tar.gz | tar -xvz -C /usr/share/filebeat/module wazuh/alerts/ wazuh/alerts/config/ wazuh/alerts/config/alerts.yml wazuh/alerts/manifest.yml wazuh/alerts/ingest/ wazuh/alerts/ingest/pipeline.json wazuh/archives/ wazuh/archives/config/ wazuh/archives/config/archives.yml wazuh/archives/manifest.yml wazuh/archives/ingest/ wazuh/archives/ingest/pipeline.json wazuh/module.yml [root@centos7 vagrant]# NODE_NAME=wazuh-1 [root@centos7 vagrant]# mkdir /etc/filebeat/certs [root@centos7 vagrant]# tar -xf ./wazuh-certificates.tar -C /etc/filebeat/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem [root@centos7 vagrant]# mv -n /etc/filebeat/certs/$NODE_NAME.pem /etc/filebeat/certs/filebeat.pem [root@centos7 vagrant]# mv -n /etc/filebeat/certs/$NODE_NAME-key.pem /etc/filebeat/certs/filebeat-key.pem [root@centos7 vagrant]# chmod 500 /etc/filebeat/certs [root@centos7 vagrant]# chmod 400 /etc/filebeat/certs/* [root@centos7 vagrant]# chown -R root:root /etc/filebeat/certs [root@centos7 vagrant]# systemctl daemon-reload [root@centos7 vagrant]# systemctl enable filebeat Created symlink from /etc/systemd/system/multi-user.target.wants/filebeat.service to /usr/lib/systemd/system/filebeat.service. [root@centos7 vagrant]# systemctl start filebeat [root@centos7 vagrant]# filebeat test output elasticsearch: https://127.0.0.1:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 127.0.0.1 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 7.10.2 ````

wazuh-dashboard

```` [root@centos7 vagrant]# yum install -y wazuh-dashboard wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm wazuh-dashboard.x86_64 [root@centos7 vagrant]# yum install -y wazuh-dashboard wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm wazuh-dashboard.x86_64 [root@centos7 vagrant]# yum install -y wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm Loaded plugins: fastestmirror Examining wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm: wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 Marking wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package wazuh-dashboard.x86_64 0:4.4.0-0.40400.20220808 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================== Package Arch Version Repository Size ========================================================================================== Installing: wazuh-dashboard x86_64 4.4.0-0.40400.20220808 /wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 645 M Transaction Summary ========================================================================================== Install 1 Package Total size: 645 M Installed size: 645 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 1/1 Verifying : wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 1/1 Installed: wazuh-dashboard.x86_64 0:4.4.0-0.40400.20220808 Complete! [root@centos7 vagrant]# NODE_NAME=dashboard [root@centos7 vagrant]# mkdir /etc/wazuh-dashboard/certs [root@centos7 vagrant]# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-dashboard/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem [root@centos7 vagrant]# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME.pem /etc/wazuh-dashboard/certs/dashboard.pem mv: ‘/etc/wazuh-dashboard/certs/dashboard.pem’ and ‘/etc/wazuh-dashboard/certs/dashboard.pem’ are the same file [root@centos7 vagrant]# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem mv: ‘/etc/wazuh-dashboard/certs/dashboard-key.pem’ and ‘/etc/wazuh-dashboard/certs/dashboard-key.pem’ are the same file [root@centos7 vagrant]# chmod 500 /etc/wazuh-dashboard/certs [root@centos7 vagrant]# chmod 400 /etc/wazuh-dashboard/certs/* [root@centos7 vagrant]# chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs [root@centos7 vagrant]# systemctl daemon-reload [root@centos7 vagrant]# systemctl enable wazuh-dashboard Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-dashboard.service to /etc/systemd/system/wazuh-dashboard.service. [root@centos7 vagrant]# systemctl start wazuh-dashboard [root@centos7 vagrant]# systemctl status wazuh-dashboard ● wazuh-dashboard.service - wazuh-dashboard Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2022-09-08 07:04:35 UTC; 29s ago Main PID: 5267 (node) CGroup: /system.slice/wazuh-dashboard.service └─5267 /usr/share/wazuh-dashboard/bin/../node/bin/node --no-warnings --max-h... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Sep 08 07:04:39 centos7.localdomain opensearch-dashboards[5267]: {"type":"log","@timest... Hint: Some lines were ellipsized, use -l to show in full. ````

wazuh-cluster.log

```` [root@centos7 vagrant]# cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -e "Error" [2022-09-08T06:42:12,714][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-7869858717041396688, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-08T06:42:19,161][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. [2022-09-08T09:05:17,772][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5387032630944188174, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-08T09:05:23,459][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. [2022-09-08T09:05:26,172][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:05:26,228][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:05:26,236][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:05:26,242][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:05:43,984][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-8343828391135558508, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-08T09:05:48,486][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. [2022-09-08T09:05:50,918][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:05:50,937][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:05:50,940][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:05:50,942][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:08:23,727][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-14877440632485736212, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-08T09:08:28,177][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. [2022-09-08T09:08:30,966][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:08:30,981][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:08:30,983][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:08:30,986][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:11:35,665][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-10970786788831328113, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-08T09:11:44,172][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. ```` It has a random behavior, sometimes the message that the cluster is not initialized appears, and sometimes it does not. When rebooting the machine it seems that the message does not appear but restarting the indexer may not be enough

Ubuntu20

wazuh-indexer

```` root@ubuntu2004:/home/vagrant# apt-get install ./wazuh-indexer_4.4.0-0.40400.20220808_amd64.deb Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'wazuh-indexer' instead of './wazuh-indexer_4.4.0-0.40400.20220808_amd64.deb' The following NEW packages will be installed: wazuh-indexer 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/398 MB of archives. After this operation, 668 MB of additional disk space will be used. Get:1 /home/vagrant/wazuh-indexer_4.4.0-0.40400.20220808_amd64.deb wazuh-indexer amd64 4.4.0-0.40400.20220808 [398 MB] Selecting previously unselected package wazuh-indexer. (Reading database ... 111637 files and directories currently installed.) Preparing to unpack .../wazuh-indexer_4.4.0-0.40400.20220808_amd64.deb ... Creating wazuh-indexer group... OK Creating wazuh-indexer user... OK Unpacking wazuh-indexer (4.4.0-0.40400.20220808) ... Setting up wazuh-indexer (4.4.0-0.40400.20220808) ... Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore Processing triggers for systemd (245.4-4ubuntu3.17) ... Processing triggers for libc-bin (2.31-0ubuntu9.9) ... root@ubuntu2004:/home/vagrant# nano /etc/wazuh-indexer/opensearch.yml root@ubuntu2004:/home/vagrant# NODE_NAME=node-1 root@ubuntu2004:/home/vagrant# mkdir /etc/wazuh-indexer/certs root@ubuntu2004:/home/vagrant# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem root@ubuntu2004:/home/vagrant# mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem root@ubuntu2004:/home/vagrant# mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem root@ubuntu2004:/home/vagrant# chmod 500 /etc/wazuh-indexer/certs root@ubuntu2004:/home/vagrant# chmod 400 /etc/wazuh-indexer/certs/* root@ubuntu2004:/home/vagrant# chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs root@ubuntu2004:/home/vagrant# systemctl daemon-reload root@ubuntu2004:/home/vagrant# systemctl enable wazuh-indexer Synchronizing state of wazuh-indexer.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable wazuh-indexer Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service → /lib/systemd/system/wazuh-indexer.service. root@ubuntu2004:/home/vagrant# systemctl start wazuh-indexer root@ubuntu2004:/home/vagrant# curl -k -u admin:admin https://localhost:9200 { "name" : "node-1", "cluster_name" : "wazuh-cluster", "cluster_uuid" : "0t-loY0yRte-vu_MnenVhw", "version" : { "number" : "7.10.2", "build_type" : "rpm", "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32", "build_date" : "2022-06-30T21:31:04.823801692Z", "build_snapshot" : false, "lucene_version" : "9.2.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } root@ubuntu2004:/home/vagrant# curl -k -u admin:admin https://localhost:9200/_cat/nodes?v ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles cluster_manager name 192.168.121.146 23 96 4 0.08 0.14 0.09 dimr cluster_manager,data,ingest,remote_cluster_client * node-1 ````

wazuh-manager

```` root@ubuntu2004:/home/vagrant# apt install ./wazuh-manager_4.4.0-0.40400.20220808_amd64.deb Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'wazuh-manager' instead of './wazuh-manager_4.4.0-0.40400.20220808_amd64.deb' Suggested packages: expect The following NEW packages will be installed: wazuh-manager 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/120 MB of archives. After this operation, 460 MB of additional disk space will be used. Get:1 /home/vagrant/wazuh-manager_4.4.0-0.40400.20220808_amd64.deb wazuh-manager amd64 4.4.0-0.40400.20220808 [120 MB] Selecting previously unselected package wazuh-manager. (Reading database ... 112687 files and directories currently installed.) Preparing to unpack .../wazuh-manager_4.4.0-0.40400.20220808_amd64.deb ... Unpacking wazuh-manager (4.4.0-0.40400.20220808) ... Setting up wazuh-manager (4.4.0-0.40400.20220808) ... Processing triggers for systemd (245.4-4ubuntu3.17) ... root@ubuntu2004:/home/vagrant# systemctl daemon-reload root@ubuntu2004:/home/vagrant# systemctl enable wazuh-manager Synchronizing state of wazuh-manager.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable wazuh-manager Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-manager.service → /lib/systemd/system/wazuh-manager.service. root@ubuntu2004:/home/vagrant# systemctl start wazuh-manager root@ubuntu2004:/home/vagrant# systemctl status wazuh-manager ● wazuh-manager.service - Wazuh manager Loaded: loaded (/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-09-08 06:54:40 UTC; 1min 1s ago Process: 40693 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/S> Tasks: 115 (limit: 2273) Memory: 231.5M CGroup: /system.slice/wazuh-manager.service ├─40764 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─40803 /var/ossec/bin/wazuh-authd ├─40816 /var/ossec/bin/wazuh-db ├─40837 /var/ossec/bin/wazuh-execd ├─40848 /var/ossec/bin/wazuh-analysisd ├─40857 /var/ossec/bin/wazuh-syscheckd ├─40903 /var/ossec/bin/wazuh-remoted ├─40932 /var/ossec/bin/wazuh-logcollector ├─40948 /var/ossec/bin/wazuh-monitord ├─40957 /var/ossec/bin/wazuh-modulesd ├─41117 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py └─41122 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py Sep 08 06:54:37 ubuntu2004.localdomain env[40693]: Started wazuh-db... Sep 08 06:54:37 ubuntu2004.localdomain env[40693]: Started wazuh-execd... Sep 08 06:54:37 ubuntu2004.localdomain env[40693]: Started wazuh-analysisd... Sep 08 06:54:38 ubuntu2004.localdomain env[40693]: Started wazuh-syscheckd... Sep 08 06:54:38 ubuntu2004.localdomain env[40693]: Started wazuh-remoted... Sep 08 06:54:38 ubuntu2004.localdomain env[40693]: Started wazuh-logcollector... Sep 08 06:54:38 ubuntu2004.localdomain env[40693]: Started wazuh-monitord... Sep 08 06:54:38 ubuntu2004.localdomain env[40693]: Started wazuh-modulesd... Sep 08 06:54:40 ubuntu2004.localdomain env[40693]: Completed. Sep 08 06:54:40 ubuntu2004.localdomain systemd[1]: Started Wazuh manager. root@ubuntu2004:/home/vagrant# apt-get -y install filebeat Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: filebeat 0 upgraded, 1 newly installed, 0 to remove and 32 not upgraded. Need to get 22.1 MB of archives. After this operation, 73.6 MB of additional disk space will be used. Get:1 https://packages.wazuh.com/4.x/apt stable/main amd64 filebeat amd64 7.10.2 [22.1 MB] Fetched 22.1 MB in 4s (5,047 kB/s) Selecting previously unselected package filebeat. (Reading database ... 131403 files and directories currently installed.) Preparing to unpack .../filebeat_7.10.2_amd64.deb ... Unpacking filebeat (7.10.2) ... Setting up filebeat (7.10.2) ... Processing triggers for systemd (245.4-4ubuntu3.17) ... root@ubuntu2004:/home/vagrant# curl -so /etc/filebeat/filebeat.yml https://packages.wazuh.com/4.3/tpl/wazuh/filebeat/filebeat.yml root@ubuntu2004:/home/vagrant# filebeat keystore create Created filebeat keystore root@ubuntu2004:/home/vagrant# echo admin | filebeat keystore add username --stdin --force Successfully updated the keystore root@ubuntu2004:/home/vagrant# echo admin | filebeat keystore add password --stdin --force Successfully updated the keystore root@ubuntu2004:/home/vagrant# curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/4.3/extensions/elasticsearch/7.x/wazuh-template.json root@ubuntu2004:/home/vagrant# chmod go+r /etc/filebeat/wazuh-template.json root@ubuntu2004:/home/vagrant# curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.2.tar.gz | tar -xvz -C /usr/share/filebeat/module wazuh/alerts/ wazuh/alerts/config/ wazuh/alerts/config/alerts.yml wazuh/alerts/manifest.yml wazuh/alerts/ingest/ wazuh/alerts/ingest/pipeline.json wazuh/archives/ wazuh/archives/config/ wazuh/archives/config/archives.yml wazuh/archives/manifest.yml wazuh/archives/ingest/ wazuh/archives/ingest/pipeline.json wazuh/module.yml root@ubuntu2004:/home/vagrant# NODE_NAME=wazuh-1 root@ubuntu2004:/home/vagrant# mkdir /etc/filebeat/certs root@ubuntu2004:/home/vagrant# tar -xf ./wazuh-certificates.tar -C /etc/filebeat/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem root@ubuntu2004:/home/vagrant# mv -n /etc/filebeat/certs/$NODE_NAME.pem /etc/filebeat/certs/filebeat.pem root@ubuntu2004:/home/vagrant# mv -n /etc/filebeat/certs/$NODE_NAME-key.pem /etc/filebeat/certs/filebeat-key.pem root@ubuntu2004:/home/vagrant# chmod 500 /etc/filebeat/certs root@ubuntu2004:/home/vagrant# chmod 400 /etc/filebeat/certs/* root@ubuntu2004:/home/vagrant# chown -R root:root /etc/filebeat/certs root@ubuntu2004:/home/vagrant# systemctl daemon-reload root@ubuntu2004:/home/vagrant# systemctl enable filebeat Synchronizing state of filebeat.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable filebeat Created symlink /etc/systemd/system/multi-user.target.wants/filebeat.service → /lib/systemd/system/filebeat.service. root@ubuntu2004:/home/vagrant# systemctl start filebeat root@ubuntu2004:/home/vagrant# filebeat test output elasticsearch: https://127.0.0.1:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 127.0.0.1 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 7.10.2 ````

wazuh-dashboard

```` root@ubuntu2004:/home/vagrant# apt install ./wazuh-dashboard_4.4.0-0.40400.20220808_amd64.deb Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'wazuh-dashboard' instead of './wazuh-dashboard_4.4.0-0.40400.20220808_amd64.deb' The following NEW packages will be installed: wazuh-dashboard 0 upgraded, 1 newly installed, 0 to remove and 32 not upgraded. Need to get 0 B/142 MB of archives. After this operation, 708 MB of additional disk space will be used. Get:1 /home/vagrant/wazuh-dashboard_4.4.0-0.40400.20220808_amd64.deb wazuh-dashboard amd64 4.4.0-0.40400.20220808 [142 MB] Selecting previously unselected package wazuh-dashboard. (Reading database ... 131722 files and directories currently installed.) Preparing to unpack .../wazuh-dashboard_4.4.0-0.40400.20220808_amd64.deb ... Creating wazuh-dashboard group... OK Creating wazuh-dashboard user... OK Unpacking wazuh-dashboard (4.4.0-0.40400.20220808) ... Setting up wazuh-dashboard (4.4.0-0.40400.20220808) ... root@ubuntu2004:/home/vagrant# NODE_NAME=dashboard root@ubuntu2004:/home/vagrant# mkdir /etc/wazuh-dashboard/certs root@ubuntu2004:/home/vagrant# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-dashboard/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem root@ubuntu2004:/home/vagrant# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME.pem /etc/wazuh-dashboard/certs/dashboard.pem root@ubuntu2004:/home/vagrant# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem root@ubuntu2004:/home/vagrant# chmod 500 /etc/wazuh-dashboard/certs root@ubuntu2004:/home/vagrant# chmod 400 /etc/wazuh-dashboard/certs/* root@ubuntu2004:/home/vagrant# chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs root@ubuntu2004:/home/vagrant# systemctl daemon-reload root@ubuntu2004:/home/vagrant# systemctl enable wazuh-dashboard Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-dashboard.service → /etc/systemd/system/wazuh-dashboard.service. root@ubuntu2004:/home/vagrant# systemctl start wazuh-dashboard root@ubuntu2004:/home/vagrant# systemctl status wazuh-dashboard ● wazuh-dashboard.service - wazuh-dashboard Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-09-08 07:04:37 UTC; 18s ago Main PID: 43037 (node) Tasks: 11 (limit: 2273) Memory: 206.4M CGroup: /system.slice/wazuh-dashboard.service └─43037 /usr/share/wazuh-dashboard/bin/../node/bin/node --no-warnings --max-http-header-> Sep 08 07:04:40 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022> Sep 08 07:04:40 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022> Sep 08 07:04:40 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022> Sep 08 07:04:41 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022> Sep 08 07:04:41 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022> Sep 08 07:04:41 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022> Sep 08 07:04:41 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022> Sep 08 07:04:41 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022> Sep 08 07:04:41 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022> Sep 08 07:04:41 ubuntu2004.localdomain opensearch-dashboards[43037]: {"type":"log","@timestamp":"2022 ````

wazuh-cluster.log

```` root@ubuntu2004:/home/vagrant# cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -e "Error" [2022-09-08T06:42:15,278][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-14658143183749157729, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-08T06:42:20,727][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. [2022-09-08T07:05:33,952][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a203139322e3136382e3132312e3134363a393230300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a557067726164652d496e7365637572652d52657175657374733a20310d0a557365722d4167656e743a204d6f7a696c6c612f352e3020285831313b204c696e7578207838365f363429204170706c655765624b69742f3533372e333620284b48544d4c2c206c696b65204765636b6f29204368726f6d652f3130342e302e353131322e313032205361666172692f3533372e33360d0a4163636570743a20746578742f68746d6c2c6170706c69636174696f6e2f7868746d6c2b786d6c2c6170706c69636174696f6e2f786d6c3b713d302e392c696d6167652f617669662c696d6167652f776562702c696d6167652f61706e672c2a2f2a3b713d302e380d0a5365632d4750433a20310d0a4163636570742d4c616e67756167653a20656e2d55532c656e0d0a4163636570742d456e636f64696e673a20677a69702c206465666c6174650d0a0d0a [2022-09-08T07:05:34,969][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a203139322e3136382e3132312e3134363a393230300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a43616368652d436f6e74726f6c3a206d61782d6167653d300d0a557067726164652d496e7365637572652d52657175657374733a20310d0a557365722d4167656e743a204d6f7a696c6c612f352e3020285831313b204c696e7578207838365f363429204170706c655765624b69742f3533372e333620284b48544d4c2c206c696b65204765636b6f29204368726f6d652f3130342e302e353131322e313032205361666172692f3533372e33360d0a4163636570743a20746578742f68746d6c2c6170706c69636174696f6e2f7868746d6c2b786d6c2c6170706c69636174696f6e2f786d6c3b713d302e392c696d6167652f617669662c696d6167652f776562702c696d6167652f61706e672c2a2f2a3b713d302e380d0a5365632d4750433a20310d0a4163636570742d4c616e67756167653a20656e2d55532c656e0d0a4163636570742d456e636f64696e673a20677a69702c206465666c6174650d0a0d0a [2022-09-08T09:03:52,363][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-12044025639855870866, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-08T09:04:00,292][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. [2022-09-08T09:04:02,613][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:04:02,653][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:04:02,657][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:04:02,658][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) [2022-09-08T09:05:10,930][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-11375286781245784906, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-09-08T09:05:18,906][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. ````

Distributed, one indexer

wazuh-indexer

```` [root@ip-172-31-8-198 ec2-user]# yum install -y wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm .ssh/ wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm [root@ip-172-31-8-198 ec2-user]# yum install -y wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Examining wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm: wazuh-indexer-4.4.0-0.40400.20220808.x86_64 Marking wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm to be installed Resolving Dependencies There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help). --> Running transaction check ---> Package wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 will be installed --> Finished Dependency Resolution amzn2-core/2/x86_64 | 3.7 kB 00:00:00 Dependencies Resolved ================================================================================= Package Arch Version Repository Size ================================================================================= Installing: wazuh-indexer x86_64 4.4.0-0.40400.20220808 /wazuh-indexer-4.4.0-0.40400.20220808.x86_64 642 M Transaction Summary ================================================================================= Install 1 Package Total size: 642 M Installed size: 642 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore Verifying : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Installed: wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 Complete! [root@ip-172-31-8-198 ec2-user]# nano /etc/wazuh-indexer/opensearch.yml [root@ip-172-31-8-198 ec2-user]# nano config.yml [root@ip-172-31-8-198 ec2-user]# NODE_NAME=node-1 [root@ip-172-31-8-198 ec2-user]# mkdir /etc/wazuh-indexer/certs [root@ip-172-31-8-198 ec2-user]# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem [root@ip-172-31-8-198 ec2-user]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem [root@ip-172-31-8-198 ec2-user]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem [root@ip-172-31-8-198 ec2-user]# chmod 500 /etc/wazuh-indexer/certs [root@ip-172-31-8-198 ec2-user]# chmod 400 /etc/wazuh-indexer/certs/* [root@ip-172-31-8-198 ec2-user]# chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs [root@ip-172-31-8-198 ec2-user]# systemctl daemon-reload [root@ip-172-31-8-198 ec2-user]# systemctl enable wazuh-indexer Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service to /usr/lib/systemd/system/wazuh-indexer.service. [root@ip-172-31-8-198 ec2-user]# systemctl start wazuh-indexer [root@ip-172-31-8-198 ec2-user]# curl -k -u admin:admin https://localhost:9200 { "name" : "node-1", "cluster_name" : "wazuh-cluster", "cluster_uuid" : "hlRKMbalSISLAiJwZFSWcg", "version" : { "number" : "7.10.2", "build_type" : "rpm", "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32", "build_date" : "2022-06-30T21:31:04.823801692Z", "build_snapshot" : false, "lucene_version" : "9.2.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } [root@ip-172-31-8-198 ec2-user]# curl -k -u admin:admin https://3.144.83.0:9200 { "name" : "node-1", "cluster_name" : "wazuh-cluster", "cluster_uuid" : "hlRKMbalSISLAiJwZFSWcg", "version" : { "number" : "7.10.2", "build_type" : "rpm", "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32", "build_date" : "2022-06-30T21:31:04.823801692Z", "build_snapshot" : false, "lucene_version" : "9.2.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } ````

wazuh-server

```` [root@ip-172-31-11-12 ec2-user]# yum install -y wazuh-manager-4.4.0-0.40400.20220816.x86_64.rpm Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Examining wazuh-manager-4.4.0-0.40400.20220816.x86_64.rpm: wazuh-manager-4.4.0-0.40400.20220816.x86_64 Marking wazuh-manager-4.4.0-0.40400.20220816.x86_64.rpm to be installed Resolving Dependencies There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help). --> Running transaction check ---> Package wazuh-manager.x86_64 0:4.4.0-0.40400.20220816 will be installed --> Finished Dependency Resolution amzn2-core/2/x86_64 | 3.7 kB 00:00:00 wazuh | 3.4 kB 00:00:00 Not using downloaded wazuh/repomd.xml because it is older than what we have: Current : Wed Sep 7 23:30:55 2022 Downloaded: Wed Aug 31 13:56:41 2022 Dependencies Resolved ================================================================================= Package Arch Version Repository Size ================================================================================= Installing: wazuh-manager x86_64 4.4.0-0.40400.20220816 /wazuh-manager-4.4.0-0.40400.20220816.x86_64 440 M Transaction Summary ================================================================================= Install 1 Package Total size: 440 M Installed size: 440 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-manager-4.4.0-0.40400.20220816.x86_64 1/1 Verifying : wazuh-manager-4.4.0-0.40400.20220816.x86_64 1/1 Installed: wazuh-manager.x86_64 0:4.4.0-0.40400.20220816 Complete! [root@ip-172-31-11-12 ec2-user]# systemctl daemon-reload [root@ip-172-31-11-12 ec2-user]# systemctl enable wazuh-manager Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-manager.service to /usr/lib/systemd/system/wazuh-manager.service. [root@ip-172-31-11-12 ec2-user]# systemctl start wazuh-manager [root@ip-172-31-11-12 ec2-user]# systemctl status wazuh-manager ● wazuh-manager.service - Wazuh manager Loaded: loaded (/usr/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2022-09-08 07:54:49 UTC; 45s ago Process: 5239 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/SUCCESS) CGroup: /system.slice/wazuh-manager.service ├─5303 /var/ossec/framework/python/bin/python3 /var/ossec/api/scrip... ├─5345 /var/ossec/bin/wazuh-authd ├─5362 /var/ossec/bin/wazuh-db ├─5375 /var/ossec/framework/python/bin/python3 /var/ossec/api/scrip... ├─5378 /var/ossec/framework/python/bin/python3 /var/ossec/api/scrip... ├─5393 /var/ossec/bin/wazuh-execd ├─5407 /var/ossec/bin/wazuh-analysisd ├─5452 /var/ossec/bin/wazuh-syscheckd ├─5468 /var/ossec/bin/wazuh-remoted ├─5501 /var/ossec/bin/wazuh-logcollector ├─5525 /var/ossec/bin/wazuh-monitord └─5546 /var/ossec/bin/wazuh-modulesd Sep 08 07:54:40 ip-172-31-11-12.us-east-2.compute.internal env[5239]: Started ... Sep 08 07:54:41 ip-172-31-11-12.us-east-2.compute.internal env[5239]: Started ... Sep 08 07:54:42 ip-172-31-11-12.us-east-2.compute.internal env[5239]: Started ... Sep 08 07:54:43 ip-172-31-11-12.us-east-2.compute.internal env[5239]: Started ... Sep 08 07:54:44 ip-172-31-11-12.us-east-2.compute.internal env[5239]: Started ... Sep 08 07:54:45 ip-172-31-11-12.us-east-2.compute.internal env[5239]: Started ... Sep 08 07:54:46 ip-172-31-11-12.us-east-2.compute.internal env[5239]: Started ... Sep 08 07:54:47 ip-172-31-11-12.us-east-2.compute.internal env[5239]: Started ... Sep 08 07:54:49 ip-172-31-11-12.us-east-2.compute.internal env[5239]: Completed. Sep 08 07:54:49 ip-172-31-11-12.us-east-2.compute.internal systemd[1]: Started... Hint: Some lines were ellipsized, use -l to show in full. [root@ip-172-31-11-12 ec2-user]# yum -y install filebeat Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Resolving Dependencies There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help). --> Running transaction check ---> Package filebeat.x86_64 0:7.10.2-1 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================= Package Arch Version Repository Size ================================================================================= Installing: filebeat x86_64 7.10.2-1 wazuh 21 M Transaction Summary ================================================================================= Install 1 Package Total download size: 21 M Installed size: 70 M Downloading packages: filebeat-oss-7.10.2-x86_64.rpm | 21 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : filebeat-7.10.2-1.x86_64 1/1 Verifying : filebeat-7.10.2-1.x86_64 1/1 Installed: filebeat.x86_64 0:7.10.2-1 Complete! [root@ip-172-31-11-12 ec2-user]# curl -so /etc/filebeat/filebeat.yml https://packages.wazuh.com/4.3/tpl/wazuh/filebeat/filebeat.yml [root@ip-172-31-11-12 ec2-user]# nano /etc/filebeat/filebeat.yml [root@ip-172-31-11-12 ec2-user]# filebeat keystore create Created filebeat keystore [root@ip-172-31-11-12 ec2-user]# echo admin | filebeat keystore add username --stdin --force Successfully updated the keystore [root@ip-172-31-11-12 ec2-user]# echo admin | filebeat keystore add password --stdin --force Successfully updated the keystore [root@ip-172-31-11-12 ec2-user]# curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/4.3/extensions/elasticsearch/7.x/wazuh-template.json [root@ip-172-31-11-12 ec2-user]# chmod go+r /etc/filebeat/wazuh-template.json [root@ip-172-31-11-12 ec2-user]# curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.2.tar.gz | tar -xvz -C /usr/share/filebeat/module wazuh/alerts/ wazuh/alerts/config/ wazuh/alerts/config/alerts.yml wazuh/alerts/manifest.yml wazuh/alerts/ingest/ wazuh/alerts/ingest/pipeline.json wazuh/archives/ wazuh/archives/config/ wazuh/archives/config/archives.yml wazuh/archives/manifest.yml wazuh/archives/ingest/ wazuh/archives/ingest/pipeline.json wazuh/module.yml [root@ip-172-31-11-12 ec2-user]# NODE_NAME=wazuh-1 [root@ip-172-31-11-12 ec2-user]# est.yml bash: est.yml: command not found [root@ip-172-31-11-12 ec2-user]# mkdir /etc/filebeat/certs [root@ip-172-31-11-12 ec2-user]# tar -xf ./wazuh-certificates.tar -C /etc/filebeat/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem [root@ip-172-31-11-12 ec2-user]# mv -n /etc/filebeat/certs/$NODE_NAME.pem /etc/filebeat/certs/filebeat.pem [root@ip-172-31-11-12 ec2-user]# mv -n /etc/filebeat/certs/$NODE_NAME-key.pem /etc/filebeat/certs/filebeat-key.pem [root@ip-172-31-11-12 ec2-user]# chmod 500 /etc/filebeat/certs [root@ip-172-31-11-12 ec2-user]# chmod 400 /etc/filebeat/certs/* [root@ip-172-31-11-12 ec2-user]# chown -R root:root /etc/filebeat/certs [root@ip-172-31-11-12 ec2-user]# systemctl daemon-reload [root@ip-172-31-11-12 ec2-user]# systemctl enable filebeat Created symlink from /etc/systemd/system/multi-user.target.wants/filebeat.service to /usr/lib/systemd/system/filebeat.service. [root@ip-172-31-11-12 ec2-user]# systemctl start filebeat [root@ip-172-31-11-12 ec2-user]# filebeat test output elasticsearch: https://172.31.8.198:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 172.31.8.198 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 7.10.2 ````

wazuh-dashboard

```` [root@ip-172-31-4-239 ec2-user]# yum install -y wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Examining wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm: wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 Marking wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package wazuh-dashboard.x86_64 0:4.4.0-0.40400.20220808 will be installed --> Finished Dependency Resolution amzn2-core/2/x86_64 | 3.7 kB 00:00:00 Dependencies Resolved ================================================================================= Package Arch Version Repository Size ================================================================================= Installing: wazuh-dashboard x86_64 4.4.0-0.40400.20220808 /wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 645 M Transaction Summary ================================================================================= Install 1 Package Total size: 645 M Installed size: 645 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 1/1 Verifying : wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 1/1 Installed: wazuh-dashboard.x86_64 0:4.4.0-0.40400.20220808 Complete! [root@ip-172-31-4-239 ec2-user]# nano /etc/wazuh-dashboard/opensearch_dashboards.yml [root@ip-172-31-4-239 ec2-user]# NODE_NAME=dashboard [root@ip-172-31-4-239 ec2-user]# mkdir /etc/wazuh-dashboard/certs [root@ip-172-31-4-239 ec2-user]# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-dashboard/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem [root@ip-172-31-4-239 ec2-user]# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME.pem /etc/wazuh-dashboard/certs/dashboard.pem mv: ‘/etc/wazuh-dashboard/certs/dashboard.pem’ and ‘/etc/wazuh-dashboard/certs/dashboard.pem’ are the same file [root@ip-172-31-4-239 ec2-user]# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem mv: ‘/etc/wazuh-dashboard/certs/dashboard-key.pem’ and ‘/etc/wazuh-dashboard/certs/dashboard-key.pem’ are the same file [root@ip-172-31-4-239 ec2-user]# chmod 500 /etc/wazuh-dashboard/certs [root@ip-172-31-4-239 ec2-user]# chmod 400 /etc/wazuh-dashboard/certs/* [root@ip-172-31-4-239 ec2-user]# chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs [root@ip-172-31-4-239 ec2-user]# systemctl daemon-reload [root@ip-172-31-4-239 ec2-user]# systemctl enable wazuh-dashboard Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-dashboard.service to /etc/systemd/system/wazuh-dashboard.service. [root@ip-172-31-4-239 ec2-user]# systemctl start wazuh-dashboard [root@ip-172-31-4-239 ec2-user]# nano /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml [root@ip-172-31-4-239 ec2-user]# systemctl status wazuh-dashboard ● wazuh-dashboard.service - wazuh-dashboard Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2022-09-08 08:08:07 UTC; 19min ago Main PID: 4154 (node) CGroup: /system.slice/wazuh-dashboard.service └─4154 /usr/share/wazuh-dashboard/bin/../node/bin/node --no-warning... Sep 08 08:15:00 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Sep 08 08:15:00 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Sep 08 08:20:00 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Sep 08 08:20:00 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Sep 08 08:20:00 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Sep 08 08:20:00 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Sep 08 08:25:01 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Sep 08 08:25:01 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Sep 08 08:25:01 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Sep 08 08:25:01 ip-172-31-4-239.us-east-2.compute.internal opensearch-dashboards[4154]: ... Hint: Some lines were ellipsized, use -l to show in full. ````

Distributed, three indexers

wazuh-indexer-1

```` [root@ip-172-31-8-198 ec2-user]# yum install -y wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Examining wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm: wazuh-indexer-4.4.0-0.40400.20220808.x86_64 Marking wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm to be installed Resolving Dependencies There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help). --> Running transaction check ---> Package wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 will be installed --> Finished Dependency Resolution amzn2-core/2/x86_64 | 3.7 kB 00:00:00 Dependencies Resolved ===================================================================================================================== Package Arch Version Repository Size ===================================================================================================================== Installing: wazuh-indexer x86_64 4.4.0-0.40400.20220808 /wazuh-indexer-4.4.0-0.40400.20220808.x86_64 642 M Transaction Summary ===================================================================================================================== Install 1 Package Total size: 642 M Installed size: 642 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore Verifying : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Installed: wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 Complete! [root@ip-172-31-8-198 ec2-user]# nano /etc/wazuh-indexer/opensearch.yml [root@ip-172-31-8-198 ec2-user]# NODE_NAME=node-1 [root@ip-172-31-8-198 ec2-user]# mkdir /etc/wazuh-indexer/certs [root@ip-172-31-8-198 ec2-user]# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem [root@ip-172-31-8-198 ec2-user]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem [root@ip-172-31-8-198 ec2-user]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem [root@ip-172-31-8-198 ec2-user]# chmod 500 /etc/wazuh-indexer/certs [root@ip-172-31-8-198 ec2-user]# chmod 400 /etc/wazuh-indexer/certs/* [root@ip-172-31-8-198 ec2-user]# chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs [root@ip-172-31-8-198 ec2-user]# systemctl daemon-reload [root@ip-172-31-8-198 ec2-user]# systemctl enable wazuh-indexer Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service to /usr/lib/systemd/system/wazuh-indexer.service. [root@ip-172-31-8-198 ec2-user]# systemctl start wazuh-indexer ^C [root@ip-172-31-8-198 ec2-user]# nano /etc/wazuh-indexer/opensearch.yml [root@ip-172-31-8-198 ec2-user]# systemctl start wazuh-indexer [root@ip-172-31-8-198 ec2-user]# systemctl restart wazuh-indexer [root@ip-172-31-8-198 ec2-user]# curl -k -u admin:admin https://localhost:9200 OpenSearch Security not initialized.[root@ip-172-31-8-198 ec2-user]# systemctl status wazuh-indexer ● wazuh-indexer.service - Wazuh-indexer Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2022-09-08 09:39:55 UTC; 1min 22s ago Docs: https://documentation.wazuh.com Main PID: 6166 (java) CGroup: /system.slice/wazuh-indexer.service └─6166 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dop... Sep 08 09:39:13 ip-172-31-8-198.us-east-2.compute.internal systemd[1]: Starting Wazuh-indexer... Sep 08 09:39:15 ip-172-31-8-198.us-east-2.compute.internal systemd-entrypoint[6166]: WARNING: A terminally deprec...d Sep 08 09:39:15 ip-172-31-8-198.us-east-2.compute.internal systemd-entrypoint[6166]: WARNING: System::setSecurity...) Sep 08 09:39:15 ip-172-31-8-198.us-east-2.compute.internal systemd-entrypoint[6166]: WARNING: Please consider rep...h Sep 08 09:39:15 ip-172-31-8-198.us-east-2.compute.internal systemd-entrypoint[6166]: WARNING: System::setSecurity...e Sep 08 09:39:16 ip-172-31-8-198.us-east-2.compute.internal systemd-entrypoint[6166]: WARNING: A terminally deprec...d Sep 08 09:39:16 ip-172-31-8-198.us-east-2.compute.internal systemd-entrypoint[6166]: WARNING: System::setSecurity...) Sep 08 09:39:16 ip-172-31-8-198.us-east-2.compute.internal systemd-entrypoint[6166]: WARNING: Please consider rep...y Sep 08 09:39:16 ip-172-31-8-198.us-east-2.compute.internal systemd-entrypoint[6166]: WARNING: System::setSecurity...e Sep 08 09:39:55 ip-172-31-8-198.us-east-2.compute.internal systemd[1]: Started Wazuh-indexer. Hint: Some lines were ellipsized, use -l to show in full. ```` After installing the other nodes ```` [root@ip-172-31-8-198 ec2-user]# curl -k -u admin:admin https://18.117.90.224:9200 { "name" : "node-2", "cluster_name" : "wazuh-cluster", "cluster_uuid" : "HdyZqB5-QKSlrje1Qa98Bw", "version" : { "number" : "7.10.2", "build_type" : "rpm", "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32", "build_date" : "2022-06-30T21:31:04.823801692Z", "build_snapshot" : false, "lucene_version" : "9.2.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } ````

wazuh-indexer-2

```` [root@migueltest ec2-user]# yum install -y wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Examining wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm: wazuh-indexer-4.4.0-0.40400.20220808.x86_64 Marking wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 will be installed --> Finished Dependency Resolution amzn2-core/2/x86_64 | 3.7 kB 00:00:00 Dependencies Resolved ===================================================================================================================== Package Arch Version Repository Size ===================================================================================================================== Installing: wazuh-indexer x86_64 4.4.0-0.40400.20220808 /wazuh-indexer-4.4.0-0.40400.20220808.x86_64 642 M Transaction Summary ===================================================================================================================== Install 1 Package Total size: 642 M Installed size: 642 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore Verifying : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Installed: wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 Complete! [root@migueltest ec2-user]# nano /etc/wazuh-indexer/opensearch.yml [root@migueltest ec2-user]# NODE_NAME=node-2 [root@migueltest ec2-user]# mkdir /etc/wazuh-indexer/certs [root@migueltest ec2-user]# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem [root@migueltest ec2-user]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem [root@migueltest ec2-user]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem [root@migueltest ec2-user]# chmod 500 /etc/wazuh-indexer/certs [root@migueltest ec2-user]# chmod 400 /etc/wazuh-indexer/certs/* [root@migueltest ec2-user]# chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs [root@migueltest ec2-user]# systemctl daemon-reload [root@migueltest ec2-user]# systemctl enable wazuh-indexer Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service to /usr/lib/systemd/system/wazuh-indexer.service. [root@migueltest ec2-user]# systemctl start wazuh-indexer [root@migueltest ec2-user]# curl -k -u admin:admin https://localhost:9200 { "name" : "node-2", "cluster_name" : "wazuh-cluster", "cluster_uuid" : "HdyZqB5-QKSlrje1Qa98Bw", "version" : { "number" : "7.10.2", "build_type" : "rpm", "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32", "build_date" : "2022-06-30T21:31:04.823801692Z", "build_snapshot" : false, "lucene_version" : "9.2.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } [root@migueltest ec2-user]# curl -k -u admin:admin https://3.144.83.0:9200/_cat/nodes?v ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles cluster_manager name 172.31.40.61 21 69 9 0.21 0.17 0.08 dimr cluster_manager,data,ingest,remote_cluster_client - node-3 172.31.8.198 24 69 1 0.02 0.03 0.03 dimr cluster_manager,data,ingest,remote_cluster_client * node-1 172.31.40.101 21 69 8 0.11 0.12 0.08 dimr cluster_manager,data,ingest,remote_cluster_client - node-2 ````

wazuh-indexer-3

```` [root@ip-172-31-40-61 ec2-user]# yum install -y wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Examining wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm: wazuh-indexer-4.4.0-0.40400.20220808.x86_64 Marking wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 will be installed --> Finished Dependency Resolution amzn2-core/2/x86_64 | 3.7 kB 00:00:00 Dependencies Resolved ===================================================================================================================== Package Arch Version Repository Size ===================================================================================================================== Installing: wazuh-indexer x86_64 4.4.0-0.40400.20220808 /wazuh-indexer-4.4.0-0.40400.20220808.x86_64 642 M Transaction Summary ===================================================================================================================== Install 1 Package Total size: 642 M Installed size: 642 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore Verifying : wazuh-indexer-4.4.0-0.40400.20220808.x86_64 1/1 Installed: wazuh-indexer.x86_64 0:4.4.0-0.40400.20220808 Complete! [root@ip-172-31-40-61 ec2-user]# nano /etc/wazuh-indexer/opensearch.yml [root@ip-172-31-40-61 ec2-user]# nano /etc/wazuh-indexer/opensearch.yml [root@ip-172-31-40-61 ec2-user]# nano /etc/wazuh-indexer/opensearch.yml [root@ip-172-31-40-61 ec2-user]# NODE_NAME=node-3 [root@ip-172-31-40-61 ec2-user]# mkdir /etc/wazuh-indexer/certs [root@ip-172-31-40-61 ec2-user]# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem [root@ip-172-31-40-61 ec2-user]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem [root@ip-172-31-40-61 ec2-user]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem [root@ip-172-31-40-61 ec2-user]# chmod 500 /etc/wazuh-indexer/certs [root@ip-172-31-40-61 ec2-user]# chmod 400 /etc/wazuh-indexer/certs/* [root@ip-172-31-40-61 ec2-user]# chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs [root@ip-172-31-40-61 ec2-user]# systemctl daemon-reload [root@ip-172-31-40-61 ec2-user]# systemctl enable wazuh-indexer Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service to /usr/lib/systemd/system/wazuh-indexer.service. [root@ip-172-31-40-61 ec2-user]# systemctl start wazuh-indexer [root@ip-172-31-40-61 ec2-user]# curl -k -u admin:admin https://3.144.83.0:9200/_cat/nodes?v ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles cluster_manager name 172.31.40.61 21 69 0 0.18 0.16 0.08 dimr cluster_manager,data,ingest,remote_cluster_client - node-3 172.31.8.198 25 69 1 0.02 0.03 0.03 dimr cluster_manager,data,ingest,remote_cluster_client * node-1 172.31.40.101 21 69 0 0.09 0.12 0.08 dimr cluster_manager,data,ingest,remote_cluster_client - node-2 ````

wazuh-server

```` [root@ip-172-31-11-12 ec2-user]# yum install -y wazuh-manager-4.4.0-0.40400.20220816.x86_64.rpm Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Examining wazuh-manager-4.4.0-0.40400.20220816.x86_64.rpm: wazuh-manager-4.4.0-0.40400.20220816.x86_64 Marking wazuh-manager-4.4.0-0.40400.20220816.x86_64.rpm to be installed Resolving Dependencies There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help). --> Running transaction check ---> Package wazuh-manager.x86_64 0:4.4.0-0.40400.20220816 will be installed --> Finished Dependency Resolution amzn2-core/2/x86_64 | 3.7 kB 00:00:00 Dependencies Resolved ================================================================================= Package Arch Version Repository Size ================================================================================= Installing: wazuh-manager x86_64 4.4.0-0.40400.20220816 /wazuh-manager-4.4.0-0.40400.20220816.x86_64 440 M Transaction Summary ================================================================================= Install 1 Package Total size: 440 M Installed size: 440 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-manager-4.4.0-0.40400.20220816.x86_64 1/1 Verifying : wazuh-manager-4.4.0-0.40400.20220816.x86_64 1/1 Installed: wazuh-manager.x86_64 0:4.4.0-0.40400.20220816 Complete! [root@ip-172-31-11-12 ec2-user]# systemctl daemon-reload [root@ip-172-31-11-12 ec2-user]# systemctl enable wazuh-manager [root@ip-172-31-11-12 ec2-user]# systemctl start wazuh-manager [root@ip-172-31-11-12 ec2-user]# systemctl status wazuh-manager ● wazuh-manager.service - Wazuh manager Loaded: loaded (/usr/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2022-09-08 09:54:54 UTC; 5s ago Process: 8472 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/SUCCESS) CGroup: /system.slice/wazuh-manager.service ├─8537 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─8577 /var/ossec/bin/wazuh-authd ├─8594 /var/ossec/bin/wazuh-db ├─8615 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─8621 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─8627 /var/ossec/bin/wazuh-execd ├─8642 /var/ossec/bin/wazuh-analysisd ├─8686 /var/ossec/bin/wazuh-syscheckd ├─8702 /var/ossec/bin/wazuh-remoted ├─8735 /var/ossec/bin/wazuh-logcollector ├─8758 /var/ossec/bin/wazuh-monitord ├─8801 /var/ossec/bin/wazuh-modulesd ├─9056 sh -c yum check-updates --security | grep "No packages" ├─9058 /usr/bin/python /usr/bin/yum check-updates --security ├─9059 grep No packages ├─9210 sh -c /bin/ps -p 7514 > /dev/null 2>&1 └─9211 /bin/ps -p 7514 Sep 08 09:54:44 ip-172-31-11-12.us-east-2.compute.internal env[8472]: Started wazuh-db... Sep 08 09:54:45 ip-172-31-11-12.us-east-2.compute.internal env[8472]: Started wazuh-execd... Sep 08 09:54:46 ip-172-31-11-12.us-east-2.compute.internal env[8472]: Started wazuh-analysisd... Sep 08 09:54:47 ip-172-31-11-12.us-east-2.compute.internal env[8472]: Started wazuh-syscheckd... Sep 08 09:54:48 ip-172-31-11-12.us-east-2.compute.internal env[8472]: Started wazuh-remoted... Sep 08 09:54:49 ip-172-31-11-12.us-east-2.compute.internal env[8472]: Started wazuh-logcollector... Sep 08 09:54:51 ip-172-31-11-12.us-east-2.compute.internal env[8472]: Started wazuh-monitord... Sep 08 09:54:52 ip-172-31-11-12.us-east-2.compute.internal env[8472]: Started wazuh-modulesd... Sep 08 09:54:54 ip-172-31-11-12.us-east-2.compute.internal env[8472]: Completed. Sep 08 09:54:54 ip-172-31-11-12.us-east-2.compute.internal systemd[1]: Started Wazuh manager. [root@ip-172-31-11-12 ec2-user]# yum -y install filebeat Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Resolving Dependencies There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help). --> Running transaction check ---> Package filebeat.x86_64 0:7.10.2-1 will be installed --> Finished Dependency Resolution Dependencies Resolved ===================================================================================================================== Package Arch Version Repository Size ===================================================================================================================== Installing: filebeat x86_64 7.10.2-1 wazuh 21 M Transaction Summary ===================================================================================================================== Install 1 Package Total download size: 21 M Installed size: 70 M Downloading packages: filebeat-oss-7.10.2-x86_64.rpm | 21 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : filebeat-7.10.2-1.x86_64 1/1 Verifying : filebeat-7.10.2-1.x86_64 1/1 Installed: filebeat.x86_64 0:7.10.2-1 Complete! [root@ip-172-31-11-12 ec2-user]# curl -so /etc/filebeat/filebeat.yml https://packages.wazuh.com/4.3/tpl/wazuh/filebeat/filebeat.yml [root@ip-172-31-11-12 ec2-user]# nano /etc/filebeat/filebeat.yml [root@ip-172-31-11-12 ec2-user]# filebeat keystore create Created filebeat keystore [root@ip-172-31-11-12 ec2-user]# echo admin | filebeat keystore add username --stdin --force Successfully updated the keystore [root@ip-172-31-11-12 ec2-user]# echo admin | filebeat keystore add password --stdin --force Successfully updated the keystore [root@ip-172-31-11-12 ec2-user]# curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/4.3/extensions/elasticsearch/7.x/wazuh-template.json [root@ip-172-31-11-12 ec2-user]# chmod go+r /etc/filebeat/wazuh-template.json [root@ip-172-31-11-12 ec2-user]# curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.2.tar.gz | tar -xvz -C /usr/share/filebeat/module wazuh/alerts/ wazuh/alerts/config/ wazuh/alerts/config/alerts.yml wazuh/alerts/manifest.yml wazuh/alerts/ingest/ wazuh/alerts/ingest/pipeline.json wazuh/archives/ wazuh/archives/config/ wazuh/archives/config/archives.yml wazuh/archives/manifest.yml wazuh/archives/ingest/ wazuh/archives/ingest/pipeline.json wazuh/module.yml [root@ip-172-31-11-12 ec2-user]# NODE_NAME=wazuh-1 [root@ip-172-31-11-12 ec2-user]# mkdir /etc/filebeat/certs [root@ip-172-31-11-12 ec2-user]# tar -xf ./wazuh-certificates.tar -C /etc/filebeat/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem [root@ip-172-31-11-12 ec2-user]# mv -n /etc/filebeat/certs/$NODE_NAME.pem /etc/filebeat/certs/filebeat.pem [root@ip-172-31-11-12 ec2-user]# mv -n /etc/filebeat/certs/$NODE_NAME-key.pem /etc/filebeat/certs/filebeat-key.pem [root@ip-172-31-11-12 ec2-user]# chmod 500 /etc/filebeat/certs [root@ip-172-31-11-12 ec2-user]# chmod 400 /etc/filebeat/certs/* [root@ip-172-31-11-12 ec2-user]# chown -R root:root /etc/filebeat/certs [root@ip-172-31-11-12 ec2-user]# systemctl daemon-reload [root@ip-172-31-11-12 ec2-user]# systemctl enable filebeat [root@ip-172-31-11-12 ec2-user]# systemctl start filebeat [root@ip-172-31-11-12 ec2-user]# filebeat test output elasticsearch: https://172.31.8.198:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 172.31.8.198 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 7.10.2 elasticsearch: https://172.31.40.101:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 172.31.40.101 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 7.10.2 elasticsearch: https://172.31.40.61:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 172.31.40.61 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 7.10.2 ````

wazuh-dashboard

```` [root@ip-172-31-4-239 ec2-user]# yum install -y wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm Loaded plugins: extras_suggestions, langpacks, priorities, update-motd Examining wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm: wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 Marking wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package wazuh-dashboard.x86_64 0:4.4.0-0.40400.20220808 will be installed --> Finished Dependency Resolution amzn2-core/2/x86_64 | 3.7 kB 00:00:00 Dependencies Resolved ===================================================================================================================== Package Arch Version Repository Size ===================================================================================================================== Installing: wazuh-dashboard x86_64 4.4.0-0.40400.20220808 /wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 645 M Transaction Summary ===================================================================================================================== Install 1 Package Total size: 645 M Installed size: 645 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 1/1 Verifying : wazuh-dashboard-4.4.0-0.40400.20220808.x86_64 1/1 Installed: wazuh-dashboard.x86_64 0:4.4.0-0.40400.20220808 Complete! [root@ip-172-31-4-239 ec2-user]# nano /etc/wazuh-dashboard/opensearch_dashboards.yml [root@ip-172-31-4-239 ec2-user]# NODE_NAME=dashboard [root@ip-172-31-4-239 ec2-user]# mkdir /etc/wazuh-dashboard/certs [root@ip-172-31-4-239 ec2-user]# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-dashboard/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem [root@ip-172-31-4-239 ec2-user]# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME.pem /etc/wazuh-dashboard/certs/dashboard.pem mv: ‘/etc/wazuh-dashboard/certs/dashboard.pem’ and ‘/etc/wazuh-dashboard/certs/dashboard.pem’ are the same file [root@ip-172-31-4-239 ec2-user]# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem mv: ‘/etc/wazuh-dashboard/certs/dashboard-key.pem’ and ‘/etc/wazuh-dashboard/certs/dashboard-key.pem’ are the same file [root@ip-172-31-4-239 ec2-user]# chmod 500 /etc/wazuh-dashboard/certs [root@ip-172-31-4-239 ec2-user]# chmod 400 /etc/wazuh-dashboard/certs/* [root@ip-172-31-4-239 ec2-user]# chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs [root@ip-172-31-4-239 ec2-user]# systemctl daemon-reload [root@ip-172-31-4-239 ec2-user]# systemctl enable wazuh-dashboard [root@ip-172-31-4-239 ec2-user]# systemctl start wazuh-dashboard ````

okynos commented 1 year ago

We have detected that the mentioned option could decrease the number of messages in the log but they won't go completely. More details about the development and the results here https://github.com/wazuh/wazuh-packages/issues/1968

wazuh / wazuh-packages