Description
I updated one of my servers where the wazuh dashboard, wazuh indexer, wazuh manager ran. after rebooting it since it included a kernel update I could no longer access wazuh.
Expected Result
I could still connect to the wazuh dashboard without a problem.
[2024-07-03T10:03:12,658][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms15824m, -Xmx15824m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5556094357872037470, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=8296333312, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2024-07-03T10:03:14,485][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /etc/wazuh-indexer/opensearch-security/roles.yml has insecure file permissions (should be 0600)
[2024-07-03T10:03:14,486][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /etc/wazuh-indexer/opensearch-security/roles_mapping.yml has insecure file permissions (should be 0600)
[2024-07-03T10:03:14,486][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /etc/wazuh-indexer/opensearch-security/internal_users.yml has insecure file permissions (should be 0600)
[2024-07-03T10:03:18,932][WARN ][o.o.s.c.Salt ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-07-03T10:03:18,973][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2024-07-03T10:03:18,974][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.
[2024-07-03T10:03:20,067][WARN ][o.o.s.p.SQLPlugin ] [node-1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2024-07-03T10:03:20,425][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,482][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,483][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,484][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,484][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,484][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,484][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,485][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,485][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,485][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,486][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,486][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,486][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,486][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,486][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,487][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,488][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,488][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,489][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,489][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,489][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,489][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,490][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,490][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,490][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,490][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,490][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,491][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,491][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,491][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,492][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,492][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,493][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,493][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,493][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:20,494][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-07-03T10:03:21,247][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-07-03T10:03:22,464][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.
[2024-07-03T10:03:22,733][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/XxPay9qERr6ww1DiCLvYMw] already exists
[2024-07-03T10:03:22,738][WARN ][o.o.s.SecurityAnalyticsPlugin] [node-1] Failed to initialize LogType config index and builtin log types
[2024-07-03T10:03:23,082][ERROR][o.o.b.Bootstrap ] [node-1] Exception
[2024-07-03T10:03:23,090][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.10.0.jar:2.10.0]
[2024-07-03T10:03:23,462][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-07-03T10:03:23,487][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-07-03T10:03:23,491][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-07-03T10:03:23,496][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
systemctl status wazuh-indexer.service wazuh-dashboard.service wazuh-manager.service
INFO: No current API selected
INFO: Getting API hosts...
INFO: API hosts found: 1
INFO: Checking API host id [default]...
INFO: Could not connect to API id [default]: 3099 - ERROR3099 - Wazuh not ready yet
INFO: Removed [navigate] cookie
ERROR: No API available to connect
Check alerts index pattern
INFO: Index pattern id in cookie: yes [wazuh-alerts-*]
INFO: Getting list of valid index patterns...
INFO: Valid index patterns found: 1
INFO: Found default index pattern with title [wazuh-alerts-*]: yes
INFO: Checking the app default pattern exists: id [wazuh-alerts-*]...
INFO: Default pattern with id [wazuh-alerts-*] exists: yes
ACTION: Default pattern id [wazuh-alerts-*] set as default index pattern
INFO: Checking the index pattern id [wazuh-alerts-*] exists...
INFO: Index pattern id exists [wazuh-alerts-*]: yes
INFO: Index pattern id in cookie: yes [wazuh-alerts-*]
INFO: Checking if the index pattern id [wazuh-alerts-*] exists...
INFO: Index pattern id [wazuh-alerts-*] found: yes title [wazuh-alerts-*]
INFO: Checking if exists a template compatible with the index pattern title [wazuh-alerts-*]
INFO: Template found for the selected index-pattern title [wazuh-alerts-*]: yes
INFO: Index pattern id in cookie: [wazuh-alerts-*]
INFO: Getting index pattern data [wazuh-alerts-*]...
INFO: Index pattern data found: [yes]
INFO: Refreshing index pattern fields: title [wazuh-alerts-*], id [wazuh-alerts-*]...
ACTION: Refreshed index pattern fields: title [wazuh-alerts-*], id [wazuh-alerts-*]
INFO: Getting settings...
INFO: Check Wazuh dashboard setting [timeline:max_buckets]: 200000
INFO: App setting [timeline:max_buckets]: 200000
INFO: Settings mismatch [timeline:max_buckets]: no
INFO: Getting settings...
INFO: Check Wazuh dashboard setting [metaFields]: ["_source","_index"]
INFO: App setting [metaFields]: ["_source","_index"]
INFO: Settings mismatch [metaFields]: no
INFO: Getting settings...
INFO: Check Wazuh dashboard setting [timepicker:timeDefaults]: {"from":"now-24h","to":"now"}
INFO: App setting [timepicker:timeDefaults]: "{\"from\":\"now-24h\",\"to\":\"now\"}"
INFO: Settings mismatch [timepicker:timeDefaults]: no
curl -k -X GET "https://<api_url>:55000/" -H "Authorization: Bearer $(curl -u <api_user>:<api_password> -k -X POST 'https://<api_url>:55000/security/user/authenticate?raw=true')"
Description I updated one of my servers where the wazuh dashboard, wazuh indexer, wazuh manager ran. after rebooting it since it included a kernel update I could no longer access wazuh.
Expected Result
Actual Result
Additional context /var/log/wazuh-indexer/wazuh-cluster.log:
systemctl status wazuh-indexer.service wazuh-dashboard.service wazuh-manager.service
Check Wazuh API connection
Check alerts index pattern
curl -k -X GET "https://<api_url>:55000/" -H "Authorization: Bearer $(curl -u <api_user>:<api_password> -k -X POST 'https://<api_url>:55000/security/user/authenticate?raw=true')"