rauldpm
commented
2 years ago Update report
WUI access error
 ``` INFO: No current API selected INFO: Getting API hosts... INFO: API hosts found: 1 INFO: Checking API host id [default]... INFO: Could not connect to API id [default]: 3099 - ERROR3099 - Wazuh not ready yet INFO: Removed [navigate] cookie ERROR: No API available to connect ```api.log error
``` root@ip-172-31-15-20:/home/ubuntu# cat /var/ossec/logs/api.log 2022/06/14 18:36:33 INFO: HTTPS is enabled but cannot find the private key and/or certificate. Attempting to generate them 2022/06/14 18:36:33 INFO: Generated private key file in WAZUH_PATH/api/configuration/ssl/server.key 2022/06/14 18:36:33 INFO: Generated certificate file in WAZUH_PATH/api/configuration/ssl/server.crt 2022/06/14 18:36:45 INFO: Listening on 0.0.0.0:55000.. 2022/06/14 18:36:45 ERROR: 2010 - Error while attempting to bind on address: address already in use. ```Process and wazuh-control status (wazuh-apid not running)
``` root@ip-172-31-15-20:/home/ubuntu# /var/ossec/bin/wazuh-control status wazuh-clusterd not running... wazuh-modulesd is running... wazuh-monitord is running... wazuh-logcollector is running... wazuh-remoted is running... wazuh-syscheckd is running... wazuh-analysisd is running... wazuh-maild not running... wazuh-execd is running... wazuh-db is running... wazuh-authd is running... wazuh-agentlessd not running... wazuh-integratord not running... wazuh-dbd not running... wazuh-csyslogd not running... wazuh-apid not running... root@ip-172-31-15-20:/home/ubuntu# ps -ef | grep wazuh wazuh-i+ 10136 1 2 18:35 ? 00:02:41 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms3936m -Xmx3936m -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/opensearch-1582401709826038832 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy -XX:MaxDirectMemorySize=2063597568 -Dopensearch.path.home=/usr/share/wazuh-indexer -Dopensearch.path.conf=/etc/wazuh-indexer -Dopensearch.distribution.type=rpm -Dopensearch.bundled_jdk=true -cp /usr/share/wazuh-indexer/lib/* org.opensearch.bootstrap.OpenSearch -p /run/wazuh-indexer/wazuh-indexer.pid --quiet root 48524 1 0 18:36 ? 00:00:18 /var/ossec/bin/wazuh-authd wazuh 48539 1 0 18:36 ? 00:00:05 /var/ossec/bin/wazuh-db root 48564 1 0 18:36 ? 00:00:00 /var/ossec/bin/wazuh-execd wazuh 48583 1 0 18:36 ? 00:00:02 /var/ossec/bin/wazuh-analysisd root 48627 1 0 18:36 ? 00:00:14 /var/ossec/bin/wazuh-syscheckd wazuh 48642 1 0 18:36 ? 00:00:10 /var/ossec/bin/wazuh-remoted root 48674 1 0 18:36 ? 00:00:01 /var/ossec/bin/wazuh-logcollector wazuh 48693 1 0 18:36 ? 00:00:00 /var/ossec/bin/wazuh-monitord root 48704 1 0 18:36 ? 00:00:02 /var/ossec/bin/wazuh-modulesd wazuh-d+ 51028 1 0 18:38 ? 00:00:26 /usr/share/wazuh-dashboard/bin/../node/bin/node --no-warnings --max-http-header-size=65536 --unhandled-rejections=warn /usr/share/wazuh-dashboard/bin/../src/cli/dist -c /etc/wazuh-dashboard/opensearch_dashboards.yml root 53340 53269 0 20:44 pts/0 00:00:00 grep --color=auto wazuh root@ip-172-31-15-20:/home/ubuntu# ```wazuh-apid debug output
``` root@ip-172-31-15-20:/var/ossec/bin# ./wazuh-apid -f -dd Starting API in foreground 2022/06/14 20:47:47 INFO: Listening on 0.0.0.0:55000.. unhandled exception during asyncio.run() shutdown task:- Port 55000 in use by ssm-agent
root@ip-172-31-15-20:/var/ossec/bin# lsof -i:55000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ssm-agent 3669 root 15u IPv4 45710 0t0 TCP ip-172-31-15-20.ec2.internal:55000->209.54.183.5:https (ESTABLISHED)
root@ip-172-31-15-20:/var/ossec/bin#
A random error has been found in the
check_wazuh_api_statustest. This error appears on various systems randomly.Related: https://github.com/wazuh/wazuh-packages/pull/1619/checks?check_run_id=6886741506#
Using the same passwords, an AIO installation on the same affected system does not show the same error when passing the test.
On the machine where the build failed, the following is observed:
Wazuh indexer journalctl output
``` root@ip-172-31-15-20:/tmp/unattended/tests/unattended/install# journalctl -r -u wazuh-indexer -- Logs begin at Mon 2021-07-12 16:47:29 UTC, end at Tue 2022-06-14 20:17:01 UTC. -- Jun 14 18:35:27 ip-172-31-15-20 systemd[1]: Started Wazuh-indexer. Jun 14 18:35:24 ip-172-31-15-20 systemd-entrypoint[10136]: WARNING: All illegal access operations will be denied in a future release Jun 14 18:35:24 ip-172-31-15-20 systemd-entrypoint[10136]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations Jun 14 18:35:24 ip-172-31-15-20 systemd-entrypoint[10136]: WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema Jun 14 18:35:24 ip-172-31-15-20 systemd-entrypoint[10136]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/wazuh-indexer/plugins/opensearch-anomaly-detect> Jun 14 18:35:24 ip-172-31-15-20 systemd-entrypoint[10136]: WARNING: An illegal reflective access operation has occurred Jun 14 18:35:08 ip-172-31-15-20 systemd[1]: Starting Wazuh-indexer... ```Wazuh dashboard journalctl output
``` root@ip-172-31-15-20:/tmp/unattended/tests/unattended/install# journalctl -r -u wazuh-dashboard | grep -i -E "error|critical|fatal|warning" Jun 14 18:38:22 ip-172-31-15-20 opensearch-dashboards[51028]: {"type":"log","@timestamp":"2022-06-14T18:38:22Z","tags":["error","opensearch","data"],"pid":51028,"message":"[ResponseError]: Response Error"} Jun 14 18:38:19 ip-172-31-15-20 opensearch-dashboards[51028]: {"type":"log","@timestamp":"2022-06-14T18:38:19Z","tags":["error","savedobjects-service"],"pid":51028,"message":"Unable to retrieve version information from OpenSearch nodes."} Jun 14 18:38:19 ip-172-31-15-20 opensearch-dashboards[51028]: {"type":"log","@timestamp":"2022-06-14T18:38:19Z","tags":["error","opensearch","data"],"pid":51028,"message":"[ResponseError]: Response Error"} Jun 14 18:37:52 ip-172-31-15-20 opensearch-dashboards[49886]: {"type":"log","@timestamp":"2022-06-14T18:37:52Z","tags":["error","opensearch","data"],"pid":49886,"message":"[ResponseError]: Response Error"} Jun 14 18:37:52 ip-172-31-15-20 opensearch-dashboards[49886]: {"type":"log","@timestamp":"2022-06-14T18:37:52Z","tags":["error","opensearch","data"],"pid":49886,"message":"[ResponseError]: Response Error"} root@ip-172-31-15-20:/tmp/unattended/tests/unattended/install# ```Wazuh indexer wazuh-cluster.log errors
``` root@ip-172-31-15-20:/tmp/unattended/tests/unattended/install# cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warning|fatal|critical" [2022-06-14T18:35:13,584][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms3936m, -Xmx3936m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-1582401709826038832, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=2063597568, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true] [2022-06-14T18:35:22,844][WARN ][stderr ] [node-1] java.util.ServiceConfigurationError: com.sun.tools.attach.spi.AttachProvider: Provider sun.tools.attach.AttachProviderImpl could not be instantiated [2022-06-14T18:35:23,884][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly. [2022-06-14T18:35:27,689][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security) [2022-06-14T18:35:28,216][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin) root@ip-172-31-15-20:/tmp/unattended/tests/unattended/install# ```Filebeat test output
``` root@ip-172-31-15-20:/tmp/unattended/tests/unattended/install# filebeat test output elasticsearch: https://127.0.0.1:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 127.0.0.1 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 7.10.2 ```Wazuh indexer cluster api calls
``` root@ip-172-31-15-20:/tmp/unattended/tests/unattended/install# curl -k -u admin:zbsw379rR0Hj8O2lW?LSafpWV+pjk*ox https://127.0.0.1:9200 { "name" : "node-1", "cluster_name" : "wazuh-cluster", "cluster_uuid" : "RvYYBRJuQ_yHeKvAabuHuQ", "version" : { "number" : "7.10.2", "build_type" : "rpm", "build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f", "build_date" : "2022-01-14T03:38:06.881862Z", "build_snapshot" : false, "lucene_version" : "8.10.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } root@ip-172-31-15-20:/tmp/unattended/tests/unattended/install# curl -k -u admin:zbsw379rR0Hj8O2lW?LSafpWV+pjk*ox https://127.0.0.1:9200/_cat/nodes?v ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 127.0.0.1 47 96 2 0.00 0.00 0.00 dimr * node-1 root@ip-172-31-15-20:/tmp/unattended/tests/unattended/install# curl -k -u admin:zbsw379rR0Hj8O2lW?LSafpWV+pjk*ox https://127.0.0.1:9200/_cluster/health?pretty { "cluster_name" : "wazuh-cluster", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "active_primary_shards" : 5, "active_shards" : 5, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ```Another error related to this test has been found, but it has not been observed again (it is the original error that was tried to be reproduced), possibly related to obtaining the password of the
wazuhuser.