Closed juliamagan closed 1 year ago
verdx
commented
1 year ago Some tests have been done and with a reboot the Wazuh Indexer has worked fine, but the logs have been lost because of networking problems with the AWS machines, the tests need to be repeated.
verdx
commented
1 year ago A test has been performed and the error has been not been replicated:
nodes:
# Wazuh indexer nodes
indexer:
- name: indexer-1
ip: 172.31.32.33
# Wazuh server nodes
# If there is more than one Wazuh server
# node, each one must have a node_type
server:
- name: server-1
ip: 172.31.34.245
# Wazuh dashboard nodes
dashboard:
- name: dashboard
ip: 172.31.42.160[ec2-user@ip-172-31-32-33 ~]$ sudo bash wazuh-install.sh --generate-config-files
09/01/2023 10:10:48 INFO: Starting Wazuh installation assistant. Wazuh version: 4.4.0
09/01/2023 10:10:48 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/01/2023 10:10:50 INFO: --- Configuration files ---
09/01/2023 10:10:50 INFO: Generating configuration files.
09/01/2023 10:10:50 INFO: Created wazuh-install-files.tar. It contains the Wazuh cluster key, certificates, and passwords necessary for installation.
[ec2-user@ip-172-31-32-33 ~]$ sudo chmod a+rw wazuh-install-files.tar
[ec2-user@ip-172-31-32-33 ~]$ sudo bash wazuh-install.sh --wazuh-indexer indexer-1
09/01/2023 10:14:41 INFO: Starting Wazuh installation assistant. Wazuh version: 4.4.0
09/01/2023 10:14:41 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/01/2023 10:14:46 INFO: Wazuh development repository added.
09/01/2023 10:14:46 INFO: --- Wazuh indexer ---
09/01/2023 10:14:46 INFO: Starting Wazuh indexer installation.
09/01/2023 10:15:40 INFO: Wazuh indexer installation finished.
09/01/2023 10:15:40 INFO: Wazuh indexer post-install configuration finished.
09/01/2023 10:15:40 INFO: Starting service wazuh-indexer.
09/01/2023 10:15:52 INFO: wazuh-indexer service started.
09/01/2023 10:15:52 INFO: Initializing Wazuh indexer cluster security settings.
09/01/2023 10:15:54 INFO: Wazuh indexer cluster initialized.
09/01/2023 10:15:54 INFO: Installation finished.
[ec2-user@ip-172-31-32-33 ~]$ sudo su
[root@ip-172-31-32-33 ec2-user]# bash wazuh-install.sh --start-cluster
09/01/2023 10:17:24 INFO: Starting Wazuh installation assistant. Wazuh version: 4.4.0
09/01/2023 10:17:24 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/01/2023 10:17:33 INFO: Wazuh indexer cluster security configuration initialized.
09/01/2023 10:17:41 INFO: Wazuh indexer cluster started.
[root@ip-172-31-32-33 ec2-user]# tar -axf wazuh-install-files.tar wazuh-install-files/wazuh-passwords.txt -O | grep -P "\'admin\'" -A 1
indexer_username: 'admin'
indexer_password: 'ICzyV*i4tpAo*NdZpGjpk9NCFkhbf8aH'
[root@ip-172-31-32-33 ec2-user]# curl -k -u admin:ICzyV*i4tpAo*NdZpGjpk9NCFkhbf8aH https://172.31.32.33:9200
{
"name" : "indexer-1",
"cluster_name" : "wazuh-indexer-cluster",
"cluster_uuid" : "BURnJ4NPSDaaVUukE84KRQ",
"version" : {
"number" : "7.10.2",
"build_type" : "rpm",
"build_hash" : "f2f809ea280ffba217451da894a5899f1cec02ab",
"build_date" : "2022-12-12T22:17:42.341124910Z",
"build_snapshot" : false,
"lucene_version" : "9.4.2",
"minimum_wire_compatibility_version" : "7.10.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}[root@ip-172-31-34-245 ec2-user]# bash wazuh-install.sh --wazuh-server server-1
bash: wazuh-install.sh: No such file or directory
[root@ip-172-31-34-245 ec2-user]# curl -sO https://packages-dev.wazuh.com/4.4/wazuh-install.sh
[root@ip-172-31-34-245 ec2-user]# bash wazuh-install.sh --wazuh-server server-1
09/01/2023 10:30:21 INFO: Starting Wazuh installation assistant. Wazuh version: 4.4.0
09/01/2023 10:30:21 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/01/2023 10:30:25 INFO: Wazuh development repository added.
09/01/2023 10:30:25 INFO: --- Wazuh server ---
09/01/2023 10:30:25 INFO: Starting the Wazuh manager installation.
09/01/2023 10:30:41 INFO: Wazuh manager installation finished.
09/01/2023 10:30:41 INFO: Starting service wazuh-manager.
09/01/2023 10:30:53 INFO: wazuh-manager service started.
09/01/2023 10:30:53 INFO: Starting Filebeat installation.
09/01/2023 10:31:05 INFO: Filebeat installation finished.
09/01/2023 10:31:05 INFO: Filebeat post-install configuration finished.
09/01/2023 10:31:09 INFO: Starting service filebeat.
09/01/2023 10:31:10 INFO: filebeat service started.
09/01/2023 10:31:10 INFO: Installation finished.
[root@ip-172-31-34-245 ec2-user]# filebeat test output
elasticsearch: https://172.31.32.33:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 172.31.32.33
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.2
dial up... OK
talk to server... OK
version: 7.10.2[root@ip-172-31-42-160 ec2-user]# curl -sO https://packages-dev.wazuh.com/4.4/wazuh-install.sh
[root@ip-172-31-42-160 ec2-user]# bash wazuh-install.sh --wazuh-dashboard dashboard
09/01/2023 10:32:29 INFO: Starting Wazuh installation assistant. Wazuh version: 4.4.0
09/01/2023 10:32:29 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/01/2023 10:32:33 INFO: Wazuh development repository added.
dashboard
09/01/2023 10:32:33 INFO: --- Wazuh dashboard ----
09/01/2023 10:32:33 INFO: Starting Wazuh dashboard installation.
09/01/2023 10:33:36 INFO: Wazuh dashboard installation finished.
09/01/2023 10:33:36 INFO: Wazuh dashboard post-install configuration finished.
09/01/2023 10:33:36 INFO: Starting service wazuh-dashboard.
09/01/2023 10:33:36 INFO: wazuh-dashboard service started.
09/01/2023 10:33:53 INFO: Initializing Wazuh dashboard web application.
09/01/2023 10:33:53 INFO: Wazuh dashboard web application initialized.
09/01/2023 10:33:53 INFO: --- Summary ---
09/01/2023 10:33:53 INFO: You can access the web interface https://172.31.42.160
User: admin
Password: ICzyV*i4tpAo*NdZpGjpk9NCFkhbf8aH
09/01/2023 10:33:53 INFO: Installation finished.Invoke-WebRequest -Uri https://packages-dev.wazuh.com/pre-release/windows/wazuh-agent-4.4.0-1.msi -OutFile ${env:tmp}\wazuh-agent-4.4.0.msi; msiexec.exe /i ${env:tmp}\wazuh-agent-4.4.0.msi /q WAZUH_MANAGER='172.31.34.245' WAZUH_AGENT_GROUP='windows' WAZUH_AGENT_NAME='windows-agent'NET Start WazuhSvc
verdx
commented
1 year ago In hold, waiting for new information by @juliamagan
verdx
commented
1 year ago | In progress again after a new test by @juliamagan has managed to replicate the error in an AIO installation. | |||
|---|---|---|---|
| Component | SO | Type of instance | |
| AIO installation | Amazon Linux 2 | c5.xlarge | |
| Agent | Windows | t3.small |
Steps to reproduce:
[root@ip-172-31-30-232 qa]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2023-01-10 08:55:30 UTC; 4min 6s ago
Docs: https://documentation.wazuh.com/
Process: 2472 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 2472 (code=exited, status=1/FAILURE)Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd-entrypoint[2472]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd-entrypoint[2472]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd-entrypoint[2472]: at org.opensearch.cli.Command.main(Command.java:101) Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd-entrypoint[2472]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd-entrypoint[2472]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd-entrypoint[2472]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-indexer-cluster.log Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd[1]: wazuh-indexer.service: main process exited, code=exited, status=1/FAILURE Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd[1]: Failed to start Wazuh-indexer. Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd[1]: Unit wazuh-indexer.service entered failed state. Jan 10 08:55:30 ip-172-31-30-232.ec2.internal systemd[1]: wazuh-indexer.service failed.
<details>
<summary>`config.yml` used</summary>
nodes:
indexer:
name: node-1 ip: 172.31.30.232
server:
name: wazuh-1 ip: 172.31.30.232
dashboard:
</details>-a to installOn the same instance as @juliamagan, and using the same config.yml, another test has been done. Steps:
As can be seen, when only installing the components and rebooting the system, the Wazuh Indexer does not give any error. This means the problem may have to do with the agent deployment.
verdx
commented
1 year ago -a to installUsing the same installation and instance as in the previous test, we now add an agent.
After installing Wazuh All-in-one and connecting the agent using the same instances, configurations and commands, the error has not been replicated.
verdx
commented
1 year ago AIO as per the documentationFinally, installing the components as shown in the documentation instead of with argument -a of wazuh-install.sh has permitted the error to be replicated.
The error is replicated when all components are installed in the same host using the Installation Assistant to install them separately, as seen in the documentation. The agent deployment is not necessary for the error to appear. As per this test, it appears it is only replicated when using one host only.
verdx
commented
1 year ago Another test was done, where the reboot was tested after the installation of each component.
Even though the error is clearly related to the Wazuh Indexer, it seems it is triggered by the installation of the Wazuh Server, which would explain why doing a distributed installation doesn't replicate it.
verdx
commented
1 year ago After more investigation, the problem has been has been reduced to a permissions error coming from a file /etc/wazuh-indexer/backup, with owner root:root. This file is created in function passwords_changePassword() and removed in function passwords_runSecurityAdmin()
The script tries to create it every time passwords_changePassword() is called:
-a is passed for an AIO installation--wazuh-indexer)--start-cluster)--wazuh-server)--wazuh-dashboard)On the other hand, function passwords_runSecurityAdmin(), which removes the file is only called in two of those occasions:
--start-cluster)-a is passed for an AIO installationThe problem in this Issue came from the fact that when the last function called from the two of them was passwords_changePassword(), file /etc/wazuh-indexer/backup with incorrect permissions was left created, and when the service for the Wazuh Indexer was restarted, it had a permissions problems with it. That explains why the error didn't appear neither when using argument -a or when just installing one node, where the last use of the script was starting the cluster.
DFolchA
commented
1 year ago verdx
commented
1 year ago -a Installation (Amazon Linux 2)On each step, the tests performed are checking if directory /etc/wazuh-indexer/backup and to restart the Wazuh Indexer service and check if it doesn't stop. Normal tests for the cluster and filebeat are also performed.
| Component | OS |
|---|---|
| CentOS 7 | Wazuh Indexer |
| RHEL 9 | Wazuh server |
| Debian 10 | Wazuh Dashboard |
| Windows XP | Agent |
On each step, the tests performed are checking if directory /etc/wazuh-indexer/backup and to restart the Wazuh Indexer service and check if it doesn't stop. Normal tests for the cluster and filebeat are also performed.
verdx
commented
1 year ago -a Installation (Ubuntu 22)verdx
commented
1 year ago | Name | SO | Component(s) |
|---|---|---|
| U1 | Ubuntu 22 | indexer-1 |
| U2 | Ubuntu 22 | server-1 |
| U1 | Ubuntu 22 | server-2 |
| A1 | Amazon Linux 2 | indexer-2 |
| A2 | Amazon Linux 2 | indexer-3, server-3 |
| A3 | Amazon Linux 2 | dashboard |
verdx
commented
1 year ago | CentOS 8 | RHEL 8 | Ubuntu 18 | |
|---|---|---|---|
| Test 1 | indexer and server | dashboard | - |
| Test 2 | server | - | indexer and dashboard |
| Test 3 | - | dashboard and server | indexer |
Environment info
Description
After the testing performed here, we could see that when we reboot our environment, wazuh-indexer can't start:
We have to reinstall it to get it working again.
In addition, when we tested some Sysmon events here, alerts stopped being indexed.
When we tried to investigate what was going on, we found that the indexer logs did not exist:
And the only information we get from the dashboard is the following:
Attached logs - Before reinstall
Attached logs - After reinstall
Logs are deleted after reinstalling wazuh-dashboard